Logs (VictoriaLogs)

Proxima Console uses VictoriaLogs for centralized log storage and search. Logs come from two sources:

1. Agent log collection -- journald and file tailing from managed hosts, shipped via NATS
2. Docker container logs -- backend/frontend container logs shipped via Vector (local dev)

Where VictoriaLogs runs (production vs local dev)

In production, the Console backend runs in the proxima-production Kubernetes cluster and ships agent logs (the LogsWorker pipeline below) to VictoriaLogs running on the box — reached over its public IP at :9428, firewalled to the cluster egress. This is interim: VictoriaLogs moves to a dedicated VM in a later phase. The backend's target is set via PROXIMA_VICTORIALOGS_URL (see Environment Variables).

In local dev, VictoriaLogs runs in docker-compose.yml at http://localhost:9428 (see Docker Compose Services). The Vector container-log pipeline that tails proxima-* Docker containers is a local-dev concept — in production, container/pod stdout is handled by the cluster's own logging, while this page's agent-log pipeline is what the Console backend ships to VictoriaLogs.

Stack

• Agent logs: journald collector + file tail collector → NATS → LogsWorker → VictoriaLogs
• Container logs (local dev): Docker JSON file log driver → Vector → VictoriaLogs
• Storage: VictoriaLogs v1.46.0 (30-day retention in dev). Production: on the box (interim, moving to a dedicated VM).
• Multi-tenancy: Native AccountID HTTP headers (same model as VictoriaMetrics)
• Query language: LogsQL
• Frontend: VictoriaLogs VMUI embedded via authenticated reverse proxy

Agent Log Collection Pipeline

The agent collects logs from managed hosts and ships them to VictoriaLogs via the backend:

Log Entry Fields

Each log entry stored in VictoriaLogs includes:

Field	Description
_msg	Log message text
_time	Timestamp (RFC3339Nano)
host_id	UUID of the source host
client_id	UUID of the client (tenant)
environment_id	UUID of the environment
source	Origin: journald or file:/path/to/log
level	Severity: emerg, alert, crit, error, warn, notice, info, debug
unit	Systemd unit name (journald only)

Multi-Tenancy

VictoriaLogs supports native multi-tenancy via AccountID HTTP headers:

• On insert: The LogsWorker resolves the host's vm_account_id via tenantcache and sets the AccountID header
• On query: The logs proxy handler (/api/v1/logs/query and /api/v1/logs/select/*) injects the AccountID header based on the authenticated user's client
• Each client's logs are physically isolated -- queries only return data for the authenticated tenant

Agent Configuration

Enable log collection in the agent config:

agent:
  log_collection:
    enabled: true
    journald:
      enabled: true
      units: []           # empty = all units
    files:
      - path: /var/log/syslog
      - path: /var/log/nginx/access.log
        name: nginx-access
        parser: json
    buffer_size: 1000     # entries to buffer before flush
    max_line_len: 8192    # truncate lines longer than this
  log_interval: 10s       # flush interval

Or via environment variables:

PROXIMA_AGENT_LOG_COLLECTION_ENABLED=true
PROXIMA_AGENT_LOG_INTERVAL=10s

Backend API Endpoints

Endpoint	Method	Description
/api/v1/logs/query	POST	LogsQL query proxy (accepts JSON body, forwards as URL params to VictoriaLogs)
/api/v1/logs/select/*	GET/POST	VictoriaLogs reverse proxy (VMUI static assets + API)

The query endpoint accepts:

{
  "query": "level:error AND unit:nginx.service",
  "start": "2026-03-05T00:00:00Z",
  "end": "2026-03-05T23:59:59Z",
  "limit": 1000
}

Docker Compose Services (Local Dev)

Local development only

This docker-compose stack (VictoriaLogs + Vector) is for local development. In production the backend ships agent logs to VictoriaLogs on the box (interim) — see the Stack note above.

VictoriaLogs and Vector run in every local environment (no profile gate) — make docker-up sets COMPOSE_PROFILES=dev locally, but both also start on a plain docker compose up -d:

docker compose up -d

Service	Image	Port	Purpose
VictoriaLogs	victoriametrics/victoria-logs:v1.46.0	9428	Log storage (30d retention)
Vector	timberio/vector:0.44.0-debian	--	Collects Docker container logs, ships to VictoriaLogs

Vector Configuration

Local development only

Vector ships local Docker container logs to VictoriaLogs in docker-compose. It is not part of the production Kubernetes deployment, where pod stdout is handled by the cluster's logging stack.

The configuration is at infra/vector/vector.yaml. Pipeline:

• docker_logs source -- collects stdout/stderr of every proxima-* container via the Docker API (so co-located CI-runner / other-app containers are not captured)
• remap transform (VRL) -- parses Go slog JSON plus regex/logfmt for PostgreSQL, NATS, and Tempo; derives container_name / service_name; normalizes level across all container types
• http sink -- POSTs newline-delimited JSON to http://victorialogs:9428/insert/jsonline (tenant 0)

Vector connects to the Docker socket (/var/run/docker.sock, read-only) and ships parsed log entries to VictoriaLogs.

Stream Fields

VictoriaLogs _stream_fields are low-cardinality labels that appear as clickable dropdown filters in the Grafana/VMUI query interface:

Stream Field	Description	Example Values
container_name	Docker container name	proxima-backend, proxima-postgres
service_name	Same as container name (matches OTel service.name for trace correlation)	proxima-backend, proxima-nats
level	Log severity (normalized to standard slog levels)	INFO, WARN, ERROR, DEBUG, FATAL
stream	Docker output stream	stdout, stderr

Log Enrichment Pipeline

Docker JSON log → docker parser (extract time, stream, log field)
  → json_log parser (parse Go slog JSON inside "log" field)
  → postgres_log parser (extract pg_level + pg_msg from PostgreSQL format)
  → nats_log parser (extract nats_level + nats_msg from NATS format)
  → tempo_log parser (extract tempo_level + tempo_msg from Tempo logfmt)
  → enrich.lua:
      1. Resolve container_name from Docker config.v2.json
      2. Set service_name = container_name
      3. Normalize level: pg_level/nats_level/tempo_level → INFO/WARN/ERROR/DEBUG/FATAL
      4. Promote per-service msg fields, fallback to raw log line
  → VictoriaLogs HTTP output

Backend JSON logs already have level set by the json_log parser; the Lua script only normalizes level for non-JSON containers (PostgreSQL, NATS, Tempo).

LogsQL Query Examples

# Errors in the last hour (level is a stream field — use := for exact match)
curl -s 'http://localhost:9428/select/logsql/query' \
  -d 'query=_time:1h AND level:="ERROR"' -d 'limit=20'

# Filter by service (stream field)
curl -s 'http://localhost:9428/select/logsql/query' \
  -d 'query=_time:1h AND service_name:="proxima-backend"' -d 'limit=20'

# Cross-signal correlation: find logs by trace_id (from Tempo)
curl -s 'http://localhost:9428/select/logsql/query' \
  -d 'query=_time:1h AND trace_id:="abcdef1234567890"' -d 'limit=50'

# By request_id
curl -s 'http://localhost:9428/select/logsql/query' \
  -d 'query=_time:1h AND request_id:="abc-123"' -d 'limit=50'

# NATS worker errors
curl -s 'http://localhost:9428/select/logsql/query' \
  -d 'query=_time:1h AND component:="worker" AND level:="ERROR"' -d 'limit=20'

# PostgreSQL errors
curl -s 'http://localhost:9428/select/logsql/query' \
  -d 'query=_time:1h AND service_name:="proxima-postgres" AND level:="ERROR"' -d 'limit=20'

# Auth failures
curl -s 'http://localhost:9428/select/logsql/query' \
  -d 'query=_time:1h AND "failed to resolve access"' -d 'limit=20'

# Panics
curl -s 'http://localhost:9428/select/logsql/query' \
  -d 'query=_time:24h AND "panic recovered"' -d 'limit=10'

# Stats: log volume by level
curl -s 'http://localhost:9428/select/logsql/stats_query' \
  -d 'query=_time:1h' -d 'stats=count() by (level)'

# Stats: log volume by service
curl -s 'http://localhost:9428/select/logsql/stats_query' \
  -d 'query=_time:1h' -d 'stats=count() by (service_name)'

Stream fields vs regular fields

Stream fields (service_name, level, container_name, stream) appear as clickable dropdown filters in the Grafana/VMUI UI. All other fields (trace_id, request_id, component, etc.) are auto-indexed by VictoriaLogs and queryable with field:="value" syntax — they just don't appear in dropdown menus.

Log Format

All backend logs use structured JSON via slog:

{
  "time": "2026-02-27T10:00:00.000Z",
  "level": "INFO",
  "msg": "request completed",
  "component": "api",
  "request_id": "abc-123",
  "trace_id": "1234567890abcdef",
  "span_id": "abcdef12",
  "method": "GET",
  "path": "/api/v1/hosts",
  "status": 200,
  "duration_ms": 12.5
}