Logs (VictoriaLogs)

Proxima Console uses VictoriaLogs for centralized log storage and search. Logs come from two sources:

Agent log collection -- journald and file tailing from managed hosts, shipped via NATS
Docker container logs -- backend/frontend container logs shipped via Vector (local dev)

Where VictoriaLogs runs (production vs local dev)

In production, the Console backend runs in the proxima-production Kubernetes cluster and ships agent logs (the LogsWorker pipeline below) to VictoriaLogs running under systemd on vmstorage02 at :9428, with Console isolated in projectID 42. The backend's target is set via PROXIMA_VICTORIALOGS_URL and PROXIMA_VL_PROJECT_ID (see Environment Variables).

In local dev, VictoriaLogs runs in docker-compose.yml at http://localhost:9428 (see Docker Compose Services). The Vector container-log pipeline that tails proxima-* Docker containers is a local-dev concept — in production, container/pod stdout is handled by the cluster's own logging, while this page's agent-log pipeline is what the Console backend ships to VictoriaLogs.

Stack

Agent logs: journald collector + file tail collector → NATS → LogsWorker → VictoriaLogs
Container logs (local dev): Docker JSON file log driver → Vector → VictoriaLogs
Storage: VictoriaLogs v1.46.0 (30-day retention in dev). Production: systemd on vmstorage02 (:9428), Console in projectID 42.
Multi-tenancy: Native AccountID HTTP headers (same model as VictoriaMetrics)
Query language: LogsQL
Frontend: VictoriaLogs VMUI embedded via authenticated reverse proxy

Agent Log Collection Pipeline

The agent collects logs from managed hosts and ships them to VictoriaLogs via the backend:

Log Entry Fields

Each log entry stored in VictoriaLogs includes:

Field	Description
`_msg`	Log message text
`_time`	Timestamp (RFC3339Nano)
`host_id`	UUID of the source host
`environment_id`	UUID of the environment
`source`	Origin: `journald` or `file:/path/to/log`
`level`	Severity: emerg, alert, crit, error, warn, notice, info, debug
`unit`	Systemd unit name (journald only)

Reserved Fields and `field.<key>`

With parser: json (or auto on a JSON line), every top-level string key the agent does not use itself is stored as a field of its own. The fields the server sets on every line are reserved: _msg, _time, _stream, _stream_id, host_id, environment_id, source, level and unit. A parsed key with one of those names never replaces the server's value. It is stored as field.<key> instead (vlclient.LogEntry). So an application line {"msg":"…","source":"api","unit":"checkout"} is stored with source=file:/path/to/log and no unit, and its own values are searchable as field.source:="api" and field.unit:="checkout".

This is what keeps host_id, environment_id and source trustworthy. The environment filter that confines an environment-scoped user's log search, the host Logs tab (host_id:) and the AI chat's per-host search all rely on them. Before this rule, a JSON line tailed on one host could set environment_id or host_id and appear as another environment's or host's log.

The logs worker also renames the keys only Console's Kubernetes worker writes — k8s_cluster_*, k8s_event_* and k8s_involved_* — to field.<key> on host logs, and stores a host log whose source is k8s-event as host:k8s-event. See Kubernetes inventory → Events Timeline. k8s_namespace, k8s_pod, k8s_container and k8s_pod_uid (pod-log enrichment) are not renamed.

Lines stored before this rule shipped keep their original fields until they age out.

There is no client_id field

Tenant scoping is carried by the VictoriaLogs AccountID header (the host's vm_account_id), not by a stored field — vlclient.LogEntry emits _msg, _time, host_id, environment_id, source, level, unit plus flattened extras, and nothing else. A LogsQL filter client_id:"…" matches nothing.

Multi-Tenancy

VictoriaLogs supports native multi-tenancy via AccountID HTTP headers:

On insert: The LogsWorker resolves the host's vm_account_id via tenantcache and sets the AccountID header
On query: The logs proxy handler (/api/v1/logs/query and /api/v1/logs/select/*) injects the AccountID header based on the authenticated user's client
Each client's logs are physically isolated -- queries only return data for the authenticated tenant

Agent Configuration

Enable log collection in the agent config:

agent:
  log_collection:
    enabled: true
    journald:
      enabled: true
      units: []           # empty = all units
    files:
      - path: /var/log/syslog
      - path: /var/log/nginx/access.log
        name: nginx-access
        parser: json
    buffer_size: 1000     # entries to buffer before flush
    max_line_len: 8192    # truncate lines longer than this
  log_interval: 10s       # flush interval

Or via environment variables:

PROXIMA_AGENT_LOG_COLLECTION_ENABLED=true
PROXIMA_AGENT_LOG_INTERVAL=10s

Backend API Endpoints

Endpoint	Method	Description
`/api/v1/logs/query`	POST	LogsQL query proxy (accepts JSON body, forwards as URL params to VictoriaLogs)
`/api/v1/logs/select/*`	GET/POST	VictoriaLogs reverse proxy (VMUI static assets + API)

The query endpoint accepts:

{
  "query": "level:error AND unit:nginx.service",
  "start": "2026-03-05T00:00:00Z",
  "end": "2026-03-05T23:59:59Z",
  "limit": 1000
}

Docker Compose Services (Local Dev)

Local development only

This docker-compose stack (VictoriaLogs + Vector) is for local development. In production the backend ships agent logs to VictoriaLogs on vmstorage02 — see the Stack note above.

VictoriaLogs and Vector run in every local environment (no profile gate) — make docker-up sets COMPOSE_PROFILES=dev locally, but both also start on a plain docker compose up -d:

docker compose up -d

Service	Image	Port	Purpose
VictoriaLogs	`victoriametrics/victoria-logs:v1.46.0`	`9428`	Log storage (30d retention)
Vector	`timberio/vector:0.44.0-debian`	--	Collects Docker container logs, ships to VictoriaLogs

Vector Configuration

Local development only

Vector ships local Docker container logs to VictoriaLogs in docker-compose. It is not part of the production Kubernetes deployment, where pod stdout is handled by the cluster's logging stack.

The configuration is at infra/vector/vector.yaml. Pipeline:

docker_logs source -- collects stdout/stderr of every proxima-* container via the Docker API (so co-located CI-runner / other-app containers are not captured)
remap transform (VRL) -- parses Go slog JSON plus regex/logfmt for PostgreSQL, NATS, and Tempo; derives container_name / service_name; normalizes level across all container types
http sink -- POSTs newline-delimited JSON to http://victorialogs:9428/insert/jsonline (tenant 0)

Vector connects to the Docker socket (/var/run/docker.sock, read-only) and ships parsed log entries to VictoriaLogs.

Stream Fields

VictoriaLogs _stream_fields are low-cardinality labels that appear as clickable dropdown filters in the Grafana/VMUI query interface:

Stream Field	Description	Example Values
`container_name`	Docker container name	`proxima-backend`, `proxima-postgres`
`service_name`	Same as container name (matches OTel `service.name` for trace correlation)	`proxima-backend`, `proxima-nats`
`level`	Log severity (normalized to standard slog levels)	`INFO`, `WARN`, `ERROR`, `DEBUG`, `FATAL`
`stream`	Docker output stream	`stdout`, `stderr`

Log Enrichment Pipeline

Enrichment is a single Vector VRL remap transform named normalize (infra/vector/vector.yaml). There is no Fluent Bit, no per-service parser chain, and no enrich.lua.

docker_logs source (container_name, message, timestamp)
  → remap "normalize":
      1. container_name (leading slash stripped); service_name mirrors it
      2. defaults: level = "INFO", msg = raw, time = Docker's collection timestamp
      3. try parse_json — a Go slog line (backend, agent, mcp) is merged whole, so
         level/msg/time/request_id/trace_id/component all lift to the top
      4. otherwise fall through inline regex fallbacks that set .level and .msg
      5. normalize .level through an inline map (LOG/NOTICE→INFO, WARNING→WARN,
         PANIC→FATAL, DETAIL/HINT/STATEMENT→DEBUG, INF/ERR/WRN/DBG/TRC→…)
      6. drop .message and .timestamp
  → VictoriaLogs HTTP sink

Go slog's own INFO/WARN/ERROR/DEBUG already equal their upcased form, so they pass through the level map unchanged; the map exists for PostgreSQL, NATS and Tempo output.

LogsQL Query Examples

# Errors in the last hour (level is a stream field — use := for exact match)
curl -s 'http://localhost:9428/select/logsql/query' \
  -d 'query=_time:1h AND level:="ERROR"' -d 'limit=20'

# Filter by service (stream field)
curl -s 'http://localhost:9428/select/logsql/query' \
  -d 'query=_time:1h AND service_name:="proxima-backend"' -d 'limit=20'

# Cross-signal correlation: find logs by trace_id (from Tempo)
curl -s 'http://localhost:9428/select/logsql/query' \
  -d 'query=_time:1h AND trace_id:="abcdef1234567890"' -d 'limit=50'

# By request_id
curl -s 'http://localhost:9428/select/logsql/query' \
  -d 'query=_time:1h AND request_id:="abc-123"' -d 'limit=50'

# NATS worker errors
curl -s 'http://localhost:9428/select/logsql/query' \
  -d 'query=_time:1h AND component:="worker" AND level:="ERROR"' -d 'limit=20'

# PostgreSQL errors
curl -s 'http://localhost:9428/select/logsql/query' \
  -d 'query=_time:1h AND service_name:="proxima-postgres" AND level:="ERROR"' -d 'limit=20'

# Auth failures
curl -s 'http://localhost:9428/select/logsql/query' \
  -d 'query=_time:1h AND "failed to resolve access"' -d 'limit=20'

# Panics
curl -s 'http://localhost:9428/select/logsql/query' \
  -d 'query=_time:24h AND "panic recovered"' -d 'limit=10'

# Stats: log volume by level
curl -s 'http://localhost:9428/select/logsql/stats_query' \
  -d 'query=_time:1h' -d 'stats=count() by (level)'

# Stats: log volume by service
curl -s 'http://localhost:9428/select/logsql/stats_query' \
  -d 'query=_time:1h' -d 'stats=count() by (service_name)'

Stream fields vs regular fields

Stream fields (service_name, level, container_name, stream) appear as clickable dropdown filters in the Grafana/VMUI UI. All other fields (trace_id, request_id, component, etc.) are auto-indexed by VictoriaLogs and queryable with field:="value" syntax — they just don't appear in dropdown menus.

Log Format

All backend logs use structured JSON via slog:

{
  "time": "2026-02-27T10:00:00.000Z",
  "level": "INFO",
  "msg": "request completed",
  "component": "api",
  "request_id": "abc-123",
  "trace_id": "1234567890abcdef",
  "span_id": "abcdef12",
  "method": "GET",
  "path": "/api/v1/hosts",
  "status": 200,
  "duration_ms": 12.5
}

Stack​

Agent Log Collection Pipeline​

Log Entry Fields​

Reserved Fields and field.<key>​

Multi-Tenancy​

Agent Configuration​

Backend API Endpoints​

Docker Compose Services (Local Dev)​

Vector Configuration​

Stream Fields​

Log Enrichment Pipeline​

LogsQL Query Examples​

Log Format​