Observability Overview

Proxima Console implements the three pillars of observability: logs, traces, and metrics. All three signals share correlation fields (request_id, trace_id) that allow operators to jump between signals when troubleshooting.

Architecture

                     +-----------+
                     |  Browser  |
                     +-----+-----+
                           |
                     +-----v-----+
                     |   Nginx   |
                     +-----+-----+
                           |
                     +-----v-----+
                     |  Backend  |---> stdout (structured JSON)
                     +--+--+--+--+         |
                        |  |  |            |
              +---------+  |  +------+     |  (Docker JSON log driver)
              |            |         |     |
         +----v----+  +---v---+  +--v--+  |     +------------+
         |   DB    |  | NATS  |  | VM  |  +---->|   Vector   |
         +----+----+  +---+---+  +-----+        +-----+------+
              |            |                           |
              |            |                     +-----v---------+
              |       +----v----+                | VictoriaLogs  |
              |       | Workers |                | :9428         |
              |       +---------+                +---------------+
              |
         (OTel SDK)
              |
         +----v----+
         |  Tempo  |
         |  :3200  |
         +---------+

note

The diagram and the Signal Destinations table below depict the local-dev (Docker Compose) topology. In production (Kubernetes via ArgoCD), pod stdout is handled by the cluster's logging stack and the backend ships agent logs directly to VictoriaLogs — there is no Vector or Docker JSON log driver. See Logs for details.

Signal Destinations

Signal	Source	Destination	Port	Protocol
Logs	Docker stdout (JSON)	VictoriaLogs (via Vector)	9428	HTTP jsonline
Traces	OTel SDK (gRPC)	Tempo	4317	OTLP gRPC
Metrics (backend)	/metrics endpoint	vmagent → VictoriaMetrics	8080	Prometheus scrape
Metrics (agents)	Agent → NATS → Backend	VictoriaMetrics	8480	HTTP import

Signal Map

Component	Logs	Traces	Metrics
Backend HTTP handlers	slog JSON to stdout	OTel spans (otelhttp + manual)	proxima_http_requests_total, proxima_http_request_duration_seconds
Auth middleware	slog JSON (ErrorContext)	auth.validate_token, auth.resolve_access, auth.check_permission	proxima_auth_login_total, proxima_auth_mfa_verification_total, etc.
NATS workers	slog JSON (ErrorContext)	process.registration, process.heartbeat, process.inventory	proxima_nats_messages_total
Credential operations	slog JSON (ErrorContext)	api.CreateCredential, api.TestCredential, vault.transit.*	proxima_credential_operations_total, proxima_credential_encrypt_total, etc.
Config sync	slog JSON (ErrorContext)	configsync.resolve, configsync.resolve_all, configsync.resolve_credentials	proxima_configsync_push_total, proxima_configsync_apply_total, etc.
Healthcheck	slog JSON (ErrorContext)	worker.HealthcheckProcess, healthcheck.score.*, healthcheck.trigger	proxima_healthcheck_runs_processed_total, proxima_healthcheck_score_calculated
Log collection	slog JSON (ErrorContext)	logs.process, logs.query, logs.proxy	proxima_logs_entries_processed_total, proxima_logs_query_total
Change detection	slog JSON (ErrorContext)	worker.ChangesProcess	proxima_changes_processed_total, proxima_file_versions_created_total
Stale detector	slog JSON (ErrorContext)	--	proxima_stale_hosts_marked_offline_total
Agent	slog JSON to stdout	OTel spans	Host system metrics (CPU, memory, disk, network)

Correlation Fields

These fields connect signals across the three pillars:

Field	Where it appears	How to search
request_id	Log fields, OTel span attributes, HTTP X-Request-ID header	LogsQL: "request_id":"abc123" / Grafana Explore (Tempo): tag request_id=abc123
trace_id	Log fields (via TraceHandler), OTel spans	LogsQL: "trace_id":"abc..." / Grafana Explore (Tempo): by trace ID directly
span_id	Log fields, OTel spans	LogsQL: "span_id":"abc..."
correlation_id	NATS message headers	LogsQL: "correlation_id":"abc123"
host_id	Log fields, metric labels, DB records	LogsQL: "host_id":"uuid" / MetricsQL: {host_id="uuid"}

Cross-Signal Correlation Workflow

User report / alert
       |
       v
  Metrics (error rate spike?)
       |
       v
  Logs (request_id → error details + trace_id)
       |
       v
  Grafana/Tempo (trace_id → full span tree)
       |
       v
  Root cause identified

1. Start with a symptom (error alert, user report, or metrics anomaly)
2. Find request_id from logs or the X-Request-ID response header
3. Search logs by request_id to get full request context + trace_id
4. Open the trace in Grafana Explore (Tempo datasource) to see the span waterfall and pinpoint errors
5. Check metrics for broader patterns (error rate spikes, latency increases)

Dev vs Production

In local development:

• When running with go run, logs go to the terminal only, so VictoriaLogs will not capture them
• Only Docker container logs (backend/frontend running in Docker Compose) are captured by the Vector container-log pipeline, which ships them to VictoriaLogs. Vector and the Docker JSON log driver are a local-dev concept only

In production (Kubernetes via ArgoCD):

• The backend ships agent logs (the LogsWorker pipeline) to VictoriaLogs over PROXIMA_VICTORIALOGS_URL
• Pod/container stdout is handled by the cluster's own logging stack — there is no Vector or Docker JSON log driver in production

See Logs for the full production-vs-local-dev breakdown.

Quick Reference

Service	URL	Purpose
Grafana (Tempo)	http://localhost:3004	Trace visualization and search (via Tempo datasource)
VictoriaLogs	http://localhost:9428	Log storage and LogsQL queries
Prometheus metrics	http://localhost:8080/metrics	Backend metrics endpoint
Grafana	http://localhost:3004	Metrics dashboards (Executive Overview + Backend Observability + Agent Resources)
NATS monitoring	http://localhost:8222	JetStream stats, connections