Distributed Tracing (Agent)

Proxima can join agent spans and backend spans into a single distributed trace in Tempo, correlated by trace_id. When an operation crosses from the backend into an agent (for example, a backend-dispatched command), you see one end-to-end trace covering both sides instead of two disconnected halves.

This is off by default and enabled per host from the UI.

How it works

Agent spans travel to Tempo over the same NATS channel the agent already uses for metrics, logs, and inventory — the backend acts as the gateway to storage. No OTLP ingress, firewall rule, or new outbound dependency is required on the agent side; its only network requirement stays "reach NATS".

 agent (tracing enabled)
   │  spans → OTLP (raw protobuf)
   ▼  publish on  proxima.{client}.{env}.{agent}.traces
 NATS ─────────────────────────────►  backend trace-ingest worker
                                         │  forward via OTLP/gRPC
                                         ▼
                                       Tempo

The backend forwards agent spans to the same Tempo it sends its own spans to, preserving the original trace_id/span_id/timestamps. Agent spans land under service.name=proxima-agent and join backend spans (service.name=proxima-backend) under one trace.

No OTLP endpoint to expose

Because traces ride the existing NATS channel, remote hosts that can't reach Tempo directly still get their spans into central Tempo. This is less attack surface than a public OTLP collector.

Enabling it for a host

Tracing is controlled per host through the agent's remote config — no agent restart required (the agent hot-reloads the tracer when the config changes).

1. Open the host's agent configuration page in the console.
2. Find the Distributed tracing section.
3. Toggle Enable tracing on.
4. Set the Sample ratio (0–1). The default 0.1 samples 10% of agent-root traces.
5. Save. The new config is pushed to the agent, which switches its tracer on live.

When tracing is disabled (the default), the agent runs a no-op tracer with no exporter and no batching — zero overhead.

Sampling & volume

• The sample ratio applies to agent-root traces. Traces that start at the backend and flow into the agent inherit the backend's sampling decision, so a sampled end-to-end trace stays whole on both sides.
• Span volume travels over NATS. Keep the ratio modest (the 0.1 default is a good starting point) and raise it only when actively investigating; lower it again afterward to keep NATS volume sane.

Environment override (advanced)

For local or OTLP-debugging scenarios you can force tracing on with agent environment variables, which override the UI config:

Variable	Default	Description
PROXIMA_AGENT_OTEL_ENABLED	false	Force NATS span export on.
PROXIMA_AGENT_OTEL_SAMPLE_RATIO	0.1	Head-sampling ratio for agent-root traces ([0,1]).

The preferred control is the per-host UI toggle — reach for the env vars only when you can't use remote config. See Environment Variables.

Viewing traces

Open Grafana Explore with the Tempo datasource and search by trace_id, or follow the cross-signal workflow: find a request_id in the logs, get its trace_id, then open the full span waterfall in Tempo. A joined trace shows both proxima-backend and proxima-agent spans under one trace ID.

For the full catalog of backend span names and attributes, see the OpenTelemetry Tracing Reference.

Verifying the pipeline

After enabling tracing on a host, walk the pipeline from front to back. Each step has a concrete check.

1. Config reached the agent. The agent logs tracing config reloaded with enabled=true (no restart). If you don't see it, the config push didn't land.
2. Spans are forwarding. The backend trace-ingest worker meters every forward — confirm it's non-zero and healthy:

   # backend tenant-0 metrics
   curl -s 'http://localhost:8481/select/0/prometheus/api/v1/query' \
     --data-urlencode 'query=sum by (result)(proxima_trace_ingest_total)'
   # result="ok" climbing = spans are reaching Tempo

3. The agent service is known to Tempo.

   curl -s 'http://localhost:3200/api/v2/search/tag/resource.service.name/values' | jq .
   # expect "proxima-agent" alongside "proxima-backend"

4. A real backend↔agent join. Trigger any backend→agent command other than a config change (a runbook, an on-demand healthcheck, or a terminal session — not the tracing toggle itself; see gotchas). Grab the request's trace_id from the backend log and fetch the trace by ID (fetch-by-ID is authoritative; Tempo search has an ingester→block consistency lag):

   curl -s 'http://localhost:3200/api/traces/<trace_id>' \
     | jq '[.batches[].resource.attributes[]
            | select(.key=="service.name") | .value.stringValue] | unique'
   # → ["proxima-agent","proxima-backend"]  → the agent.command.* span hangs off
   #    the backend handler span under one trace_id. 

Shipping agent self-logs

Alongside its traces, an agent can ship its own operational logs to central VictoriaLogs so you can see what the agent itself is doing — without SSHing to the host. The agent tees every slog record at/above a threshold (WARN and above by default) into the existing NATS logs pipeline; the backend routes those self-log batches to VictoriaLogs tenant 0 (the ops tenant, where backend logs live, not the per-client tenant). Records carry the agent's identity (agent_id, hostname, client_slug, environment_slug) and, when the log was emitted inside a span, a trace_id — so a self-log line joins to the same trace as the operation that produced it.

This is best-effort and non-blocking: local journald stays the source of truth, and the shipper drops (never blocks) under backpressure. So if an agent can't reach NATS, fall back to tsh/kubectl logs on the host.

Enabling it / raising the level

• Per host (standalone agents): use the log_shipping toggle in the agent-config UI — turn shipping on/off and set the level (error / warn / info / debug). It hot-reloads on the agent with no restart. Raise to info or debug while investigating, then lower it again to keep log volume sane.

• k8s agents (node-monitor DaemonSet / cluster agent): these have no remote-config channel, so set it declaratively via the chart's extraEnv:

  nodeAgent:
    extraEnv:
      - name: PROXIMA_AGENT_LOGSHIP_ENABLED
        value: "true"      # default true (ships WARN+)
      - name: PROXIMA_AGENT_LOGSHIP_LEVEL
        value: "info"      # raise to info/debug when investigating

Querying

Self-logs land in tenant 0 with the identity fields flattened to top-level VictoriaLogs keys, so you can filter on them directly with LogsQL:

# All ERROR-or-above self-logs from agents, last 15m
obs.sh logs 'source:self AND level:ERROR'

# Everything one agent has shipped
obs.sh logs 'agent_id:"<agent-uuid>"' 1h

# A specific agent's warnings/errors
obs.sh logs 'agent_id:"<agent-uuid>" AND level:(warn OR error)'

Grab a trace_id off any of these lines and open it in Tempo (or use the cross-signal workflow) to see the agent log next to the spans for the same operation.

Troubleshooting

Symptom	Cause	Fix
Agent-root spans appear, but no joined backend↔agent trace	The command you used to test was the tracing config change itself — config_update reloads the provider mid-dispatch and orphans its own span	Test with a different command (runbook / healthcheck / terminal)
No agent spans at all, even after enabling	Agent predates v0.6.10 — the dispatcher pinned its tracer to the startup NeverSample provider	Upgrade the agent to ≥ v0.6.10
proxima_trace_ingest_total is empty	No spans have been published yet (agents only span on inbound commands + sampling), or the backend trace-ingest worker isn't running (PROXIMA_OTEL_EXPORTER_ENDPOINT unset)	Trigger a command; check the worker started in backend logs
A trace is missing right after a backend deploy	Agent traces ride core NATS (no persistence); spans published while the worker was restarting had no subscriber	Re-trigger after the backend settles — not a permanent failure
Tracing toggle doesn't appear / doesn't apply for k8s node-monitor or collector pods	Those agents aren't linked to a host record, so they receive no remote config	Set PROXIMA_AGENT_OTEL_ENABLED / PROXIMA_AGENT_OTEL_SAMPLE_RATIO via the chart's extraEnv instead

Scheduled vs. command spans

Only inbound command-dispatch spans (agent.command.*) join the backend trace. The agent's scheduled collectors (metrics/inventory/heartbeat) run on a background context and produce agent-root traces — useful on their own, but not stitched to a backend request.