OpenTelemetry Tracing Reference

All backend spans are created via the OpenTelemetry SDK using otel.Tracer(observability.TracerName). Traces are exported to Tempo via OTLP gRPC (port 4317) and can be visualized in Grafana Explore using the Tempo datasource.

Auth Middleware

Span	Attributes	Source
auth.validate_token	auth.valid	api/auth_middleware.go
auth.resolve_access	auth.subject_id, auth.subject_type, auth.is_super_admin	api/auth_middleware.go
auth.check_permission	auth.permission_checked, auth.subject_id, auth.subject_type, auth.is_super_admin, auth.permitted, auth.client_id	api/auth_middleware.go

HTTP Middleware

Span	Attributes	Source
otelhttp (auto-instrumented)	request_id, method, path, status	observability/middleware.go

Credential API

Span	Attributes	Source
api.CreateCredential	—	api/credential_handler.go
api.ListCredentials	—	api/credential_handler.go
api.GetCredential	—	api/credential_handler.go
api.UpdateCredential	—	api/credential_handler.go
api.DeleteCredential	—	api/credential_handler.go
api.TestCredential	—	api/credential_handler.go

Healthcheck API

Span	Attributes	Source
healthcheck.score.host	—	api/healthcheck_handler.go
healthcheck.score.environment	—	api/healthcheck_handler.go
healthcheck.score.client	—	api/healthcheck_handler.go
healthcheck.trigger	—	api/healthcheck_handler.go

Metric API

Span	Attributes	Source
api.ListMetricNames	proxima.host_id	api/metric_handler.go
api.QueryMetrics	proxima.host_id, proxima.metric_name, proxima.derivative	api/metric_handler.go
api.ListMetricSeries	proxima.host_id, proxima.metric_name	api/metric_handler.go
api.QueryLatestMetrics	proxima.host_count, proxima.metric_count	api/metric_handler.go
api.QueryProcessMetrics	proxima.host_id, proxima.pid, proxima.process_name	api/metric_handler.go
api.QueryContainerMetrics	proxima.host_id, proxima.container_id	api/metric_handler.go
api.QueryContainerLatestMetrics	proxima.host_id	api/metric_handler.go

Aggregate Metric API

Span	Attributes	Source
api.ListMetricNamesByEnvironment	—	api/aggregate_metric_handler.go
api.QueryMetricsByEnvironment	—	api/aggregate_metric_handler.go
api.ListMetricSeriesByEnvironment	—	api/aggregate_metric_handler.go
api.ListMetricNamesByClient	—	api/aggregate_metric_handler.go
api.QueryMetricsByClient	—	api/aggregate_metric_handler.go
api.ListMetricSeriesByClient	—	api/aggregate_metric_handler.go

Search API

Span	Attributes	Source
api.Search	—	api/search_handler.go
api.SearchCount	—	api/search_handler.go
api.SearchFields	—	api/search_handler.go
api.SearchValues	—	api/search_handler.go
api.SearchValidate	—	api/search_handler.go
api.SearchAggregate	—	api/search_handler.go

Logs API

Span	Attributes	Source
logs.query	—	api/logs_handler.go
logs.proxy	—	api/logs_handler.go

Saved Search API

Span	Attributes	Source
api.SavedSearchList	—	api/saved_search_handler.go
api.SavedSearchCreate	—	api/saved_search_handler.go
api.SavedSearchGetByID	—	api/saved_search_handler.go
api.SavedSearchUpdate	—	api/saved_search_handler.go
api.SavedSearchDelete	—	api/saved_search_handler.go

NATS Workers

The span name is process.<envelope type>

Consumer spans are built as "process." + envelope.Type, so the span set is exactly the set of envelope types each consumer dispatches. There is no process.registration, process.telemetry or process.heartbeat_v2 — those name a consumer or a stream, not an envelope type, and searching Tempo for them returns nothing.

Span	Attributes	Source
process.inventory	messaging.system, messaging.subject, proxima.correlation_id, proxima.agent_id, proxima.message_type	worker/events.go
process.changes	same	worker/events.go
process.healthcheck	same	worker/events.go
process.kubernetes	same	worker/events.go
process.connections	same	worker/events.go
process.metrics	messaging.system, messaging.subject, proxima.agent_id	worker/telemetry.go
process.processes	same	worker/telemetry.go
process.logs	same	worker/telemetry.go
process.heartbeat	messaging.system, messaging.subject, proxima.agent_id	worker/heartbeat_consumer.go
process.config_fetch	messaging.system, messaging.subject, proxima.host_id	worker/config_fetch.go
process.config_apply	messaging.system, messaging.subject, proxima.host_id, proxima.config_type, proxima.status, proxima.version	worker/config_apply.go

Worker Processing

Span	Attributes	Source
worker.HeartbeatProcess	proxima.host_id, proxima.collectors_count, proxima.scanners_count	worker/heartbeat.go
worker.ProcessesProcess	proxima.host_id, proxima.process_count	worker/processes.go
worker.MetricsProcess	proxima.host_id, vm.account_id, vm.metric_count	worker/metrics.go
worker.InventoryProcess	—	worker/inventory.go
worker.ChangesProcess	proxima.host_id, proxima.scan_type, proxima.file_count, proxima.batch_seq, proxima.batch_total	worker/changes.go
worker.HealthcheckProcess	proxima.host_id, proxima.run_id, proxima.scanner, proxima.environment_id, proxima.client_id, proxima.total_checks, proxima.passed, proxima.failed, proxima.findings_count	worker/healthcheck.go
logs.process	proxima.agent_id, vl.account_id, vl.entry_count, vl.self_logs	worker/logs_worker.go

Compliance Worker

Span	Attributes	Source
process.compliance.evaluate	messaging.system, messaging.subject, proxima.host_id, proxima.run_id, proxima.environment_id, proxima.client_id	worker/compliance_consumer.go
worker.ComplianceEvaluate	proxima.host_id, proxima.run_id, proxima.environment_id, proxima.client_id, proxima.framework.{slug}.controls, proxima.framework.{slug}.evaluations	worker/compliance.go

Config Sync

Span	Attributes	Source
configsync.resolve	proxima.config_type, proxima.host_id, proxima.environment_id, proxima.source	agentconfig/resolver.go
configsync.resolve_all	proxima.host_id, proxima.environment_id, proxima.config_count	agentconfig/resolver.go
configsync.resolve_credentials	proxima.config_type, proxima.client_id, credential_id, credential_type	agentconfig/credential_resolver.go

Search Engine

Span	Attributes	Source
search.Compile	proxima.query	search/search.go

VictoriaMetrics Client

Span	Attributes	Source
vmclient.Import	—	vmclient/client.go
vmclient.QueryRange	—	vmclient/client.go
vmclient.Query	—	vmclient/client.go
vmclient.LabelValues	—	vmclient/client.go
vmclient.Health	—	vmclient/client.go

VictoriaLogs Client

Span	Attributes	Source
vlclient.Insert	—	vlclient/client.go
vlclient.Health	—	vlclient/client.go

Vault Transit

Span	Attributes	Source
vault.transit.encrypt	—	credential/vault_transit.go
vault.transit.decrypt	—	credential/vault_transit.go

Store

Span	Attributes	Source
store.SavedSearchList	—	store/saved_search.go
store.SavedSearchGetByID	—	store/saved_search.go
store.SavedSearchCreate	—	store/saved_search.go
store.SavedSearchUpdate	—	store/saved_search.go
store.SavedSearchDelete	—	store/saved_search.go
store.SearchEntityHosts	—	store/search.go
store.SearchEntity	—	store/search.go
store.CountEntity	—	store/search.go
store.SearchAggregate	—	store/search.go
store.SearchFieldValues	—	store/search.go

AI Chat & MCP

Span	Attributes	Source
chat.llm_call	model	chat/llm_client.go
chat.model_route.llm	message_preview	chat/router.go
chat.context_prefetch	—	chat/prefetch.go
chat.encryption.encrypt	—	chat/encryption.go
chat.encryption.decrypt	—	chat/encryption.go
chat.encryption.dek_resolve	—	chat/encryption.go
chat.encryption.vault_encrypt	—	chat/encryption.go
chat.encryption.vault_decrypt	—	chat/encryption.go

Interactive Terminal (gRPC)

Span	Attributes	Source
TerminalGRPC.Session	hostname, login, ...	grpcserver/terminal.go
TerminalGRPC.JoinSession	session_id	grpcserver/terminal.go
TerminalGRPC.PortForward	user_id	grpcserver/tunnel.go
TerminalGRPC.ProxySSH	user_id	grpcserver/proxyssh.go

L1 Incident Agent & Alerts

Span	Attributes	Source
process.escalation.tick	proxima.escalation.planned	worker/escalation_timer_worker.go
process.memory.draft	messaging.system, messaging.subject, proxima.alert_group_id	worker/memory.go
process.alerts.ingest	messaging.system, messaging.subject	worker/alert.go
process.alerts.triage	messaging.system, messaging.subject, proxima.alert_group_id	worker/triage.go
process.alerts.correlate	messaging.system, messaging.subject	worker/correlation.go

Fleet

Span	Attributes	Source
fleet.worker_tick	—	worker/fleet_update_worker.go

Service Desk & JSM

Span	Attributes	Source
api.ServiceDesk.GetTicket.SLA	—	api/service_desk_ticket_handler.go
api.ServiceDesk.GetTicket.Delivery	—	api/service_desk_ticket_handler.go
api.ServiceDesk.GetTicket.StatusHistory	—	api/service_desk_ticket_handler.go
jsm.add_attachment	—	jsmclient/client.go
jsm.proxy_download	—	jsmclient/client.go

Briefing

Span	Attributes	Source
briefing.Assemble	—	service/briefing_service.go
briefing.<source>	(per-source child spans)	service/briefing_service.go

OIDC

Span	Attributes	Source
auth.oidc_exchange	(client span)	auth/oidc.go
auth.oidc_verify	(client span)	auth/oidc.go

---

All span names follow the subsystem.operation naming convention. Attributes use the proxima. prefix for domain-specific values and OpenTelemetry semantic conventions (e.g., messaging.system) for infrastructure attributes.

note

This page lists the primary span subsystems. The backend instruments additional spans (e.g. Kubernetes sync, ClickHouse, email) and the agent emits its own spans that are forwarded to Tempo via the trace-ingest worker — see Agent Tracing. The authoritative source is always the *.Start(ctx, "...") calls in the backend and agent code.