Prometheus Metrics Reference
All backend metrics are created via OpenTelemetry instruments and exported on the /metrics endpoint (port 8080) in Prometheus format. Metrics are defined in backend/internal/observability/metrics.go.
HTTP
| Metric | Type | Labels | Description |
|---|---|---|---|
proxima_http_requests_total | counter | method, path, status | Total HTTP requests |
proxima_http_request_duration_seconds | histogram | method, path | HTTP request duration in seconds |
Auth
| Metric | Type | Labels | Description |
|---|---|---|---|
proxima_auth_login_total | counter | provider, result | Total login attempts (result: success, failed, mfa_required) |
proxima_auth_token_refresh_total | counter | result | Total token refresh attempts (result: success, invalid, revoked, expired) |
proxima_auth_mfa_verification_total | counter | result | Total MFA verification attempts (result: success, failed, recovery_code) |
proxima_auth_session_revocation_total | counter | reason | Total session revocations (reason: logout, disabled) |
proxima_auth_cache_hit_total | counter | — | Total auth cache hits |
proxima_auth_cache_miss_total | counter | — | Total auth cache misses |
proxima_auth_access_resolution_seconds | histogram | — | Access resolution latency in seconds |
proxima_auth_lockout_active | updowncounter | — | Currently locked out accounts |
proxima_auth_password_reset_total | counter | type | Total password resets (type: admin) |
PQL Search
| Metric | Type | Labels | Description |
|---|---|---|---|
proxima_search_query_total | counter | — | Total PQL queries |
proxima_search_query_duration_seconds | histogram | — | End-to-end PQL query latency in seconds |
proxima_search_results_total | counter | — | Total search result rows returned |
NATS Workers
| Metric | Type | Labels | Description |
|---|---|---|---|
proxima_nats_messages_received_total | counter | type | Total NATS messages received |
proxima_nats_messages_processed_total | counter | type, status | Total NATS messages processed |
proxima_nats_message_processing_duration_seconds | histogram | subject_class, status | Per-message processing time in seconds |
VictoriaMetrics Client
| Metric | Type | Labels | Description |
|---|---|---|---|
proxima_vm_operation_total | counter | — | Total VictoriaMetrics API calls |
proxima_vm_operation_duration_seconds | histogram | — | VictoriaMetrics API call latency in seconds |
Tenant Cache
| Metric | Type | Labels | Description |
|---|---|---|---|
proxima_tenantcache_lookup_total | counter | — | Tenant cache hit/miss counter |
proxima_tenantcache_lookup_duration_seconds | histogram | — | Tenant cache lookup latency in seconds |
Stale Detector
| Metric | Type | Labels | Description |
|---|---|---|---|
proxima_stale_hosts_marked_offline_total | counter | — | Cumulative hosts marked offline by stale detector |
Panic Recovery
| Metric | Type | Labels | Description |
|---|---|---|---|
proxima_api_panic_recovery_total | counter | — | Recovered panics in HTTP handlers |
Change Detection
| Metric | Type | Labels | Description |
|---|---|---|---|
proxima_changes_processed_total | counter | scan_type, event_type | Total file change events processed |
proxima_changes_processing_duration_seconds | histogram | — | Change detection message processing duration in seconds |
proxima_file_versions_created_total | counter | — | Total file versions created |
Log Collection
| Metric | Type | Labels | Description |
|---|---|---|---|
proxima_logs_entries_processed_total | counter | source | Total log entries processed from agents |
proxima_logs_processing_duration_seconds | histogram | — | Log message processing duration in seconds |
proxima_logs_query_total | counter | — | Total log queries from frontend |
proxima_logs_query_duration_seconds | histogram | — | Log query latency in seconds |
NATS JWT Lifecycle
| Metric | Type | Labels | Description |
|---|---|---|---|
proxima_install_token_created_total | counter | — | Total install tokens created |
proxima_jwt_issued_total | counter | — | Total NATS user JWTs issued |
proxima_jwt_renewed_total | counter | method | Total NATS user JWTs renewed (method: nats, https) |
proxima_jwt_revoked_total | counter | — | Total NATS user JWTs revoked |
proxima_jwt_credential_challenge_total | counter | status | Total credential challenge requests |
proxima_jwt_credential_exchange_total | counter | status | Total credential exchange requests |
Healthcheck
| Metric | Type | Labels | Description |
|---|---|---|---|
proxima_healthcheck_runs_processed_total | counter | scanner, status | Total healthcheck runs processed |
proxima_healthcheck_results_processed_total | counter | status, source | Total healthcheck results processed |
proxima_healthcheck_findings_processed_total | counter | scanner, severity, type | Total scanner findings processed |
proxima_healthcheck_processing_duration_seconds | histogram | — | Healthcheck processing duration in seconds |
proxima_healthcheck_trigger_total | counter | — | Total healthcheck triggers via API |
proxima_healthcheck_score_calculated | histogram | — | Distribution of calculated healthcheck scores (0–100) |
Compliance
| Metric | Type | Labels | Description |
|---|---|---|---|
proxima_compliance_evaluations_total | counter | framework, status | Total control evaluations performed |
proxima_compliance_evaluation_duration_seconds | histogram | — | Time to evaluate all controls for one host |
proxima_compliance_controls_total | updowncounter | — | Size of the control library: +1 on create, −1 on delete. Not a status distribution — it carries no status attribute. |
Alerting
| Metric | Type | Labels | Description |
|---|---|---|---|
proxima_alerts_resolved_total | counter | — | Total alert groups resolved |
proxima_alert_resolve_no_open_group_total | counter | source_type | Resolve-only deliveries dropped because no open alert group exists for their (source, grouping key). A resolve never creates a group — see A resolution never creates an alert group |
Telegram Alert Card
Collected only: no alert rule ships for either. See When edits fail: what operators see.
| Metric | Type | Labels | Description |
|---|---|---|---|
proxima_telegram_card_edits_total | counter | result (ok, not_modified, transient, permanent, skipped_known_failure) | Outcomes of the alert card Syncer's edits, one per tracked message a run tried to bring up to date. transient is retried (up to 3 retries per burst); permanent is not retried by the run; skipped_known_failure made no call because the same edit already failed permanently within the hour. A message already showing its group's state is not counted |
proxima_telegram_card_edit_retries_exhausted_total | counter | — | Bursts of card sync runs for one alert group that still needed a retry after 3 retries; the group's messages wait for its next sync or a reconciler sweep |
Credentials
| Metric | Type | Labels | Description |
|---|---|---|---|
proxima_credential_operations_total | counter | operation, status | Total credential CRUD operations (operation: create, list, get, update, delete, test) |
proxima_credential_operation_duration_seconds | histogram | — | Credential operation duration |
proxima_credential_encrypt_total | counter | credential_type | Total Vault Transit encrypt operations |
proxima_credential_decrypt_total | counter | credential_id, credential_type | Total Vault Transit decrypt operations |
proxima_credential_test_total | counter | credential_type, success | Total credential connectivity tests |
proxima_credential_rotation_total | counter | — | Total credential secret rotations |
Config Sync
| Metric | Type | Labels | Description |
|---|---|---|---|
proxima_configsync_push_total | counter | — | Total config push operations to agents |
proxima_configsync_push_duration_seconds | histogram | — | Config push duration |
proxima_configsync_apply_total | counter | config_type, status | Total config apply acks from agents |
proxima_configsync_resolve_total | counter | config_type | Total config resolution operations |
proxima_configsync_resolve_duration_seconds | histogram | — | Config resolution duration |
Database
| Metric | Type | Labels | Description |
|---|---|---|---|
proxima_db_connections_open | gauge (observable) | — | Number of open database connections |
proxima_db_connections_idle | gauge (observable) | — | Number of idle database connections |
Business
| Metric | Type | Labels | Description |
|---|---|---|---|
proxima_agents_online_total | gauge (observable) | — | Number of agents currently online |
proxima_hosts_total | gauge (observable) | — | Total number of hosts |
Terminal
| Metric | Type | Labels | Description |
|---|---|---|---|
proxima_terminal_sessions_total | counter | login, role | Total sessions started |
proxima_terminal_sessions_active | updowncounter | — | Currently active sessions |
proxima_terminal_session_duration_seconds | histogram | end_reason | Session duration |
proxima_terminal_auth_denied_total | counter | login, role | RBAC denials |
proxima_terminal_errors_total | counter | reason | Session errors |
proxima_terminal_peers_active | updowncounter | — | Currently connected session peers (session sharing) |
proxima_terminal_peer_joins_total | counter | — | Total peer join events |
proxima_terminal_file_transfer_total | counter | direction, status | Total terminal file transfers |
proxima_terminal_file_transfer_bytes_total | counter | direction | Total bytes transferred via terminal file transfer |
proxima_terminal_file_transfer_duration_seconds | histogram | direction | Terminal file transfer duration in seconds |
Fleet
| Metric | Type | Labels | Description |
|---|---|---|---|
proxima_fleet_rollouts_total | counter | status (created/paused/completed/aborted) | Rollout lifecycle transitions |
proxima_fleet_rollouts_active | gauge | — | Currently active rollouts |
proxima_fleet_agent_updates_total | counter | status (sent/healthy/failed/timed_out/skipped_offline) | Per-agent update transitions |
proxima_fleet_update_duration_seconds | histogram | — | Rollout duration from start to terminal status |
proxima_fleet_auth_denied_total | counter | — | Tenant/scope denials on rollout requests |
AI Chat & MCP
| Metric | Type | Labels | Description |
|---|---|---|---|
proxima_chat_messages_total | counter | role | Total chat messages |
proxima_chat_tokens_total | counter | model, direction (input/output) | Total tokens consumed |
proxima_chat_cost_usd_total | counter | model | Estimated AI cost in USD |
proxima_chat_tool_calls_total | counter | tool | Total MCP tool calls from chat |
proxima_chat_tool_duration_seconds | histogram | tool | Tool call duration |
proxima_chat_response_duration_seconds | histogram | — | Full chat response duration |
proxima_chat_model_routing_total | counter | tier, reason | Model routing decisions |
proxima_chat_conversations_active | updowncounter | — | Active conversations |
proxima_chat_errors_total | counter | error_type | Chat errors |
proxima_chat_credential_source_total | counter | source | AI credential source usage |
L1 Incident Agent
| Metric | Type | Labels | Description |
|---|---|---|---|
proxima_l1_triage_tokens_total | counter | — | Tokens consumed by unattended L1 triage runs |
proxima_l1_triage_cost_usd_total | counter | — | LLM cost in USD for unattended L1 triage runs |
proxima_l1_engage_tokens_total | counter | — | Tokens consumed by reactive L1 engage (listening-bot) runs |
proxima_l1_engage_cost_usd_total | counter | — | LLM cost in USD for reactive L1 engage runs |
proxima_escalation_steps_total | counter | client_id | Escalation groups advanced one step by the timer worker |
proxima_escalation_drift_total | counter | — | Shadow-poll on-call drift events |
Runbook Execution
| Metric | Type | Labels | Description |
|---|---|---|---|
proxima_runbook_executions_total | counter | status | Runbook executions that reached a terminal state |
proxima_runbook_dispatched_total | counter | trigger_source | Runbook executions dispatched to agents |
proxima_runbook_executions_timed_out_total | counter | — | Executions force-failed by the watchdog for exceeding their timeout |
proxima_runbook_watchdog_errors_total | counter | — | Watchdog DB errors while sweeping for stale executions |
Trace Ingest
| Metric | Type | Labels | Description |
|---|---|---|---|
proxima_trace_ingest_total | counter | result (ok/malformed/forward_error) | Agent trace batches ingested |
proxima_trace_ingest_spans_total | counter | — | Agent spans forwarded to Tempo |
Agent
| Metric | Type | Labels | Description |
|---|---|---|---|
proxima_agent_selflogs_ingested_total | counter | result (ok/error) | Agent self-log entries forwarded to the ops VictoriaLogs tenant (account 0) |
agent_healthcheck_last_findings | gauge | agent_version, hostname | Findings count from last scan |
All metrics follow OpenMetrics naming conventions with the proxima_ prefix for backend metrics and agent_ prefix for agent-side metrics.
note
This reference covers the primary subsystems. The backend registers additional metrics for other subsystems (JSM, OIDC, ClickHouse, email, Kubernetes sync, service desk, tunnel/transport, briefing, and more). The authoritative list is always the /metrics endpoint and the backend/internal/observability/metrics_*.go registration files.