Grafana Dashboards

Proxima Console ships a set of pre-built Grafana dashboards provisioned automatically via infra/grafana/provisioning/dashboards/default.yml. They are split across two folders: the Proxima folder (Executive Overview, Backend Observability, Agent Resource Usage, MCP Server Observability, System Health) and the Infrastructure folder (NATS, nginx, PostgreSQL, Valkey, VictoriaMetrics Cluster, vmagent, and an infrastructure executive overview). Most dashboards auto-refresh every 30 seconds.

Executive Overview

File: infra/grafana/dashboards/executive-overview.json Datasources: victoriametrics-backend (account 0) + victoriametrics (account 1+)

A single-screen "is the platform healthy?" dashboard combining backend and agent fleet signals. Designed for NOC wall displays and executive visibility. Uses large stat panels with traffic-light background colors so status is readable at a glance. Dashboard-level links provide drill-down to Backend Observability, Agent Resources, and the Metrics Reference docs page.

Structure

Row	Panels	Purpose
Platform Health	6 stat panels	Fleet online %, request success rate %, P95 latency, total hosts, active lockouts, panic recoveries
Request Traffic	2 timeseries	Request rate (success/4xx/5xx stacked area), latency percentiles (p50/p95/p99 with threshold bands)
Agent Fleet	4 stat panels	Agents online, avg agent CPU %, avg agent RSS memory, NATS reconnects/min
Host Infrastructure	3 gauges + 1 bar chart	Avg host CPU gauge, avg host memory gauge, max host disk gauge, stale hosts over time
Operations	4 stat panels	Config pushes/min, credential ops/min, avg healthcheck score, NATS message rate

Thresholds

Metric	Green	Orange	Red
Fleet Online	≥95%	≥80%	<80%
Request Success Rate	≥99%	≥95%	<95%
P95 Latency	<500ms	<2s	≥2s
Avg Agent CPU	<50%	<80%	≥80%
Avg Agent Memory	<256 MB	<512 MB	≥512 MB
NATS Reconnects	0	>0	>5/min
Avg Host CPU	<70%	<90%	≥90%
Avg Host Memory	<80%	<95%	≥95%
Max Host Disk	<80%	<90%	≥90%
Active Lockouts	0	≥1	≥5
Panics	0	—	≥1
Healthcheck Score	≥80	≥50	<50

Backend Observability

File: infra/grafana/dashboards/backend-observability.json Datasource: victoriametrics-backend (VictoriaMetrics account 0)

The backend dashboard follows two methodologies:

• RED (Rate, Errors, Duration) for request-driven subsystems (HTTP, Auth, NATS, VM client, PQL search, change detection, healthcheck, credentials, config sync)
• USE (Utilization, Saturation, Errors) for infrastructure resources (DB connections, tenant cache)

Structure

Row	Panels	Purpose
Golden Signals	6 stat panels	Top-of-dashboard summary: request rate, error rate %, P95 latency, agents online, total hosts, active lockouts
HTTP RED	3 timeseries	Request rate by status, latency percentiles (p50/p95/p99), error rate by method
Auth RED	3 timeseries	Login rate (success/failure), auth operations rate (token refresh, MFA, password reset, session revocation), access resolution latency
Auth Cache	1 timeseries + 1 stat	Cache hit/miss rate, cache hit ratio
NATS Workers RED	2 timeseries + 1 stat	Message rate by type, processing duration P95, error rate
NATS JWT Lifecycle	3 timeseries	Install tokens created, JWT operations (issued/renewed/revoked), credential flow (challenge/exchange)
VictoriaMetrics Client RED	2 timeseries + 1 stat	Operation rate, latency P95, error rate
PQL Search RED	1 stat + 2 timeseries	Search QPS, latency P95, results rate
Change Detection RED	2 timeseries + 2 stat + 1 timeseries	Event rate by type, processing duration, batch reassemblies, file versions, changes by scan type
Healthcheck RED	2 timeseries + 1 stat + 1 timeseries	Runs/results rate, processing duration P50/P95/P99, score distribution, API trigger rate
Credentials RED	3 timeseries + 3 stat	Operation rate by type, operation duration P50/P95/P99, Vault Transit encrypt/decrypt rate, error rate %, test rate, rotation rate
Config Sync RED	3 timeseries + 2 timeseries	Push rate (push/apply), push duration P50/P95/P99, resolution rate by config type, resolution duration P50/P95, apply status by outcome
Infrastructure USE	2 timeseries + 1 stat + 1 timeseries	DB connections (open/idle), tenant cache hit/miss rate, tenant cache hit ratio, tenant cache lookup latency
System Health	1 stat + 1 timeseries	Panic recoveries, stale hosts marked offline

Thresholds

Latency panels include visual threshold lines:

Panel	Warning	Critical
HTTP Request Latency	500ms	2s
Access Resolution Latency	100ms	500ms
NATS Processing Duration	1s	5s
VM Latency	500ms	2s
Search Latency	200ms	1s
Change Detection Duration	2s	10s
Healthcheck Processing Duration	1s	5s
Credential Operation Duration	500ms	2s
Config Push Duration	1s	5s
Config Resolution Duration	500ms	2s
Tenant Cache Lookup Latency	10ms	100ms
DB Connections	20 open	45 open

Agent Resource Usage

File: infra/grafana/dashboards/agent-resources.json Datasource: victoriametrics (VictoriaMetrics account 1+) Template variables: $version (agent version), $host (hostname)

Monitors agent self-monitoring metrics for resource usage and cross-version regression detection.

Structure

Row	Panels	Purpose
Overview	6 stat panels	RSS memory, CPU usage, goroutines, open FDs, uptime, OS threads
Memory (OS Process)	2 timeseries	RSS and VMS memory by version
Memory (Go Runtime)	2 timeseries	Heap/stack/sys breakdown, heap objects
CPU	2 timeseries	CPU usage by version, context switches (voluntary/involuntary)
Disk I/O	1 timeseries	Read/write bytes rate
NATS Transport	5 timeseries	Throughput (bytes/s), messages (rate/s), RTT, outbox queue depth, reconnect rate
GC & Runtime	4 timeseries	GC pause duration, GC cycles, goroutines by version, open FDs by version
Healthcheck	3 timeseries	Scan rate, last scan duration, passed/failed results (stacked, color-coded)
Config Cache	2 timeseries	Cached config count, config cache version
Version Comparison	1 table	Side-by-side avg/max RSS, CPU, goroutines, GC pause per version

MCP Server Observability

File: infra/grafana/dashboards/mcp-observability.json Datasource: victoriametrics-backend (VictoriaMetrics account 0)

Monitors the MCP server / AI chat tool-calling layer. Golden-signal stat panels (tool call rate, tool success rate, P95 tool latency, auth cache hit rate, auth errors, backend error rate) sit above per-tool breakdowns (call rate, latency p95, errors, and top tools by total calls), an auth-cache section, and a backend-proxy section (request rate by status, latency, requests by path). Dashboard links cross-navigate to Executive Overview and Backend Observability.

System Health

File: infra/grafana/dashboards/system-health.json Datasource: victoriametrics-backend (VictoriaMetrics account 0)

An "is the system stable?" dashboard oriented around stability signals. Sections cover an at-a-glance summary (API error rate, API p99 latency, dead-lettered messages, agents online, DB pool utilization, dependency errors), HTTP (request rate by status, latency percentiles, 5xx error rate), the async NATS pipeline (publish rate by result, received vs processed, dead-letters and rejections), dependencies (email, OIDC/JSM/ClickHouse rate by status, dependency latency p95), and saturation/domain (DB connections, runbook executions by status).

Infrastructure Dashboards

Folder: infra/grafana/dashboards/infrastructure/ (provisioned into the Infrastructure Grafana folder)

Per-component dashboards for the backing services and edge: nats.json, nginx.json, postgresql.json, valkey.json, victoriametrics-cluster.json, vmagent.json, plus infra-executive.json (a roll-up executive view across the infrastructure components).

Accessing Grafana

Environment	URL
Local dev	http://localhost:3004
Production	Behind nginx with basic auth

Dashboards are provisioned on startup and automatically reloaded when JSON files change (on container restart).