Metrics Collected

The agent collects system metrics via gopsutil v4 and Linux /proc files, plus plugin metrics from configured integrations. All metrics follow the OpenMetrics naming convention.

Raw values: The agent emits raw counters and gauges. The derived ratio metrics the UI shows (cpu_usage_ratio, memory_used_ratio, disk_used_ratio, …) are computed at query time in the backend from these raw values, scoped to the caller's tenant — see Derived metrics.

System Metrics

Collected automatically on every managed host. No configuration required.

CPU

Metric	Type	Unit	Description
cpu_user_seconds_total	Counter	seconds	Time spent in user mode
cpu_system_seconds_total	Counter	seconds	Time spent in kernel mode
cpu_nice_seconds_total	Counter	seconds	Time spent in low-priority user mode
cpu_idle_seconds_total	Counter	seconds	Time spent idle
cpu_iowait_seconds_total	Counter	seconds	Time spent waiting for I/O
cpu_irq_seconds_total	Counter	seconds	Time spent servicing hardware interrupts
cpu_softirq_seconds_total	Counter	seconds	Time spent servicing software interrupts
cpu_steal_seconds_total	Counter	seconds	Time stolen by hypervisor
cpu_load1	Gauge	—	1-minute load average
cpu_load5	Gauge	—	5-minute load average
cpu_load15	Gauge	—	15-minute load average

Derived (computed at query time)

cpu_usage_ratio and cpu_iowait_ratio are computed at query time in the backend from the per-mode counters above. They produce the same 0.0–1.0 ratios the frontend displays.

Memory

Metric	Type	Unit	Description
memory_total_bytes	Gauge	bytes	Total physical memory
memory_used_bytes	Gauge	bytes	Memory in use (excluding buffers/cache)
memory_available_bytes	Gauge	bytes	Memory available without swapping
memory_buffers_bytes	Gauge	bytes	Kernel buffer cache
memory_cached_bytes	Gauge	bytes	Page cache (reclaimable)
memory_swap_total_bytes	Gauge	bytes	Total swap space
memory_swap_free_bytes	Gauge	bytes	Free swap space

Derived (computed at query time)

memory_used_ratio and memory_swap_used_ratio are computed at query time in the backend from raw bytes.

Disk

Per-partition and per-device metrics. Pseudo-filesystems (tmpfs, sysfs, proc, overlay) are excluded.

Usage (per partition):

Metric	Type	Unit	Labels	Description
disk_total_bytes	Gauge	bytes	mount, device	Total partition size
disk_used_bytes	Gauge	bytes	mount, device	Used space
disk_free_bytes	Gauge	bytes	mount, device	Free space
disk_inodes_total	Gauge	—	mount, device	Total inodes
disk_inodes_free	Gauge	—	mount, device	Free inodes

I/O (per device):

Metric	Type	Unit	Labels	Description
disk_read_bytes_total	Counter	bytes	device	Total bytes read
disk_written_bytes_total	Counter	bytes	device	Total bytes written
disk_reads_total	Counter	—	device	Total read operations
disk_writes_total	Counter	—	device	Total write operations
disk_io_time_ms_total	Counter	ms	device	Time spent doing I/O
disk_io_weighted_ms_total	Counter	ms	device	Weighted I/O time (saturation signal)
disk_io_in_progress	Gauge	—	device	I/O operations currently in flight
disk_read_time_ms_total	Counter	ms	device	Total read latency
disk_write_time_ms_total	Counter	ms	device	Total write latency

Derived (computed at query time)

disk_used_ratio and disk_inodes_used_ratio are computed at query time in the backend from raw values.

Disk saturation

disk_io_in_progress > 0 consistently indicates disk saturation. disk_io_weighted_ms_total is the USE method's saturation signal for storage.

Network

Per-interface metrics. The loopback interface (lo) is excluded.

Metric	Type	Unit	Labels	Description
network_receive_bytes_total	Counter	bytes	interface	Total bytes received
network_transmit_bytes_total	Counter	bytes	interface	Total bytes transmitted
network_receive_packets_total	Counter	—	interface	Total packets received
network_transmit_packets_total	Counter	—	interface	Total packets transmitted
network_receive_errors_total	Counter	—	interface	Total receive errors
network_transmit_errors_total	Counter	—	interface	Total transmit errors
network_receive_drops_total	Counter	—	interface	Total receive drops (buffer exhaustion)
network_transmit_drops_total	Counter	—	interface	Total transmit drops

Pressure Stall Information (PSI)

Available on Linux kernel 4.20+. Skipped silently on older kernels.

Metric	Type	Description
psi_cpu_some_avg10	Gauge	% of time tasks stalled on CPU (10s avg)
psi_cpu_some_avg60	Gauge	% of time tasks stalled on CPU (60s avg)
psi_cpu_some_total_microseconds	Counter	Total CPU stall time
psi_memory_some_avg10	Gauge	% of time some tasks stalled on memory (10s avg)
psi_memory_some_avg60	Gauge	% of time some tasks stalled on memory (60s avg)
psi_memory_full_avg10	Gauge	% of time all tasks stalled on memory (10s avg)
psi_memory_full_avg60	Gauge	% of time all tasks stalled on memory (60s avg)
psi_io_some_avg10	Gauge	% of time some tasks stalled on I/O (10s avg)
psi_io_some_avg60	Gauge	% of time some tasks stalled on I/O (60s avg)
psi_io_full_avg10	Gauge	% of time all tasks stalled on I/O (10s avg)
psi_io_full_avg60	Gauge	% of time all tasks stalled on I/O (60s avg)

Root cause analysis

PSI is the single most valuable metric for automated root cause analysis. Unlike CPU usage (which can be 100% and healthy), PSI measures actual stall time — when tasks are blocked waiting for resources.

Kernel & System State

Metric	Type	Description
vmstat_oom_kill_total	Counter	OOM kill events (from /proc/vmstat)
vmstat_pgmajfault_total	Counter	Major page faults (memory pressure → disk I/O)
vmstat_pswpin_total	Counter	Pages swapped in
vmstat_pswpout_total	Counter	Pages swapped out
context_switches_total	Counter	System-wide context switches
processes_created_total	Counter	Total processes created (fork rate)
processes_running	Gauge	Processes currently running
processes_blocked	Gauge	Processes blocked on I/O
entropy_available	Gauge	Available entropy (always 256 on kernel 5.4+)

TCP & Resource Limits

Metric	Type	Description
sockstat_tcp_inuse	Gauge	TCP sockets in use
sockstat_tcp_orphan	Gauge	Orphaned TCP sockets (no process)
sockstat_tcp_tw	Gauge	TCP sockets in TIME_WAIT state
sockstat_sockets_used	Gauge	Total sockets in use
filefd_allocated	Gauge	System-wide allocated file descriptors
filefd_max	Gauge	System-wide file descriptor limit
conntrack_entries	Gauge	Conntrack table entries (skipped if netfilter not loaded)

Process Metrics

Top-N process metrics are exported to VictoriaMetrics alongside system metrics. Only the top 20 processes by CPU usage are exported per collection cycle to control cardinality.

Metric	Type	Unit	Labels	Description
process_cpu_usage_ratio	Gauge	ratio	pid, process_name, username	Per-process CPU usage (0.0–1.0)
process_memory_usage_ratio	Gauge	ratio	pid, process_name, username	Per-process memory usage (0.0–1.0)
process_memory_rss_bytes	Gauge	bytes	pid, process_name, username	Resident Set Size
process_threads	Gauge	—	pid, process_name, username	Thread count

note

The full process list is stored in PostgreSQL and available via GET /api/v1/hosts/{id}/processes. Only the top 20 by CPU are exported as time-series metrics to VictoriaMetrics.

Drill-down endpoint: GET /api/v1/hosts/{id}/processes/{pid}/metrics?process_name=X returns all 4 process metrics as time-series for a specific process. The process_name parameter is required to disambiguate PID reuse across collection cycles.

Integration Metrics

Plugin collectors extend monitoring beyond system metrics. Each plugin injects a collector=<name> label into all its metrics for differentiation.

PostgreSQL

See the PostgreSQL Integration page for setup and configuration.

Metric	Type	Unit	Labels	Description
pg_connections_active	Gauge	—	—	Active connections
pg_connections_idle	Gauge	—	—	Idle connections
pg_connections_max	Gauge	—	—	Maximum allowed connections
pg_blocks_hit_total	Counter	—	—	Buffer cache block hits
pg_blocks_read_total	Counter	—	—	Disk block reads
pg_replication_lag_seconds	Gauge	seconds	—	Replication lag
pg_database_size_bytes	Gauge	bytes	database	Database size
pg_deadlocks_total	Counter	—	—	Total deadlocks
pg_xact_commit_total	Counter	—	—	Total committed transactions
pg_xact_rollback_total	Counter	—	—	Total rolled-back transactions
pg_dead_tuples	Gauge	—	table	Dead rows (top 50 tables)
pg_stat_statements_count	Gauge	—	—	Tracked query statements
pg_stat_statements_calls_total	Counter	—	—	Total statement executions
pg_stat_statements_time_seconds_total	Counter	seconds	—	Total statement execution time
pg_stat_statements_slow_queries	Gauge	—	—	Statements with mean > 100ms

Derived (computed at query time)

pg_cache_hit_ratio is computed at query time in the backend from pg_blocks_hit_total and pg_blocks_read_total.

note

pg_stat_statements metrics require the PostgreSQL extension to be enabled. If the extension is not installed, these metrics are silently skipped.

Nginx

See the Nginx Integration page for setup and configuration.

Metric	Type	Unit	Labels	Description
nginx_connections_active	Gauge	—	—	Current active connections
nginx_connections_accepted_total	Counter	—	—	Total accepted connections
nginx_connections_handled_total	Counter	—	—	Total handled connections
nginx_requests_total	Counter	—	—	Total HTTP requests
nginx_connections_reading	Gauge	—	—	Connections reading request header
nginx_connections_writing	Gauge	—	—	Connections writing response
nginx_connections_waiting	Gauge	—	—	Idle keep-alive connections

Docker

See the Docker Integration page for setup and configuration.

Metric	Type	Unit	Labels	Description
docker_containers	Gauge	—	state	Container count by state (running, exited, created, etc.)
docker_container_cpu_seconds_total	Counter	seconds	container_name, container_id	Cumulative CPU time consumed
docker_container_memory_used_bytes	Gauge	bytes	container_name, container_id	Memory usage minus cache
docker_container_memory_limit_bytes	Gauge	bytes	container_name, container_id	Memory limit (omitted if unlimited)
docker_container_network_receive_bytes_total	Counter	bytes	container_name, container_id	Total bytes received (all interfaces)
docker_container_network_transmit_bytes_total	Counter	bytes	container_name, container_id	Total bytes transmitted (all interfaces)
docker_container_block_read_bytes_total	Counter	bytes	container_name, container_id	Total block device bytes read
docker_container_block_write_bytes_total	Counter	bytes	container_name, container_id	Total block device bytes written

Derived (computed at query time)

docker_container_cpu_usage_ratio and docker_container_memory_usage_ratio are computed at query time in the backend.

note

Per-container metrics are only collected for running containers. Memory limit metrics are omitted when the container has no memory limit set.

Agent Self-Monitoring

The agent collects metrics about its own resource usage. These are always enabled and require no configuration. All metrics carry agent_version and hostname labels, enabling cross-version regression detection and multi-host comparison.

A pre-built Grafana dashboard is available at infra/grafana/dashboards/agent-resources.json.

Process

Metric	Type	Unit	Description
agent_build_info	Gauge	—	Constant 1; build identity carried in labels (agent_version, hostname, go_version). Use count by (agent_version) (agent_build_info) to see the fleet's version distribution and observe a rollout landing.
agent_uptime_seconds	Gauge	seconds	Agent process uptime
agent_cpu_seconds_total	Counter	seconds	Cumulative CPU time (user + system)
agent_memory_rss_bytes	Gauge	bytes	Resident set size
agent_memory_vms_bytes	Gauge	bytes	Virtual memory size
agent_open_fds	Gauge	—	Open file descriptors
agent_threads	Gauge	—	OS thread count
agent_ctx_switches_voluntary_total	Counter	—	Voluntary context switches
agent_ctx_switches_involuntary_total	Counter	—	Involuntary context switches
agent_io_read_bytes_total	Counter	bytes	Process disk read bytes
agent_io_write_bytes_total	Counter	bytes	Process disk write bytes

Derived (computed at query time)

agent_cpu_ratio is computed at query time in the backend from agent_cpu_seconds_total.

Go Runtime

Metric	Type	Unit	Description
agent_go_heap_alloc_bytes	Gauge	bytes	Go heap allocation
agent_go_heap_sys_bytes	Gauge	bytes	Go heap obtained from OS
agent_go_heap_objects	Gauge	—	Live heap object count
agent_go_stack_bytes	Gauge	bytes	Go stack memory in use
agent_go_sys_bytes	Gauge	bytes	Total Go memory from OS
agent_go_memlimit_bytes	Gauge	bytes	Soft memory limit (only present when PROXIMA_AGENT_MEMORY_LIMIT_MB is set)
agent_goroutines	Gauge	—	Goroutine count
agent_gc_cycles_total	Counter	—	GC cycle count
agent_gc_pause_seconds_total	Counter	seconds	Total GC pause time

NATS Transport

Metric	Type	Unit	Description
agent_nats_messages_out_total	Counter	—	Messages published to NATS
agent_nats_bytes_out_total	Counter	bytes	Bytes published to NATS
agent_nats_messages_in_total	Counter	—	Messages received from NATS (request-reply)
agent_nats_bytes_in_total	Counter	bytes	Bytes received from NATS
agent_nats_reconnects_total	Counter	—	NATS reconnection count
agent_nats_outbox_length	Gauge	—	Outbox queue depth (indicates backpressure when > 0)
agent_nats_rtt_seconds	Gauge	seconds	Round-trip time to NATS server

Offline Buffer

These metrics are present only when the transport exposes disk-buffer stats. While NATS is unreachable, the agent buffers outgoing messages to disk and replays them on reconnect.

Metric	Type	Unit	Description
agent_buffer_messages	Gauge	—	Messages currently buffered to disk
agent_buffer_bytes	Gauge	bytes	Bytes currently buffered to disk
agent_buffer_dropped_total	Counter	—	Messages dropped because the buffer was full
agent_buffer_replayed_total	Counter	—	Buffered messages successfully replayed after reconnect

Healthcheck

These metrics are present only when the healthcheck subsystem is active (i.e., healthcheck rules are configured for the host).

Metric	Type	Unit	Description
agent_healthcheck_scans_total	Counter	—	Total healthcheck scans completed
agent_healthcheck_last_duration_seconds	Gauge	seconds	Duration of last healthcheck scan
agent_healthcheck_last_passed	Gauge	—	Checks passed in last scan
agent_healthcheck_last_failed	Gauge	—	Checks failed in last scan
agent_healthcheck_last_total	Gauge	—	Total checks in last scan
agent_healthcheck_last_findings	Gauge	—	Detailed scanner findings (e.g. CVEs, OpenSCAP rule results) recorded in last scan

Config Cache

These metrics are present only when the agent config sync subsystem is active.

Metric	Type	Unit	Description
agent_configcache_configs_cached	Gauge	—	Number of cached config types
agent_configcache_version	Gauge	—	Max version across cached configs

Resource Limits

The agent supports configurable resource limits:

Environment Variable	Default	Description
PROXIMA_AGENT_MEMORY_LIMIT_MB	— (disabled)	Go soft memory limit via debug.SetMemoryLimit
PROXIMA_AGENT_COLLECTOR_TIMEOUT	30s	Per-collector context deadline; timed-out collectors degrade gracefully

Derived metrics (computed at query time)

The ratio metrics the UI and pc CLI show are not stored — the backend computes them from the raw values above at query time, scoped to the caller's VictoriaMetrics tenant. A request for a derived name is rewritten into the equivalent MetricsQL expression, so callers keep querying the same names:

Derived Metric	Source Expression
cpu_usage_ratio	Sum of non-idle CPU modes / total CPU time
cpu_iowait_ratio	iowait / total CPU time
memory_used_ratio	memory_used_bytes / memory_total_bytes
memory_swap_used_ratio	1 - (swap_free / swap_total)
disk_used_ratio	disk_used_bytes / disk_total_bytes
disk_inodes_used_ratio	1 - (inodes_free / inodes_total)
docker_container_cpu_usage_ratio	rate(cpu_seconds_total)
docker_container_memory_usage_ratio	memory_used / memory_limit
pg_cache_hit_ratio	rate(blocks_hit) / (rate(blocks_hit) + rate(blocks_read))
agent_cpu_ratio	rate(agent_cpu_seconds_total)

Why query-time, not recording rules

These were originally produced by vmalert recording rules, which wrote the computed series back into VictoriaMetrics. That does not work with the platform's per-client VM tenancy: host metrics live in one tenant per client, but OSS vmalert is single-tenant per instance and cannot write derived series into each client tenant (per-group tenant / -clusterMode is a VictoriaMetrics Enterprise feature, and this cluster's multitenant query endpoint won't enumerate the client tenants). Computing at query time needs no recording rules, no per-tenant infrastructure, and covers any new client automatically — so vmalert and its rules were removed entirely. Background: docs/superpowers/specs/2026-06-05-vmalert-multitenant-recording-rules-design.md.

Adding a derived metric

Add an entry to the computedMetrics registry in backend/internal/metrics/computed.go — a build function that renders the MetricsQL expression for a label scope (set container: true for per-container metrics, seriesSource for the raw metric whose labels_key enumerates its series). The host / environment / client / container query builders and the metric-name lists pick it up automatically.

Backend Self-Monitoring

The backend exposes Prometheus metrics at /metrics via the OpenTelemetry SDK. These are scraped by vmagent and stored in VictoriaMetrics under account 0 (separate from tenant-scoped agent metrics). Subsystems include HTTP, Auth, PQL Search, NATS Workers, VictoriaMetrics Client, Tenant Cache, Change Detection, Log Collection, Healthcheck, Compliance, Alerting, Credentials, Config Sync, Terminal, Fleet, and Database connection pools.

For the full metric list with types, labels, and descriptions, see Prometheus Metrics Reference.

Fleet (agent self-update)

Metric	Type	Labels	Description
proxima_fleet_rollouts_total	counter	status (created/paused/completed/aborted)	Rollout lifecycle transitions
proxima_fleet_rollouts_active	gauge	—	Currently active rollouts
proxima_fleet_agent_updates_total	counter	status (sent/healthy/failed/timed_out/skipped_offline)	Per-agent update transitions
proxima_fleet_update_duration_seconds	histogram	—	Rollout duration from start to terminal status
proxima_fleet_auth_denied_total	counter	—	Tenant/scope denials on rollout requests

Label Schema

Every metric stored in VictoriaMetrics includes these system labels:

Label	Description
__name__	Metric name (e.g., cpu_user_seconds_total)
host_id	Host UUID
environment_id	Environment UUID
labels_key	Deterministic key from sorted label pairs (e.g., device=sda,mount=/)

Plus the original agent labels (e.g., device, mount, interface, collector, database, table).

The labels_key enables series discovery — the API endpoint GET /hosts/:id/metrics/:name/series returns all distinct labels_key values, allowing the frontend to enumerate and query each series individually.