Data Flow

This page describes how data moves through the Proxima Console system, from agent collection to frontend display.

Deployment Context

The data flows below are unchanged at the logic level, but in production they run inside the proxima-production Kubernetes cluster (namespace console-system, deployed via GitOps — see the Architecture Overview). This changes where each hop terminates:

• Frontend / API requests arrive over Cloudflare → the istio Gateway at app-console / api-console.prxm.uz, then route to the frontend and backend pods via HTTPRoutes (cert-manager-issued TLS). The pc CLI's gRPC terminal traffic shares api-console.prxm.uz:443 (HTTP/2) and is routed by a longest-prefix rule to the backend's gRPC port :9090.
• Metrics are written to and read from the in-cluster VictoriaMetrics cluster (monitoring namespace) through the vmauth-proxy (vmauth-proxy.monitoring.svc.cluster.local:8427). Console authenticates as the vmuser-console VMUser and is isolated in projectID 42, so every /insert/.../import and query path below targets tenant <accountID>:42.
• Logs and traces still terminate on the box (interim): VictoriaLogs (:9428) and Tempo (:4317), firewalled to the cluster egress. These move to dedicated VMs in Phase L.
• NATS (agent telemetry + commands) is also still on the box at nats://nats.prxm.uz:4222 (interim — moves to a dedicated VM in Phase 4). The agent↔NATS↔backend flow itself is unchanged.
• PostgreSQL is the external managed psql01 instance (10.10.3.11:30034). Schema migrations run as an ArgoCD PreSync-hook Job (console-migrate, migrate up) before each backend sync — not from CI.
• Secrets (DB/NATS credentials, the agent-enrollment CA, NATS JWT seeds) are injected into pods by the External-Secrets Operator from in-cluster Vault (vault.prxm.uz); none live in manifests.

The remainder of this page describes the application-level data flows, which behave identically regardless of where they are deployed.

Inventory Flow (Event-Driven)

The inventory flow is the primary data collection mechanism. It is event-driven and uses NATS JetStream for reliable delivery.

Processing Steps

1. The agent collects a full system snapshot every 15 minutes, including host info (uptime, boot time, timezone, virtualization, cloud provider/region/zone), services, packages, open ports, network interfaces, users, current labels (static + dynamic), and expanded inventory (swap, SELinux/AppArmor status, DNS servers, NTP, kernel parameters, locale, cloud instance type, failed systemd units, firewall rules, cron jobs, kernel modules, TLS certificates, security configuration items, GPU devices).
2. The snapshot is published to the NATS subject proxima.{client}.{env}.{agent_id}.inventory, where {agent_id} is the UUID from the agents table assigned during enrollment.
3. The EventsWorker receives the message via a durable JetStream consumer and dispatches it to the InventoryWorker (which implements the MessageProcessor interface).
4. The worker resolves the agent record from the agents table and verifies the agent type is host (collectors do not create host records). It then upserts the host record in a single database transaction keyed on (environment_id, hostname). The agent_id column on the host row tracks which agent owns it. Labels from the payload are stored as JSON in the hosts.tags column.
5. Sub-resources are synced by deleting removed items and upserting current items: services, packages, ports, network interfaces, users, CPU hardware, disk hardware, filesystems, containers, firewall rules, cron jobs, kernel modules, certificates, security items, GPU devices.
6. After successful persistence, the worker ACKs the NATS message.

Host Record Creation

Agent enrollment always creates an agent record in the agents table. For a host agent that supplies a hostname, the backend also pre-links the hosts row at enrollment (best-effort INSERT … ON CONFLICT, agent_status='online') so per-host config resolves on the agent's first config fetch; the InventoryWorker later upserts the same row, preserving the agent_id via COALESCE. When no hostname is supplied (or for collector agents, which never own a host row), the host record is instead created when the InventoryWorker processes the first inventory message — letting agents connect and report before a host entry exists.

Heartbeat Flow

Heartbeats provide lightweight liveness signals between full inventory snapshots.

1. The agent publishes a heartbeat message every 30 seconds, including current labels and config state.
2. The backend heartbeat worker receives the message and updates the host record:
   • agent_version — Current version of the reporting agent
   • last_seen_at — Timestamp of the heartbeat
   • agent_status — Set to online
   • tags — Updated from labels using COALESCE (if labels are present they replace existing tags; if absent, existing tags are preserved)
   • config_hash — SHA-256 fingerprint of the agent's cached config state (enables drift detection)
   • config_version — Maximum config version across all cached config types (enables staleness detection)
   • Collector versions — Each collector's reported server version (e.g., "PostgreSQL 18.2", "nginx/1.27.4") is upserted into host_integrations.version

If heartbeats stop, the Stale Agent Detection worker will mark the host offline after the configured timeout (default 5 minutes).

Integration-to-Asset Sync

Heartbeat messages now auto-create CMDB assets for host integrations. When the heartbeat worker upserts host_integrations (collector names and versions), a follow-up sync step creates or updates corresponding entries in the assets table. This ensures that discovered infrastructure components are automatically represented in the CMDB without manual registration.

Flow: agent heartbeat → heartbeat worker → upsert host_integrations → sync CMDB assets.

The mapping from integration name to CMDB asset type is:

Integration	Asset Type
postgresql	database
redis	cache
nginx	web_server
docker	container_runtime

Each synced asset is scoped to the host's client and environment and linked back to the originating host integration.

Stale Agent Detection

The StaleDetectorWorker runs on a timer (default every 60 seconds) and marks agents as offline when they stop sending heartbeats.

1. The worker queries for hosts where agent_status = 'online' and last_seen_at is older than the stale timeout (default 5 minutes).
2. Matching hosts are updated to agent_status = 'offline' in a single UPDATE query.
3. The count of newly-offline hosts is logged at info level.

This closes the loop on agent liveness — heartbeats set agent_status = 'online', and the stale detector reverts it to offline when heartbeats stop. The frontend uses this status to skip unnecessary metric fetches for offline agents and to display the correct status badge.

Metrics Flow

The metrics flow handles high-volume time-series data from agents to VictoriaMetrics.

Processing Steps

1. The agent collects CPU, memory, disk, and network metrics every 60 seconds (configurable via PROXIMA_AGENT_METRICS_INTERVAL).
2. Readings are batched into a single NATS message and published to proxima.{client}.{env}.{agent_id}.metrics.
3. The TelemetryWorker receives the message via a durable JetStream consumer on the TELEMETRY stream and dispatches it to the MetricsWorker (which implements the MessageProcessor interface).
4. The MetricsWorker parses the MetricsPayload, resolves timestamps (per-metric if provided, otherwise envelope timestamp), and generates a deterministic labels_key for each metric.
5. Metrics are converted to Prometheus format and sent via HTTP POST to VictoriaMetrics vminsert using the tenant-specific import endpoint (/insert/{accountID}/prometheus/api/v1/import).
6. After successful persistence, the worker ACKs the NATS message.

Wire Format

Each metric item in the payload includes type (gauge/counter/rate) and unit (percent/bytes/etc.) for future use by plugins and dashboards, but these are not stored per-row in VictoriaMetrics. They describe the metric name itself (metadata), not individual data points.

Storage

• Metrics are stored in the in-cluster VictoriaMetrics Cluster (monitoring namespace) with 120-day retention. The backend reaches it through the vmauth-proxy (:8427) as the vmuser-console VMUser, isolated in projectID 42.
• Each metric includes labels: host_id, environment_id, labels_key, plus original agent labels.
• Multi-tenancy is achieved via vm_account_id in the URL path — each client's data is physically isolated in VictoriaMetrics. Combined with projectID 42, Console's tenants are written as <accountID>:42, separate from the shared cluster's :0 tenants.
• VictoriaMetrics handles deduplication natively.

Process Snapshot Flow

The process flow handles full process list snapshots from agents, using hybrid storage — PostgreSQL for the queryable snapshot and VictoriaMetrics for top-N process time-series metrics.

Processing Steps

1. The agent's process collector uses gopsutil to enumerate all running processes (PID, name, command, username, state, PPID, threads, CPU%, memory%, RSS, VMS, start time) and publishes the full list to proxima.{client}.{env}.{agent_id}.processes on the TELEMETRY stream every 30 seconds (configurable via agent.process_interval).
2. The TelemetryWorker dispatches the message to the ProcessesWorker (which implements the MessageProcessor interface).
3. The worker parses the ProcessesPayload and stores the full snapshot in PostgreSQL using a delete-replace pattern within a transaction (same approach as containers).
4. The worker selects the top 20 processes by CPU usage and exports 4 metrics per process to VictoriaMetrics: process_cpu_usage_ratio, process_memory_usage_ratio, process_memory_rss_bytes, process_threads.
5. Process metrics include labels: host_id, environment_id, pid, process_name, username.

Frontend

The Processes tab on the Host Detail Page provides an htop-like experience with live data.

The table features sortable columns (CPU%, MEM%, RSS, threads, PID) defaulting to CPU descending, color-coded usage bars (green/blue/amber/red by utilization), client-side search filtering, and 30-second auto-refresh matching the agent's publish interval.

Log Collection Flow

The log collection flow handles host-level logs (journald + file tailing) from agents to VictoriaLogs.

Processing Steps

1. The agent collects logs from two sources: journald (spawns journalctl --output=json --follow as a subprocess) and file tailers (poll-based with 500ms interval, inode rotation detection).
2. Log entries are buffered and flushed every 10 seconds (configurable via PROXIMA_AGENT_LOG_INTERVAL).
3. The LogsPayload is published to proxima.{client}.{env}.{host}.logs on the TELEMETRY stream.
4. The TelemetryWorker dispatches the message to the LogsWorker (which implements the MessageProcessor interface).
5. The LogsWorker resolves the host's client_id and environment_id via the host store, then converts entries to VictoriaLogs LogEntry format with tenant fields.
6. Entries are inserted into VictoriaLogs via HTTP POST to /insert/jsonline with the AccountID header set to the client's vm_account_id.

Storage

• Logs are stored in VictoriaLogs (currently still on the box at :9428 — interim, moves to a dedicated VM in Phase L) with 30-day retention (configurable).
• Each entry includes fields: _msg, _time, host_id, client_id, environment_id, source, level, unit.
• Multi-tenancy is achieved via VictoriaLogs native AccountID headers -- the same model used by VictoriaMetrics for metrics.
• VictoriaLogs auto-indexes all fields for LogsQL queries.

Query Proxy

The backend provides two proxy endpoints for querying logs:

• POST /api/v1/logs/query -- Accepts a JSON body with query, start, end, limit fields. Parses and forwards as URL parameters to VictoriaLogs /select/logsql/query with the AccountID header.
• GET /api/v1/logs/select/* -- Reverse proxy to VictoriaLogs VMUI and API endpoints. Strips the /api/v1/logs prefix, injects AccountID header. Used by the frontend for the embedded VMUI log viewer.

Both endpoints require authentication and resolve the tenant via tenantcache (client UUID → vm_account_id).

Kubernetes Inventory Flow

The Kubernetes inventory flow uses a separate cluster-agent binary (deployed as a single-replica Deployment) alongside the existing node agent (deployed as a DaemonSet). The two agents serve different purposes and publish to different NATS subjects.

Two-Agent Architecture

Component	Deployment	Collects	NATS Subject
Node agent	DaemonSet (every node)	OS metrics, inventory, processes, logs, changes	proxima.{client}.{env}.{host-id}.{type}
Cluster agent	Deployment (1 replica)	K8s API: cluster, nodes, namespaces, workloads, pods	proxima.{client}.{env}.{cluster-slug}.kubernetes

Both agents use the same enrollment flow (POST /api/v1/agents/enroll) and NATS authentication (JWT + Nkey).

Collection Cycle

Processing Steps

1. The cluster-agent queries the K8s API every 5 minutes (configurable via PROXIMA_K8S_SYNC_INTERVAL) using client-go with in-cluster service account authentication.
2. It collects: cluster metadata (version, platform), all nodes (status, capacity, addresses), all namespaces, all workloads (Deployments, StatefulSets, DaemonSets), and all pods (phase, containers, resources, owner references).
3. Platform is auto-detected from kubelet version (rke2, k3s) or node provider ID prefix (eks, gke, aks).
4. Pod owners are resolved: if a pod is owned by a ReplicaSet, the collector infers the parent Deployment by trimming the ReplicaSet's hash suffix.
5. If PROXIMA_K8S_POD_LIMIT is set (default 5000), pods are prioritized: non-Running pods first, then by restart count descending.
6. The payload is published to NATS subject proxima.{client}.{env}.{cluster-slug}.kubernetes.
7. The KubernetesWorker processes the message in a single transaction using upsert-then-prune: entities are upserted with ON CONFLICT, then entities not seen in the current sync cycle are pruned. This preserves stable entity IDs across sync cycles.
8. Node↔host linkage: for each K8s node, the worker attempts to match it to an existing Proxima host by hostname or IP address. If matched, k8s_nodes.host_id is set, linking K8s-level and OS-level data.

Storage

Table	Purpose
clusters	Cluster metadata, version, platform, denormalized counts
k8s_nodes	Node status, capacity, allocatable, host linkage
k8s_namespaces	Namespace status, labels, resource counts
k8s_workloads	Deployments, StatefulSets, DaemonSets with replica status
k8s_pods	Pod phase, containers, restarts, resource requests/limits
k8s_services	Service type, ports, selector, cluster IP, load balancer
k8s_ingresses	Ingress rules, TLS config, ingress class
k8s_pvcs	PVC phase, storage class, capacity, access modes
k8s_events	K8s events with type, reason, involved object, count
k8s_jobs	Jobs and CronJobs with status, completions, schedule
k8s_hpas	HPA auto-scaling config, target, current/desired replicas
k8s_network_policies	Network segmentation rules, pod selectors
k8s_resource_quotas	Per-namespace resource limits and usage
k8s_endpoint_slices	Service endpoint addresses and ports
assets	Universal CMDB layer (each cluster gets an asset_type=cluster entry)

Self-Monitoring

The cluster-agent exposes a Prometheus-compatible /metrics endpoint (default port 9090) with:

• proxima_collector_syncs_total — total sync count
• proxima_collector_sync_errors_total — error count
• proxima_collector_last_sync_duration_seconds — last sync duration
• proxima_collector_resources_collected{resource="..."} — per-resource-type gauges

A /healthz endpoint is available for Kubernetes liveness and readiness probes.

Integration Metrics Flow

Integration (plugin) metrics follow the same storage path as system metrics but include a collector=<name> label for differentiation.

Collection

1. The agent runs each configured collector plugin on the metrics interval.
2. Each collector produces metrics under its namespace (e.g., pg_* for PostgreSQL, nginx_* for Nginx).
3. The metrics task injects a collector=<name> label into every metric from a plugin collector (e.g., collector=pg-main).
4. All metrics (system + plugin) are batched into a single NATS message and published.

Storage

Plugin metrics are stored in the same VictoriaMetrics cluster as system metrics with the same tenant isolation. The collector=<name> label becomes part of the labels_key (e.g., collector=pg-main), which uniquely identifies the series. This prevents collisions when multiple collectors of the same type run on the same host.

Frontend Query

The integration Metrics tab uses useIntegrationMetricDashboard to fetch plugin metrics:

1. Discover series for each metric name (e.g., pg_connections_active).
2. Filter series by collector=<name> prefix in the labels_key.
3. Fetch data for each matching series in parallel.
4. If no collector-labeled series exist (pre-label-injection data), fall back to empty labels_key.

When multiple collectors exist, a picker lets the operator select which collector's metrics to display.

Metrics Query Flow

The frontend queries metrics via the REST API. The useMetricDashboard hook fetches all metrics for a host in parallel. For per-series metrics (e.g., per-disk), it discovers available series first, then fetches each in parallel. For network metrics, it skips series discovery and fetches aggregated data directly. The backend automatically selects the optimal query step based on the requested time range.

Server-Side Aggregation

When labels_key is empty (or not provided), the backend aggregates across all series for that metric using MetricsQL sum(). This is used for network metrics, where per-interface breakdown is not needed on the dashboard.

When labels_key is non-empty, the original per-series behavior is preserved (filtered to that specific series). Derivative queries use MetricsQL rate() for counter-type metrics.

Endpoints

• GET /hosts/:id/metrics — Discover which metrics are available for a host.
• GET /hosts/:id/metrics/:name?start=&end=&labels_key= — Query time-series data points. Defaults to last 1 hour. Empty labels_key returns aggregated data across all series.
• GET /hosts/:id/metrics/:name/series — Discover labels_key values for labeled metrics (e.g. per-disk, per-interface).

Aggregate Endpoints (Environment / Client)

In addition to per-host queries, the backend supports aggregate queries across all hosts in an environment or all hosts of a client:

• GET /environments/:id/metrics / GET /clients/:id/metrics — List metric names across all hosts.
• GET /environments/:id/metrics/:name / GET /clients/:id/metrics/:name — Query aggregated data points (AVG across hosts).
• GET /environments/:id/metrics/:name/series / GET /clients/:id/metrics/:name/series — List available series across all hosts.

These endpoints use the same query parameters (start, end, labels_key, derivative) and response format as the per-host endpoints. The aggregation uses MetricsQL AVG across hosts to combine data into a single time series. Step auto-calculation and derivative support are preserved.

Frontend Rendering

The MetricChart component renders data using Recharts:

• Single-series metrics (e.g., CPU usage): One Area with gradient fill for avg_value.
• Multi-series metrics (e.g., per-disk usage): Data is merged by timestamp, one colored Area per labels_key, with a legend.
• Chart gap detection: When data points are missing (gap > 2x expected interval), fillTimeGaps() inserts null placeholders and Recharts disconnects the line (connectNulls={false}).
• Time ranges: 1h, 6h, 24h, 7d — selectable via TimeRangeSelector button group.

Change Detection Flow

The change detection flow handles file integrity monitoring — tracking file changes across managed hosts.

Processing Steps

1. The agent's file watch scanner runs every 60 seconds (configurable), walking all configured paths and comparing file hashes against a local JSON database.
2. Changed files are classified as created, modified, deleted, or metadata_only and published to proxima.{client}.{env}.{host}.changes on the EVENTS stream.
3. For delta scans, non-sensitive/non-binary file content is gzip-compressed and base64-encoded in the payload for server-side diff generation.
4. The ChangesWorker processes the message, reassembles multi-part batches if needed, and generates unified diffs for modified files.
5. Three tables are updated: watched_files (current state), file_versions (version history), and change_events (audit log).

Message Batching

If a scan produces more data than max_message_size (default 800 KB), the agent splits the payload into multiple batches sharing a batch_id. Each batch has batch_seq and batch_total fields. The backend's ChangesWorker reassembles batches before processing.

Storage

Table	Purpose
watched_files	Current state of each monitored file (path, hash, size, owner, sensitivity)
file_versions	Version history with content snapshots and unified diffs
change_events	Audit log of all file changes with severity, event type, and details

Frontend

The change detection UI provides:

• Changes page (/changes) -- paginated timeline of all change events with filters (severity, event type, source)
• Change detail page (/changes/:id) -- event metadata, file details, and inline diff viewer
• File versions page (/hosts/:hostId/files/:fileId) -- version history with click-to-view diffs
• Host Changes tab -- watched files table and recent changes for a specific host

External Change Events Flow (Webhooks)

External tools send webhook events to the backend, which normalizes them and stores them as change events alongside agent-detected file changes. See Webhook Sources for full setup details.

Processing Steps

1. The external tool sends a POST request to the source-specific webhook URL (/api/v1/webhooks/{gitlab,github,argocd}/{token}).
2. The handler validates the token by computing its SHA-256 hash and looking it up in the webhook_sources table. Invalid or disabled sources return 404.
3. If the webhook source has a secret configured, the handler decrypts it (via Vault Transit) and validates: GitLab compares the X-Gitlab-Token header to the decrypted secret, GitHub validates the X-Hub-Signature-256 HMAC-SHA256 signature using the decrypted secret as the HMAC key, ArgoCD compares the X-Argocd-Token header to the decrypted secret.
4. The source-specific normalizer extracts structured metadata into one of three schemas: GitMetadata (push, merge, tag, release), CICDMetadata (pipeline, workflow), or DeployMetadata (ArgoCD sync).
5. If the webhook source has a host_mapping configured, the handler attempts to resolve the event to a specific host UUID using source-specific keys (e.g., repo:org/name or app:frontend).
6. The normalized event is inserted into the change_events table with the appropriate source, event_type, severity, and JSONB details.

Supported Events

Source	Events	Severity
GitLab	push, tag, merge_request (merged), pipeline (success/failed), job	info / warning
GitHub	push, pull_request (merged), release (published), workflow_run, check_suite	info / warning
ArgoCD	sync, health_degraded, sync_failed	info / warning / critical

Storage

Webhook events are stored in the same change_events table used by agent-detected file changes. The source column distinguishes the origin (gitlab, github, deployment). Events with host mapping are linked to a specific host via host_id; unmapped events appear only on environment-level timelines.

Config Sync Flow (Backend → Agent)

The config sync flow enables operators to manage agent configuration centrally, using a two-level model (environment defaults + per-host overrides) with both push and pull delivery. See Agent Config Sync for full details.

Two-Level Config Model

Configs are stored at two levels in PostgreSQL:

• Environment-level (environment_configs) — fleet-wide defaults for all hosts in an environment
• Host-level (host_configs) — per-host overrides

The Config Resolver merges these using a shallow merge strategy: top-level JSON keys from the host override win over environment keys. The effective config includes a source field ("environment", "host_override", or "merged") indicating which level contributed.

Key Design Decisions

• Two-level hierarchy -- environment configs provide sensible defaults; host overrides allow per-host customization without duplicating the full config for every host.
• Database-first -- configs are always persisted before pushing to agents. If the agent misses the push (e.g., it was offline), it will pull the config on next startup.
• Push + pull delivery -- push for immediate updates; pull on startup for resilience. The agent also caches configs to disk for restart recovery even without backend connectivity.
• Best-effort push -- the NATS publish is fire-and-forget. Errors are logged but do not fail the API request.
• Version tracking -- each config update increments a version counter. The agent reports config_hash and config_version in heartbeats, enabling drift detection.

Idempotent Processing

All message processing is designed to be idempotent. If a NATS message is redelivered (due to network issues, worker restart, or delayed ACK), processing it a second time must not create duplicate data or corrupt state.

Inventory Messages

Inventory processing uses a full upsert strategy. Each message contains the complete current snapshot of a host. Processing replaces the previous state entirely:

• Host record: INSERT ... ON CONFLICT (hostname, environment_id) DO UPDATE
• Sub-resources: Delete items no longer present, upsert items that exist in the snapshot

Re-processing an inventory message simply overwrites the same data, producing the same result.

Metrics Messages

VictoriaMetrics handles deduplication natively. Each metric reading is uniquely identified by its timestamp, metric name, and label combination (e.g., host_id, device=sda,mount=/). Sending the same reading twice has no effect.

Heartbeat Messages

Heartbeats update last_seen_at, agent_version, and optionally tags (via COALESCE — only when labels are present in the payload). Re-processing a heartbeat simply writes the same values again, which is a no-op in practice.

Terminal Session Flow

The terminal flow establishes an interactive shell session from the browser to a managed host via a WebSocket–NATS bridge. Unlike all other backend communication, this uses Core NATS (not JetStream) for low-latency bidirectional streaming.

Terminal Access Data Model