Architecture & Data Flows

The single reference for how Proxima Console is deployed and how every kind of data moves through it. Reflects the current (post-2026 migration) state: the app tier runs in the proxima-production RKE2 cluster; backing, observability, and messaging services run on dedicated hosts.

Deployment topology

Tier	Where	What
Ingress	Cloudflare DNS → Hetzner LB (46.225.45.185) → istio-gateway (NodePort)	HTTPS terminate (HTTPRoute) + TCP passthrough (TCPRoute)
App	ns console-system (istio ambient)	backend, mcp, frontend, docs, telegram bot + miniapp (Argo Rollouts)
Database	psql01 10.10.3.11:30034	PostgreSQL
Cache	in-cluster redis-master:6379 db3	Valkey/Redis
Secrets	vault.prxm.uz	Vault (AppRole, Transit KEK)
Messaging	nats01 10.10.4.15	NATS + JetStream (systemd)
Metrics	vmcluster (k8s compute) + vmstorage01/02	VictoriaMetrics, behind vmauth
Logs	vmstorage02 :9428	VictoriaLogs (systemd)
Traces	Tempo (k8s) + Cloudflare R2	OTLP, metrics-generator → VM
LLM	litellm.prxm.uz	AI chat / triage

NATS subjects & JetStream streams

All agent↔backend and internal async work flows over NATS JetStream. Per-host subjects use proxima.<client>.<env>.<host>.<type>; platform subjects use proxima.<domain>.<action>. Agents and the backend authenticate with operator/JWT creds; the backend connects on the private path, agents via nats.prxm.uz.

Stream	Subjects	Carries
TELEMETRY	proxima.*.*.*.{metrics,logs,processes}	agent metrics / log lines / process samples
HEARTBEAT	proxima.*.*.*.heartbeat	agent liveness
SNAPSHOTS	proxima.*.*.*.{inventory,kubernetes}	full host inventory / k8s state
EVENTS	proxima.*.*.*.{changes,healthcheck}	file/config changes, healthcheck results
ALERTS	proxima.alerts.{ingest,correlate,triage}	alert pipeline stages
RUNBOOK	proxima.*.*.*.runbook.{step,complete}	runbook step dispatch + completion
COMPLIANCE	proxima.compliance.evaluate	compliance evaluation jobs
NOTIFICATIONS	proxima.notifications.dispatch	outbound notifications (workqueue)
MEMORY	proxima.memory.draft	L1 resolution-memory drafts (workqueue)

Data flows

1 · Agent enrollment & connection

2 · Telemetry, logs & heartbeat ingestion

Metrics and logs are tenant-scoped: each client maps to a VictoriaMetrics/Logs accountID, all under console's projectID 42.

3 · Inventory & change detection

Agents publish full inventory + k8s state to SNAPSHOTS, and file/config changes

• healthcheck results to EVENTS. Backend workers reconcile these into PostgreSQL (hosts, watched_files/file_versions/change_events, compliance inputs).

4 · Metrics & logs query (console UI)

5 · Alert correlation & AI triage

6 · Runbook execution

User (or an alert action) triggers a runbook → backend publishes runbook.step per step → the target agent executes and replies on runbook.complete → backend records the execution. preview_runbook renders the plan before run_runbook.

7 · Compliance evaluation

A trigger (scheduled or on-demand) puts a job on COMPLIANCE.evaluate → backend evaluates frameworks against host inventory/state → writes evaluations + scores to PostgreSQL, surfaced as compliance scores and top failures.

8 · AI chat & MCP

The MCP server exposes read + action tools (hosts, metrics, alerts, runbooks, …) over the same backend, RBAC-scoped to the caller's API key.

9 · Notifications & L1 memory

Backend dispatches outbound notifications via NOTIFICATIONS.dispatch (Telegram / webhooks). When an alert is resolved, a resolution stub is drafted to MEMORY.draft and embedded into pgvector, so future triage retrieves "this was fixed before" (flow 5).

10 · Traces & self-monitoring

Backend + MCP export OTLP traces → Tempo (blocks in R2); Tempo's metrics-generator emits span-RED + service-graph metrics to VictoriaMetrics. vmagent scrapes the cluster and the dedicated hosts (vmstorage, nats01) into VM tenant 1; each host ships its own journald to VictoriaLogs (0:42) via Vector.

Multi-tenancy

VictoriaMetrics and VictoriaLogs are addressed as AccountID:ProjectID. Console owns projectID 42, with one accountID per client — so a client can only ever read/write its own namespace. Cluster self-monitoring lives in tenant 1. RBAC in the backend scopes every API/MCP call to the caller's permitted clients before any store query.