Architecture Overview

Proxima Console is an AI-powered infrastructure operations platform that collects inventory and telemetry from managed servers via lightweight agents, stores it centrally, and presents it through a web console.

High-Level Architecture

The application tier runs in the proxima-production Kubernetes cluster (namespace console-system), deployed by GitOps: desired state lives in a separate argocd-infra repository, and ArgoCD reconciles it. Each service is an ArgoCD app running the image registry.prxm.uz/proxima/console/<svc>:<git-sha>. The backend ships via Argo Rollouts blue-green deployment. Platform concerns are handled by cluster operators — External-Secrets Operator (ESO) injects secrets from Vault, cert-manager issues TLS certificates, and the istio Gateway API routes ingress.

Agents continue to connect outbound over NATS (currently still on the box — interim) and the pc CLI reaches the backend's gRPC terminal endpoint over the shared public ingress.

Interim vs. settled: Everything except the box-resident services is fully migrated. NATS moves to a dedicated VM in Phase 4, and VictoriaLogs + Tempo move to dedicated VMs in Phase L, after which the legacy single box is decommissioned. Until then those three remain on the box, firewalled to the cluster egress. Local development is unaffected — see Dev/Prod Parity for how docker-compose still mirrors this stack locally.

Components

Each application service is an ArgoCD app under console-system, deployed from the image registry.prxm.uz/proxima/console/<svc>:<git-sha>.

Component	Technology	Purpose
Agent	Go 1.24+	Deployed on managed servers. Collects host inventory, heartbeats, system metrics, and file changes, publishes to NATS. Receives runtime config updates from backend via NATS command channel.
Backend	Go 1.24+ (chi, sqlx, slog)	Central API server and NATS consumer workers. Serves the REST API (:8080) and the gRPC terminal service (:9090) for the frontend and pc CLI, processes inventory events (EVENTS stream), and ingests metrics into VictoriaMetrics (TELEMETRY stream). Deployed via Argo Rollouts blue-green (active + preview Services, auto-promote on readiness, auto-rollback on failure).
Frontend	React 18 + TypeScript + Tailwind	Console UI for operators. Displays clients, environments, hosts, and sub-resources.
Database	PostgreSQL 16 (external psql01)	Primary data store for relational data (clients, environments, hosts, integrations). Managed instance at 10.10.3.11:30034 with pgBackRest backups. Migrations run as an ArgoCD PreSync-hook Job (console-migrate) before each backend sync.
Metrics	VictoriaMetrics Cluster (in-cluster, monitoring ns)	Time-series metrics storage with native multi-tenancy via vm_account_id. Console reaches it through the vmauth-proxy (vmauth-proxy.monitoring.svc.cluster.local:8427) as the vmuser-console VMUser, isolated in projectID 42 (tenants <accountID>:42). 120-day retention.
Messaging	NATS 2.10+ with JetStream	Persistent message bus for agent-to-backend communication (at-least-once delivery, durable consumers). Currently still on the box at nats://nats.prxm.uz:4222 (interim — moves to a dedicated VM in Phase 4).
Cache	Valkey (in-cluster, Redis-compatible)	In-memory cache for auth sessions, MFA tokens, rate limiting, lockout tracking, and access resolution.
Secrets	Vault + External-Secrets Operator	Secrets are stored in the in-cluster vault.prxm.uz and injected into pods by ESO — no secrets live in manifests. NATS JWT seeds and the agent-enrollment CA also come from Vault.
AI	LiteLLM (litellm.prxm.uz)	Central LLM gateway used by the AI chat / MCP features.
Observability	OpenTelemetry + Prometheus + Tempo + VictoriaLogs	Distributed tracing (OTel → Tempo, on the box, interim), metrics (Prometheus exposition → VictoriaMetrics), logs (stdout JSON → VictoriaLogs, on the box, interim). Tempo and VictoriaLogs move to dedicated VMs in Phase L.
GitOps / Platform	ArgoCD, Argo Rollouts, cert-manager, istio Gateway API	ArgoCD reconciles the argocd-infra repo into the cluster; Argo Rollouts drives blue-green backend deploys; cert-manager issues TLS; the istio Gateway API (shared istio-gateway) routes public traffic via HTTPRoutes.

Key Architectural Decisions

Single-Process Backend

The backend runs the HTTP API server and NATS consumer workers in a single process. This simplifies deployment and operations for the MVP phase. The internal architecture maintains clean separation between API handlers, domain logic, and data access, allowing the workers to be extracted into separate services in a future phase if scaling requires it.

Service Layer

Business logic lives in backend/internal/service/ (the service layer), sitting between HTTP handlers and data-access stores. Handlers are thin HTTP glue -- they parse requests, call a service method, and serialize the response. Services encapsulate domain rules, orchestrate multiple stores, and keep handlers free of business logic. Current services include AlertService, PasswordService, HostLifecycleService, RunbookService, ComplianceService, and ChatToolService.

Outbound-Only Agent Connections

Agents make outbound NATS connections only. No inbound ports need to be opened on client-managed servers. This simplifies firewall configuration and reduces the attack surface on managed infrastructure. While the backend can push commands to agents via NATS (e.g., config updates), this still operates over the agent's outbound NATS connection -- the agent subscribes to its per-host command subject, so no additional inbound ports are required.

Static Frontend

The frontend is built as static files and served from its own pod (an nginx-based container image) behind the istio Gateway. There is no server-side rendering (SSR). This keeps the frontend deployment simple and allows it to be served from any static file host or CDN.

External Change Events via Webhooks

External tools (GitLab, GitHub, ArgoCD) send webhook events directly to the backend via token-authenticated HTTP endpoints. The backend normalizes these events and stores them as change events alongside agent-detected file changes. This enables timeline correlation across the full delivery pipeline (merge → deploy → config change → metric spike). See Webhook Sources for details.

Explicit Non-Goals

The following technologies are deliberately excluded from the architecture:

• No WebSocket — REST with polling and SSE provides sufficient data freshness for the current use cases. Sole exception: /api/v1/terminal/ws for browser-based interactive terminal access (WebSocket is the only browser-native option for bidirectional real-time streaming).
• No ORM — Raw SQL via sqlx gives full control over queries and avoids abstraction overhead.
• No GraphQL — REST endpoints are simpler and sufficient for the current API surface.
• gRPC for CLI terminal only — The pc CLI uses gRPC bidirectional streaming for terminal sessions, port forwarding, and session join. gRPC shares the public api-console.prxm.uz:443 endpoint (HTTP/2): a longest-prefix HTTPRoute rule sends the proxima.terminal.v1.TerminalService path to the backend's gRPC port (:9090), while REST falls through to :8080. Agent communication still uses NATS exclusively.