Production Deployment (Kubernetes)

This is the canonical reference for how Proxima Console is deployed in production.

The application tier runs in the proxima-production Kubernetes cluster, in the console-system namespace, and is deployed entirely through GitOps: the desired state lives in a separate argocd-infra repository (not the console repo), and ArgoCD continuously reconciles the cluster to match it. No one deploys by SSHing to a server.

Historical note. Earlier releases ran the application tier on a single box (89.167.8.247) as systemd-managed Docker containers, deployed over SSH with docker-compose. That model is retired for application workloads. Docker Compose now applies to local development only — see Docker. "Dev/prod parity" still holds, but at the container-image level, not the orchestration level.

Topology at a glance

 agents ──NATS TLS+JWT──► NATS (box, interim) ──► backend (k8s)
        ──gRPC terminal (api-console:443/h2)───► backend :9090 (k8s)

 ┌ k8s proxima-production · ns console-system · GitOps (ArgoCD + Argo Rollouts) ┐
 │ backend(blue-green) · frontend · docs · mcp · telegram-bot · telegram-miniapp │
 │ · Valkey      images: registry.prxm.uz/proxima/console/<svc>:<sha>            │
 │ platform: External-Secrets←Vault · cert-manager · istio Gateway API           │
 └───┬───────────────┬─────────────────┬────────────────┬───────────────────────┘
 SQL │        metrics │ vmauth :8427    │ secrets        │ logs/traces (interim, box)
     ▼               ▼  (project 42)    ▼                ▼
 psql01:30034   in-cluster VM       vault.prxm.uz    VictoriaLogs/Tempo @ 89.167.8.247
 (external)     (monitoring ns)     (ESO)

 public (Cloudflare → istio gateway): api|app|docs|mcp|tg-console.prxm.uz
 CI: GitLab build → registry → deploy:k8s-gitops (SSH→argocd-infra) → ArgoCD → Rollouts

Cluster & namespace

Property	Value
Cluster	proxima-production
Namespace	console-system
GitOps repo	argocd-infra (separate from console)
Manifests path	clusters/production/console-system/
Reconciler	ArgoCD
Progressive delivery	Argo Rollouts (blue-green, backend)

Components

Each Console service is an ArgoCD application under console-system, running the image registry.prxm.uz/proxima/console/<svc>:<git-sha>. The deployed tag is the Git SHA of the commit that built the image, so the running version is always traceable back to a single commit.

Service	Image	Notes
backend	…/console/backend:<sha>	REST API (:8080) + gRPC terminal/SSH (:9090). Argo Rollouts blue-green.
frontend	…/console/frontend:<sha>	React SPA served by nginx
docs	…/console/docs:<sha>	This Docusaurus documentation site
mcp	…/console/mcp:<sha>	MCP server for AI chat
telegram-bot	…/console/telegram-bot:<sha>	Telegram bot
telegram-miniapp	…/console/telegram-miniapp:<sha>	Telegram mini-app
Valkey	(in-cluster)	Redis-compatible cache (sessions, rate limits, MFA, access resolution)

Backend blue-green rollout

The backend is deployed as an Argo Rollouts Rollout (not a plain Deployment) using a blue-green strategy:

• Two Services front the pods: console-backend-active (live traffic) and console-backend-preview (the new version under test).
• On a new image, Rollouts brings up the new ReplicaSet behind the preview Service, waits for it to pass readiness, then auto-promotes it to active.
• If the preview never becomes healthy, Rollouts auto-rolls-back and the active version keeps serving traffic, so a bad image never reaches users.

Public traffic reaches the backend through an istio HTTPRoute that targets the console-backend-active Service.

Database migrations (ArgoCD PreSync hook)

Schema migrations run in the cluster, not from CI, because CI has no route to the cluster-internal database. Each backend sync runs a migration Job as an ArgoCD PreSync hook (console-migrate) that executes golang-migrate migrate up before the new backend pods start. ArgoCD waits for the hook Job to succeed; if migrations fail, the sync stops and the new backend version is not rolled out.

This guarantees the database schema is always migrated ahead of the code that depends on it.

Data plane

The k8s application consumes the following backing services. They are attached resources configured via environment / secrets (Twelve-Factor), not embedded in the app.

Concern	Backing service	Endpoint / detail
Relational DB	PostgreSQL — external managed psql01	10.10.3.11:30034 (PG 16), pgBackRest backups
Metrics	In-cluster VictoriaMetrics cluster (monitoring ns) via vmauth-proxy	vmauth-proxy.monitoring.svc.cluster.local:8427
Cache	In-cluster Valkey	Redis-compatible
Messaging	NATS (agent telemetry + commands, TLS + JWT)	nats://nats-client-console.prxm.uz:4222 — on the box (interim)
Logs	VictoriaLogs	:9428 on the box — interim
Traces	Tempo	:4317 (OTLP) on the box — interim
Secrets	Vault + External-Secrets Operator (ESO)	vault.prxm.uz (in-cluster)
AI chat	LiteLLM	litellm.prxm.uz (central)

Metrics tenancy (vmauth, project 42)

Console is multi-tenant in VictoriaMetrics — one VM accountID per client — and is isolated in projectID 42. The backend authenticates to vmauth-proxy as the vmuser-console VMUser, so it writes and reads tenant <accountID>:42. This keeps Console's series separate from the shared cluster's :0 tenants.

Secrets (Vault + External-Secrets)

There are no secrets in the manifests. The External-Secrets Operator projects values out of Vault (vault.prxm.uz) into Kubernetes Secrets via SecretStore / ExternalSecret resources. Sensitive material delivered this way includes the NATS JWT seeds and the agent-enrollment CA.

Services outside the cluster

Three pieces of the serving path do not run in Kubernetes. The migration off the old single box is complete — it is no longer in the serving path, so do not troubleshoot against it.

Service	Where it runs	Reached as
NATS + JetStream	systemd on nats01	10.10.4.15:4222 from the cluster; nats.prxm.uz:4222 for agents, via the LB :4222 TCPRoute
VictoriaLogs	systemd on vmstorage02	:9428, Console isolated in projectID 42
PostgreSQL	managed psql01	10.10.3.11:30034

Tempo runs in-cluster, with its blocks in Cloudflare R2. See Current Architecture for the authoritative layout.

Networking & ingress

• TLS certificates are issued and renewed by cert-manager.
• Routing uses the istio Gateway API through the shared istio-gateway; REST endpoints are exposed as HTTPRoute resources.
• Public hostnames are Cloudflare-proxied → istio gateway:

Hostname	Serves
api-console.prxm.uz	Backend API (REST :8080, gRPC terminal :9090)
app-console.prxm.uz	Frontend SPA
docs-console.prxm.uz	This documentation site
mcp-console.prxm.uz	MCP server
tg-console.prxm.uz	Telegram mini-app

gRPC terminal on the same host

The pc CLI's gRPC (terminal / SSH proxy) shares api-console.prxm.uz:443 over HTTP/2 with the REST API. A longest-prefix HTTPRoute rule sends the proxima.terminal.v1.TerminalService path to the backend's gRPC port :9090, while everything else falls through to REST on :8080. (This requires Cloudflare gRPC to be enabled for the zone.)

How a change reaches production

1. CI builds and pushes registry.prxm.uz/proxima/console/<svc>:<git-sha>.
2. The manual deploy:k8s-gitops job (main only) bumps the image tags in argocd-infra over an SSH deploy key and pushes.
3. ArgoCD detects the change and syncs console-system.
4. The console-migrate PreSync hook runs migrate up.
5. Argo Rollouts performs the backend blue-green promotion (or auto-rollback).

See CI/CD Pipeline for the full pipeline narrative.