Deployment

The Proxima Console backend runs in the proxima-production Kubernetes cluster, in the console-system namespace, deployed entirely via GitOps. The desired state lives in a separate argocd-infra repository (not the console repo), and ArgoCD reconciles the cluster to match it. This page covers the container image, the deployment topology, and operational considerations.

Production is Kubernetes; docker-compose is local-dev only

The single-box docker compose + SSH deploy is retired for the application tier. docker-compose / make docker-up now apply to local development only — see Quick Start. Dev/prod parity holds at the container-image level, not the orchestration level.

Docker Image

The backend uses a multi-stage Docker build to produce a minimal container image:

1. Build stage: Uses golang:1.26 to compile the Go binary with all dependencies.
2. Runtime stage: Uses a minimal base image (scratch-like) containing only the compiled binary and CA certificates.

The Dockerfile is located at:

infra/docker/backend/Dockerfile

Building the Image

docker build -f infra/docker/backend/Dockerfile -t registry.prxm.uz/proxima/console/backend:latest .

The CI pipeline builds and pushes a per-commit image to the registry on every push:

registry.prxm.uz/proxima/console/backend:<git-sha>

The <git-sha> (CI_COMMIT_SHA) tag is what GitOps pins — see How a Deploy Works below.

Deployment Topology

The backend is one of several ArgoCD apps under console-system (alongside frontend, docs, mcp, telegram-bot, telegram-miniapp, and an in-cluster Valkey cache). Each app's image is registry.prxm.uz/proxima/console/<svc>:<git-sha>.

 agents ──NATS TLS+JWT──► NATS (box, interim) ──► backend (k8s)
        ──gRPC terminal (api-console:443/h2)───► backend :9090 (k8s)

 ┌ k8s proxima-production · ns console-system · GitOps (ArgoCD + Argo Rollouts) ┐
 │ backend(blue-green) · frontend · docs · mcp · telegram-bot · telegram-miniapp │
 │ · Valkey      images: registry.prxm.uz/proxima/console/<svc>:<sha>            │
 │ platform: External-Secrets←Vault · cert-manager · istio Gateway API           │
 └───┬───────────────┬─────────────────┬────────────────┬───────────────────────┘
 SQL │        metrics │ vmauth :8427    │ secrets        │ logs/traces (interim, box)
     ▼               ▼  (project 42)    ▼                ▼
 psql01:30034   in-cluster VM       vault.prxm.uz    VictoriaLogs/Tempo @ box
 (external)     (monitoring ns)     (ESO)

Argo Rollouts (blue-green)

The backend is deployed as an Argo Rollout using a blue-green strategy rather than a plain Kubernetes Deployment:

• A new revision is brought up as the preview Service while the current revision continues serving the active Service.
• Argo Rollouts waits for the preview pods to pass their readiness probe (/readyz), then auto-promotes the preview to active.
• If the preview never becomes ready, the rollout auto-rolls-back — production traffic stays on the old, healthy revision the whole time.

Health Probes

The backend exposes two health endpoints, wired directly to the pod's Kubernetes probes:

Endpoint	Probe	Behavior
/healthz	liveness	Returns 200 OK if the process is alive. A failing liveness probe restarts the pod.
/readyz	readiness	PostgreSQL is the only critical dependency. 200 ready when everything is up; 200 degraded when Postgres is up but NATS, VictoriaMetrics, VictoriaLogs or Valkey is down — the API keeps serving and the pod keeps receiving traffic; 503 not_ready only when Postgres is down. Gates traffic and blue-green promotion.

Configuration via External Secrets

The backend receives no plaintext secrets in its manifests. Environment variables and secrets are injected by the External-Secrets Operator (ESO), which syncs values from the in-cluster Vault (vault.prxm.uz) into Kubernetes Secrets that the Rollout consumes as env.

• Non-secret defaults and endpoints are set as plain env on the Rollout pod spec.
• Secret material (PROXIMA_DB_URL, PROXIMA_JWT_SECRET, NATS JWT seeds, the agent-enrollment CA, Vault AppRole credentials, etc.) flows Vault → ESO → Kubernetes Secret → pod env.

See Environment Variables for the full variable list and the production endpoints they point at.

Database Migrations

Migrations are never run by the backend application itself, and are not run from CI (CI cannot reach the cluster-internal database). Instead, they run as an ArgoCD PreSync-hook Job (console-migrate) that executes before each backend sync:

• The PreSync Job runs golang-migrate (migrate up) against psql01 using the same image/build as the backend.
• ArgoCD only proceeds to sync the backend Rollout after the migration Job completes successfully.
• This keeps the schema ahead of the new code, and means a single GitOps sync atomically migrates and then rolls out the new backend.

Running migrations as a separate Job (rather than inside the app) ensures that:

• Migration failures block the rollout instead of crash-looping the backend.
• Multiple backend replicas never race to apply migrations.
• Migration state can be inspected independently of the application.

The server can migrate itself — just not here

PROXIMA_MIGRATE_ON_START (default false) makes the server apply the migrations embedded in its own binary at startup. It exists for a single-VM install that runs no separate migrate Job, and it stays off on Kubernetes for exactly the three reasons above. Turning it on here would put two mechanisms in a race to migrate one database during a blue-green rollout, which is worse than either alone.

How a Deploy Works

A deploy is a GitOps flow, not an SSH/docker run on a server:

1. Merge to main — the change lands in the console repo.
2. CI builds + pushes registry.prxm.uz/proxima/console/<svc>:<git-sha> for each service (docker:* jobs).
3. deploy:k8s-gitops (a manual gate, main only) clones the argocd-infra repo over an SSH deploy key, bumps every console image tag under clusters/production/console-system/ to the new <git-sha>, then commits + pushes.
4. ArgoCD detects the change and syncs: it first runs the console-migrate PreSync Job, then applies the updated backend Rollout.
5. Argo Rollouts performs the blue-green promotion (preview → active) once /readyz passes.

Verifying a Deploy

Use kubectl / ArgoCD against the console-system namespace (there is no box to SSH into):

# Rollout status and blue-green promotion state
kubectl -n console-system argo rollouts get rollout console-backend

# Pod readiness
kubectl -n console-system get pods -l app=console-backend

# Backend logs
kubectl -n console-system logs deploy/console-backend -f

# ArgoCD sync / health status
argocd app get console-backend

A healthy deploy ends with the new revision promoted to active, all pods Ready, and the public endpoint https://api-console.prxm.uz/readyz returning 200 OK.

Rollback

Because state is GitOps-managed, roll back by reverting the image-tag bump in argocd-infra (or using ArgoCD's history) so ArgoCD syncs the previous <git-sha>. Argo Rollouts brings the prior revision back as active. If a schema change must also be reversed, run the corresponding down migration via a one-off migrate down Job — migrations are applied independently of the app, so a database rollback can be performed separately from the image rollback.