Degraded Mode & NATS Outages

The backend is designed to keep serving the API when a non-critical backing service is down, instead of failing the whole platform. This page explains what "degraded" means, how to recognise it, and the runbook for a NATS outage.

Why this exists: a ~19h NATS outage in June 2026 took the entire API down (every endpoint 503'd, even DB-only ones), because readiness was gated on NATS. The backend now degrades gracefully instead. See the game-day below.

What "degraded" means

PostgreSQL is the only critical dependency. If Postgres is down, the API genuinely can't serve and the pod is pulled from rotation. Everything else — NATS, VictoriaMetrics, VictoriaLogs, Valkey — can be down while the API keeps serving in degraded mode; only the features that need that dependency are affected.

Dependency	Down ⇒	Effect
PostgreSQL	503 not_ready, pod pulled	API down (the only one that does this)
NATS	200 degraded	Live ingestion + control ops paused; agents buffer
VictoriaMetrics	200 degraded	Metrics pages show "unavailable"
VictoriaLogs	200 degraded	Logs pages show "unavailable"
Valkey	200 degraded	Cache falls back to the DB (slower)

How to recognise it

• UI banner — a persistent banner across the top of app-console, e.g. "Messaging degraded — agents are buffering recent data (no loss); live ingestion/updates paused, retrying." It names the affected component.
• /readyz (unauthenticated) — returns {"status":"degraded"} with HTTP 200 (the pod stays in rotation). It returns not_ready / 503 only when PostgreSQL is down. The per-dependency breakdown is intentionally not in this body.
• GET /api/v1/system/status (authenticated) — the authoritative view:

  { "degraded": true,
    "components": { "postgresql": "ok", "nats": "down", "victoriametrics": "ok", "victorialogs": "ok", "valkey": "ok" },
    "message": "Messaging (NATS) is down — agents are buffering recent data; live ingestion and updates are paused." }

Probe semantics (for cluster operators)

• liveness /healthz — "process alive", never NATS-gated, so a degraded backend is never restart-killed.
• readiness /readyz — PostgreSQL-only gate; degraded is a 200, so the pod stays in rotation and keeps serving.

Readiness is not the alerting mechanism — the NATSDown / NATSNoConnections VMRules still page on a NATS outage even though the pod stays Ready.

What works vs. what pauses during a NATS outage

• ✅ API + UI serve normally (reads/writes that only need PostgreSQL).
• ✅ Agents keep collecting and disk-buffer their telemetry/events — lossless for ~1h (PROXIMA_AGENT_BUFFER_MAX_AGE).
• ⏸ Live ingestion of agent data is paused; it resumes and the buffers drain automatically when NATS recovers.
• ⛔ Outbound control ops (run runbook, push agent config/override) fail fast with 503 { "error": { "code": "messaging_unavailable" } }. The UI disables those controls with a tooltip rather than letting them hang.

Runbook — NATS outage

Symptoms

• NATSDown / NATSNoConnections alerts firing (VMRules → Telegram).
• Degraded banner in the UI; /readyz = degraded; /api/v1/system/status shows nats: "down".
• Agents flip offline (heartbeats stall) — but they are buffering, not lost.

Diagnose

1. Confirm the blast radius is NATS only — /healthz 200, /readyz degraded (not not_ready), and /api/v1/system/status → only nats: "down". If /readyz is not_ready/503, it's PostgreSQL, a different incident.
2. Check nats01: tsh ssh [email protected], then systemctl status nats and ss -tlnp | grep 4222. If teleport can't reach it ("not found in inventory"), the VM is down → power-cycle it from the Hetzner console.
3. Reachability from the cluster (when teleport is the thing that's down):

   kubectl run natschk --rm -i --restart=Never --image=busybox:1.36 -n default -- \
     sh -c 'nc -w5 10.10.4.15 4222'   # INFO {...} = serving; refused/hang = down

Recover

1. Restore NATS — restart nats.service on nats01, or power-cycle the VM if it's unreachable. JetStream data lives on /var/lib/nats and survives a reboot; nats.service is enabled and auto-starts on boot.
2. The backend reconnects on its own (infinite background reconnect — no pod restart needed). Agents reconnect and drain their buffers.
3. Verify recovery: /readyz → ready, /api/v1/system/status → nats: "ok", the banner clears, and proxima_agents_online_total climbs back to the fleet count.

Time budget & data loss

Agents buffer ~1h. Recover within that window for zero loss. A longer outage drops the oldest buffered data (gaps in agent telemetry/inventory for the outage window) — but the API stays up the entire time either way.

Config note — keep NATS on the private path

The backend's PROXIMA_NATS_URL comes from the console-backend secret (Vault). Keep it on the private path nats://10.10.4.15:4222 (direct to nats01, like psql01) — not nats.prxm.uz, which routes the backend's NATS through the public Hetzner LB + istio gateway and adds the exact failure surface these outages hit. Changing it is a Vault edit + a pod restart (no manifest change).

Verified in production (game day, 2026-06-18)

We forced a real NATS outage in prod — pointed the backend's PROXIMA_NATS_URL at a closed port and let the blue-green rollout promote onto the NATS-down pods. Observed, end-to-end:

• /readyz reported degraded; /healthz stayed 200; /api/v1/system/status served; the API and UI kept working with the degraded banner shown.
• Restoring NATS (fixing the Vault value + restarting the pods) recovered it cleanly — /readyz back to ready, agents reconnected and drained their buffers, no data loss.

This is the behaviour that replaced the ~19h outage where a NATS failure 503'd the entire API.