Skip to main content

Troubleshooting

Common issues and their solutions when operating Proxima Console.

Production runs on Kubernetes — use kubectl / ArgoCD, not the box

The application tier (backend, frontend, MCP, etc.) runs in the proxima-production Kubernetes cluster, namespace console-system. To diagnose production backend/server issues, use kubectl and ArgoCD, not systemctl / journalctl / docker compose on a box:

# Backend logs (production)
kubectl -n console-system logs deploy/console-backend -f
# Pod / Rollout status
kubectl -n console-system get pods -l app=console-backend
kubectl -n console-system argo rollouts get rollout console-backend
# ArgoCD app health & sync
argocd app get console-backend
# Exec into a pod (e.g. to reach Valkey via redis-cli)
kubectl -n console-system exec -it deploy/console-backend -- sh

The journalctl -u proxima-agent commands in the Agent section still apply — agents run on the managed hosts (systemd), not in the Console cluster. The systemctl / docker compose / journalctl -u proxima-backend / lsof-on-:8080 commands below describe the local-dev (or single-box) flow; the production equivalents are noted inline. The Docker Compose section at the end is local-dev only.

Agent​

Agent shows as "offline" despite running​

Symptoms: The agent process is running, but the Console shows the host as offline.

Possible causes:

  1. NATS connectivity — The agent cannot reach the NATS server.

    # Check NATS connection
    journalctl -u proxima-agent -f | grep -i "nats\|connect"

    Verify PROXIMA_NATS_URL is correct and the NATS server is reachable from the host.

  2. Stale timeout too aggressive — The backend marks agents offline after PROXIMA_AGENT_STALE_TIMEOUT_SECONDS (default: 300s) without a heartbeat.

    # Check heartbeat interval vs stale timeout
    # Heartbeat interval should be well below the stale timeout

  3. Clock skew — Large clock differences between the agent host and backend can cause heartbeat timestamps to appear stale. Ensure NTP is configured.

  4. Enrollment failure — If the install token is expired or already used, enrollment fails. Check agent logs for enrollment errors.

    journalctl -u proxima-agent | grep -i "enroll\|credential\|token\|auth"

  5. k8s node: heartbeats arrive but status stays offline — for k8s-node-monitor hosts, node liveness (hosts.agent_status) is separate from Kubernetes readiness (k8s_nodes.status), so a Ready node can show offline. A current backend self-heals this (a heartbeat restores online); if last_seen_at is fresh but the host is still offline, confirm the deployed backend includes the self-heal fix. If heartbeats are not arriving, the agent's live NATS connection may be pinned to an expired JWT — restart the agents to reload the renewed credential from state:

    kubectl -n proxima-system rollout restart \
      daemonset/proxima-agent-node-agent deployment/proxima-agent-cluster-agent

    See the observability runbook for the connz JWT-expiry diagnostic.

Agent not collecting metrics​

Symptoms: Host shows online but no metrics appear in the dashboard.

Check:

  1. Verify the agent is running a version that includes the metrics collector (v0.2.0+).
  2. Check agent logs for collection errors: 
    journalctl -u proxima-agent | grep -i "metric\|collect\|error"
  3. Ensure the NATS TELEMETRY stream exists. The backend creates it on startup, but if NATS was restarted, streams may need to be recreated: 
    make docker-up  # Recreates NATS streams

PostgreSQL collector errors​

Symptoms: pg_* metrics are missing; agent logs show connection errors.

Check:

  1. Verify PROXIMA_PG_DSN is set and the connection string is valid.
  2. Ensure the monitoring user has the pg_monitor role: 
    SELECT rolname FROM pg_roles WHERE pg_has_role('proxima_monitor', oid, 'member');
  3. Check firewall rules — the agent needs to reach PostgreSQL on the configured port.
  4. If pg_stat_statements metrics are missing, verify the extension is installed: 
    SELECT * FROM pg_extension WHERE extname = 'pg_stat_statements';

Backend​

Backend fails to start​

Symptoms: the backend fails on startup — in production the pod crash-loops (CrashLoopBackOff) and the blue-green preview never promotes; in local dev make run-backend exits.

In production, inspect the pod first:

kubectl -n console-system get pods -l app=console-backend
kubectl -n console-system logs deploy/console-backend --previous   # last crashed container
kubectl -n console-system describe pod -l app=console-backend      # events, probe failures

Common causes:

  1. Missing/invalid PROXIMA_DB_URL — This is the only strictly required variable. In production it is injected via External-Secrets from Vault (vault.prxm.uz); check the synced Secret and the ExternalSecret status:

    kubectl -n console-system get externalsecret
    kubectl -n console-system get secret console-backend -o jsonpath='{.data.PROXIMA_DB_URL}' | base64 -d

    In local dev: echo $PROXIMA_DB_URL.

  2. Database not reachable — Production points at psql01 (10.10.3.11:30034). Check connectivity from inside the cluster:

    kubectl -n console-system exec -it deploy/console-backend -- sh -c 'psql "$PROXIMA_DB_URL" -c "SELECT 1"'

    In local dev: psql "$PROXIMA_DB_URL" -c "SELECT 1".

  3. Pending migrations — In production, migrations run as the console-migrate ArgoCD PreSync Job; if it fails, the backend sync is blocked. Check the Job:

    kubectl -n console-system get jobs -l app=console-migrate
    kubectl -n console-system logs job/console-migrate

    In local dev, run them yourself: make migrate-up.

  4. Port conflict (local dev only) — Another process is using port 8080 (or the configured PROXIMA_SERVER_PORT):

    lsof -i :8080

    In Kubernetes each pod has its own network namespace, so a host port conflict does not apply.

  5. Bind address (production) — If readiness probes fail with connection-refused, confirm PROXIMA_SERVER_BIND=0.0.0.0 is set on the Rollout (defaults to 127.0.0.1, which kubelet probes and the Service cannot reach). See Environment Variables.

NATS consumers not processing messages​

Symptoms: Agents publish messages but the backend doesn't process them (inventory/metrics not updating).

In production, NATS still runs on the box at nats://nats.prxm.uz:4222 (interim), while the backend runs in Kubernetes — so check the box for NATS health and kubectl logs for the consumer side.

Check:

  1. Verify NATS is running and the backend is connected (the varz/connz monitoring endpoint is on the NATS box): 
    curl -s http://localhost:8222/varz | jq .connections   # on the NATS box / local dev
  2. Check JetStream stream health (NATS CLI against the box, or localhost in local dev): 
    nats stream ls
    nats consumer ls EVENTS
    nats consumer ls TELEMETRY
  3. Look for consumer errors in backend logs: 
    # Production
    kubectl -n console-system logs deploy/console-backend | grep -i "consumer\|subscribe\|nats"
    # Local dev / single box
    journalctl -u proxima-backend | grep -i "consumer\|subscribe\|nats"

VictoriaMetrics import failures​

Symptoms: Metrics are collected but not appearing in dashboards.

Check:

  1. Verify VictoriaMetrics is reachable. In production, Console talks to the in-cluster VM cluster through vmauth-proxy.monitoring:8427 (authenticated as vmuser-console, projectID 42); in local dev it's the docker-compose vmselect on :8481: 
    # Production (from inside the cluster)
    kubectl -n console-system exec -it deploy/console-backend -- \
      sh -c 'curl -s "$VM_SELECT_URL/health"'
    # Local dev
    curl -s http://localhost:8481/health
  2. Check VM_INSERT_URL / VM_SELECT_URL are correct (production: vmauth-proxy.monitoring:8427), and that VM_AUTH_USERNAME / VM_AUTH_PASSWORD / VM_PROJECT_ID are set so vmauth-proxy accepts the request. See VictoriaMetrics Architecture.
  3. Look for import errors in backend logs: 
    # Production
    kubectl -n console-system logs deploy/console-backend | grep -i "vmclient\|import\|victoria\|vmauth"
    # Local dev / single box
    journalctl -u proxima-backend | grep -i "vmclient\|import\|victoria"
  4. Verify the client has a vm_account_id (production DB is psql01): 
    SELECT id, name, vm_account_id FROM clients;

API returns 401 Unauthorized​

Check:

  1. Verify the Authorization: Bearer <jwt> header contains a valid JWT token, or the X-API-Key header contains a valid service account key.
  2. Check that the JWT token has not expired (default TTL is 15 minutes).
  3. For service accounts, ensure the API key is active and has not been deactivated.
  4. Health endpoints (/healthz, /readyz, /metrics) do not require authentication.

Account locked out after failed login attempts​

Symptoms: Login returns 429 or a lockout message even with the correct password.

Progressive lockout is triggered after 5 failed attempts (1min), escalating to 1h at 20+ failures. Lockout state is stored in Valkey.

Resolution:

  1. Wait for the lockout duration to expire, then try again.
  2. To clear immediately, delete the lockout key in Valkey: 
    # Production (in-cluster Valkey)
    kubectl -n console-system exec -it deploy/console-valkey -- redis-cli DEL "auth:lockout:[email protected]"
    # Local dev (docker-compose)
    docker exec proxima-valkey redis-cli DEL "auth:lockout:[email protected]"
  3. A successful login resets the failure counter.

MFA not working​

Symptoms: TOTP codes are rejected during login or MFA setup confirmation.

Check:

  1. Clock skew — TOTP codes are time-based (30-second window with 1-period skew). Ensure the server and the user's authenticator device have accurate time (NTP synced).
  2. Wrong account — If the user has multiple TOTP entries, they may be using the wrong one. The issuer is "Proxima Console".
  3. Setup not confirmed — If MFA setup was started but not confirmed (POST /auth/mfa/confirm), the pending secret expires after 10 minutes in Valkey. The user needs to restart MFA setup.
  4. Recovery codes — If TOTP is unavailable, users can use a recovery code (xxxx-xxxx format) in the code field. Each recovery code is single-use.

To reset MFA for a user (admin action):

UPDATE users SET mfa_enabled = FALSE, mfa_secret = NULL, recovery_codes = NULL
WHERE email = '[email protected]';

The user will be prompted to set up MFA again on next login.

Google SSO not working​

Symptoms: "Sign in with Google" button doesn't appear, or the SSO flow fails.

Check:

  1. Button not showing — The frontend queries GET /api/v1/auth/config. If google_sso_enabled is false, ensure PROXIMA_GOOGLE_CLIENT_ID is set in the backend environment.
  2. Redirect fails — Verify PROXIMA_BACKEND_URL is set correctly (the callback URL is constructed from it).
  3. "Access denied" from Google — Check that:
    • The Google OAuth redirect URI matches {PROXIMA_BACKEND_URL}/api/v1/auth/google/callback
    • The user's email domain matches PROXIMA_GOOGLE_ALLOWED_DOMAIN
    • The Google Cloud OAuth consent screen is configured for the right user type
  4. "Invalid state" error — The CSRF state token may have expired (5-minute TTL in Valkey). This can happen if the user takes too long to authenticate with Google. Try again.
  5. Routes return 404 — Google SSO endpoints are only registered when PROXIMA_GOOGLE_CLIENT_ID is configured. The backend must restart to pick it up: in production, update the value in Vault and let External-Secrets re-sync, then roll the backend (kubectl -n console-system rollout restart rollout/console-backend or re-sync via ArgoCD); in local dev, restart make run-backend.

Password reset email not received​

Symptoms: User clicks "Forgot password" but no email arrives.

Check:

  1. The POST /api/v1/auth/forgot-password endpoint always returns 200 regardless of whether the email exists (by design, to prevent email enumeration). Verify the email address is registered.
  2. Check email provider configuration: 
    # SMTP
    echo $PROXIMA_EMAIL_PROVIDER   # should be "smtp"
    echo $PROXIMA_SMTP_HOST        # must be set
    echo $PROXIMA_SMTP_USERNAME    # must be set

    # Or Resend
    echo $PROXIMA_EMAIL_PROVIDER   # should be "resend"
    echo $PROXIMA_RESEND_API_KEY   # must be set
  3. Check backend logs for email sending errors: 
    # Production
    kubectl -n console-system logs deploy/console-backend | grep -i "email\|smtp\|resend"
    # Local dev / single box
    journalctl -u proxima-backend | grep -i "email\|smtp\|resend"
  4. Verify PROXIMA_FRONTEND_URL is set correctly — this is used in email links.
  5. Check spam/junk folders.

Token refresh fails​

Symptoms: The frontend logs the user out unexpectedly, or refresh requests return 401.

Check:

  1. Session revoked — If the user (or an admin) revoked the session, refresh tokens are invalidated. The user needs to log in again.
  2. Refresh token expired — Default TTL is 7 days (PROXIMA_JWT_REFRESH_TTL). Check if the user has been inactive for longer.
  3. User disabled — If an admin disabled the user account, token refresh is rejected.
  4. The frontend proactively refreshes tokens 2 minutes before expiry. If the refresh fails, a CustomEvent("auth:session-expired") is dispatched and the user is redirected to login.

API returns 403 Forbidden​

Symptoms: Authenticated user gets 403 on specific endpoints.

Check:

  1. The user needs the appropriate permission for the action. Check their role permissions: 
    curl -H "Authorization: Bearer $TOKEN" https://api-console.prxm.uz/api/v1/auth/me | jq .data.client_access
  2. For client-scoped resources, the user must have access to that specific client (via team membership or direct assignment).
  3. System roles (Super Admin, Admin, etc.) cannot be modified or deleted — these return 403 by design.
  4. Service accounts can only access clients in their client_scope.

Frontend​

Frontend shows "Failed to fetch" errors​

Symptoms: The Console UI shows error messages when loading data.

Check:

  1. Verify VITE_API_BASE_URL points to the correct backend URL.
  2. Check CORS — the backend's PROXIMA_CORS_ORIGINS must include the frontend's origin.
  3. If using basic auth (nginx), ensure the frontend requests pass through correctly.

Charts show no data​

Symptoms: Dashboard loads but charts are empty.

Check:

  1. Verify the host is online and the agent is sending metrics.
  2. Check the time range — the default is 1 hour. If the agent was recently started, try a shorter range.
  3. Open browser DevTools (Network tab) and check the API responses for the metrics endpoints.

Database​

Migration fails​

Symptoms: make migrate-up fails with errors.

Common fixes:

  1. Check the schema_migrations table for a dirty state:

    SELECT * FROM schema_migrations;

    If dirty = true, fix the issue manually and set dirty = false:

    UPDATE schema_migrations SET dirty = false WHERE version = <version>;

  2. For the TimescaleDB removal migration (000009), ensure TimescaleDB is not in use by any other application before running.

Test database issues​

If integration tests fail with database errors:

# Reset the test database completely
make test-db-reset

# Recreate from scratch
make test-db-setup

The test database (proxima_test) is separate from development (proxima).

Nginx Collector Errors​

Connection refused​

status: error
status_message: "stub_status request: Get \"http://127.0.0.1/nginx_status\": dial tcp 127.0.0.1:80: connect: connection refused"
  • Verify Nginx is running: systemctl status nginx
  • Check the url in the collector config matches the Nginx stub_status location
  • Ensure the port is correct

HTTP 403 or 404​

  • 403: The stub_status location block has allow/deny rules blocking the agent. Add allow 127.0.0.1;.
  • 404: The stub_status module is not enabled or the URL path is wrong. Verify with nginx -V 2>&1 | grep stub_status and check the location path.

Malformed response​

If the collector reports a parse error, the stub_status endpoint may be returning HTML instead of the expected plain-text format. Ensure the URL points directly to a stub_status location, not a proxy or redirect.

Docker Compose​

Services fail to start​

# Check status of all services
docker compose ps

# View logs for a specific service
docker compose logs -f postgres
docker compose logs -f nats
docker compose logs -f vmstorage

VictoriaMetrics data corruption​

If vmstorage fails after an unclean shutdown:

# Remove and recreate the volume (dev only — this deletes all metrics!)
docker compose down -v
docker compose up -d

Getting Help​