Agent Troubleshooting

This page covers common issues, edge cases, and recovery procedures for the Proxima Agent.

Quick Diagnostics

# Agent status and recent logs
systemctl status proxima-agent
journalctl -u proxima-agent --since "10 min ago" -f

# Check state.json
cat /var/lib/proxima-agent/state.json | jq .

# Check agent version
proxima-agent --version

# Run preflight checks
proxima-agent preflight --backend-url https://api-console.prxm.uz

Enrollment Issues

Agent crash-loops on first start

Symptom: Agent starts, fails enrollment, systemd restarts it repeatedly.

Diagnosis:

journalctl -u proxima-agent | grep "enrollment failed"

Common causes:

Log message	Cause	Fix
`http 401: invalid install token`	Token is wrong, expired, or deleted	Generate a new token in the Console UI
`http request: dial tcp: connection refused`	Backend is unreachable	Check `PROXIMA_BACKEND_URL` in systemd unit, verify backend is running
`enrollment canceled: context deadline exceeded`	Backend responding too slowly	Check backend health, network connectivity

Note: The agent retries enrollment with exponential backoff (5s, 10s, 20s, 40s, 60s, up to 5 minutes). It does not crash-loop — it stays active and keeps retrying. Check logs for retry_in to confirm backoff is working.

Re-enrollment is rejected with 409 Conflict

Symptom: After wiping state.json and restarting, enrollment fails with http 409: an agent with this name is already enrolled in this environment.

Why: Agent identity is keyed by agent_id and protected against takeover (AUTH H-1). The backend will not rebind the NATS key of an already-enrolled (environment_id, agent_type, name) — re-enrolling that identity returns 409 rather than silently reusing or overwriting the existing agent record.

Fix: Don't wipe state.json to rotate credentials — the agent renews its own NATS JWT automatically (NATS-first, HTTPS fallback). If you genuinely need to re-key a host (e.g., a compromised seed), remove the existing agent from the Console first, then restart so it enrolls fresh:

systemctl restart proxima-agent

If a stale host record was left behind under a previous identity, deactivate it via the API:

curl -X POST https://api-console.prxm.uz/api/v1/hosts/<old-host-id>/deactivate \
  -H "Authorization: Bearer <token>"

Expired install token

Symptom: http 401: install token expired in logs.

Fix: Generate a new token (TTL up to 30 days). Tokens are single-use by default — pass "max_uses": 0 for an unlimited/fleet token (or "max_uses": N for a bounded count):

# Via API
curl -X POST https://api-console.prxm.uz/api/v1/clients/<slug>/install-tokens \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{"description":"fleet install","expires_in":2592000,"max_uses":0}'

Update the agent's environment:

# Edit the systemd environment file
vim /opt/proxima/.env  # or wherever PROXIMA_INSTALL_TOKEN is set
systemctl restart proxima-agent

State & Credential Issues

Corrupt state.json

Symptom: Agent logs invalid state.json, will re-enroll on startup.

What happens: The agent validates state.json on load. If any required field is missing or malformed (bad UUID, missing JWT, wrong nkey seed prefix), it automatically deletes the file and re-enrolls. No manual intervention needed.

Validated fields:

agent_id — must be a valid UUID
agent_type — must be a known type (host, collector, or k8s-node-monitor)
nkey_seed — must start with SU (NATS user seed prefix)
user_jwt — must be non-empty
client_slug, environment_slug, nats_url — must be non-empty

Manual recovery (if auto-recovery fails):

systemctl stop proxima-agent
rm /var/lib/proxima-agent/state.json
# Ensure PROXIMA_INSTALL_TOKEN is set in the environment
systemctl start proxima-agent

JWT expired — agent can't communicate

Symptom: Agent is running but not sending data. Logs show JWT renewal failed via all channels.

How it should work: The agent checks JWT expiry hourly and renews when less than 50% TTL remains (default: renews after 3.5 days of a 7-day JWT). Renewal uses NATS first, then HTTPS fallback.

If both renewal channels fail for 7+ days: The JWT expires. The agent attempts automatic re-enrollment using the install token from the environment variable. Check logs for:

JWT expired and renewal failed, attempting re-enrollment

If re-enrollment also fails (token expired or missing):

# Generate a new install token, update the env, restart
# The agent will re-enroll on startup
systemctl stop proxima-agent
rm /var/lib/proxima-agent/state.json
# Update PROXIMA_INSTALL_TOKEN in /opt/proxima/.env
systemctl start proxima-agent

If renewal fails but JWT is not yet expired:

Check backend health (curl https://api-console.prxm.uz/healthz)
Check NATS connectivity (nats server ping from the host)
Check TLS certificate expiry on the NATS server
The agent will retry hourly — no action needed unless the JWT is about to expire

NATS TLS certificate expired

Symptom: Agent logs tls: failed to verify certificate: x509: certificate has expired.

Fix: Reissue the NATS server certificate:

# On the backend/infrastructure host
source /opt/proxima/infra/vault/.vault-keys
export VAULT_ADDR=http://127.0.0.1:8200 VAULT_TOKEN
./scripts/nats-cert-issue.sh
cp infra/nats/certs/* /opt/proxima/infra/nats/certs/
docker restart proxima-nats
systemctl restart proxima-backend

The agent will automatically reconnect once NATS is back with a valid certificate.

Agent has ca.pem but NATS CA was rotated

Symptom: Agent fails to reconnect after NATS CA rotation.

Fix: The agent's ca.pem was written during enrollment. If the CA was rotated:

systemctl stop proxima-agent
rm /var/lib/proxima-agent/state.json /var/lib/proxima-agent/ca.pem
systemctl start proxima-agent
# Agent re-enrolls and gets the new CA

NATS Connectivity Issues

Agent connected but not sending data

Symptom: Host shows as "online" (heartbeat works) but no metrics, inventory, or processes.

Diagnosis:

# Check scheduler is running
journalctl -u proxima-agent | grep "scheduler"

# Check for outbox full (data loss under NATS pressure)
journalctl -u proxima-agent | grep "outbox full"

If outbox is full: The agent's publish queue (256 messages) is saturated. This indicates the NATS connection is slow or blocked. Check:

NATS server health and JetStream status
Network latency between agent and NATS
JetStream storage usage (may need cleanup)

Permission violation on NATS publish

Symptom: nats async error: permissions violation in agent logs.

Cause: The agent's JWT doesn't have permission for the subject it's trying to publish to. This happens if:

The host was moved to a different client/environment but the JWT still has the old scopes
The JWT was manually tampered with

Fix: Force JWT renewal or re-enroll:

systemctl stop proxima-agent
rm /var/lib/proxima-agent/state.json
systemctl start proxima-agent

Update Issues

Self-update command failed

Symptom: Backend sent self_update command but agent didn't update.

Diagnosis:

journalctl -u proxima-agent | grep "self-update\|self_update"

Common causes:

Download failed (network issue, R2 unreachable)
Checksum mismatch (corrupted download)
Permission denied (binary not writable)
Disk full

malformed checksum file — fixed in v0.6.1+

Older agents failed self-update with malformed checksum file because the verifier required the two-field sha256sum form while the release pipeline publishes a bare hash. This was fixed in v0.6.1+: the agent now accepts the published bare-hash checksum. An agent still hitting this error is on an older build — reinstall it once via the install script to pick up the fix, after which self-updates (including console fleet rollouts) work normally.

Manual update:

proxima-agent self-update          # update to latest
proxima-agent self-update v0.5.1   # pin specific version
systemctl restart proxima-agent

For the recommended console-driven flow, see Updating Agents (Fleet Management).

Agent running old version after update

Symptom: proxima-agent --version shows old version.

Cause: The self_update handler swaps the binary and calls systemctl restart, but if the restart failed, the old process is still running.

Fix:

systemctl restart proxima-agent
proxima-agent --version

Data & Configuration Issues

Config not applied after push from backend

Symptom: Backend pushed a config update but agent behavior didn't change.

Diagnosis:

# Check config handler logs
journalctl -u proxima-agent | grep "config_update\|config applied\|config handler"

# Check config cache
cat /var/lib/proxima-agent/config.cache.json | jq .

The agent publishes a config apply ACK to proxima.system.config.apply.<agent_id>. Check the backend for the ACK status.

Agent collecting stale collector data

Symptom: Metrics reference collectors that no longer exist (e.g., a removed PostgreSQL instance).

Cause: Collector configs are pushed from the backend. If the backend still has the old config, the agent will keep trying.

Fix: Update the collector configuration in the Console UI. The change is pushed to the agent in real-time.

File Layout Reference

/usr/local/bin/proxima-agent              # Binary
/etc/systemd/system/proxima-agent.service # Systemd unit
/var/lib/proxima-agent/
  state.json          # All enrollment state (agent_id, JWT, nkey seed, slugs) — 0600
  ca.pem              # NATS CA certificate (only with private CA) — 0644
  config.cache.json   # Cached config from backend
  file_hashes.json    # Change detection baseline
  journald-cursor     # Log collection cursor
  logs/               # Log collection state

Environment Variables

Variable	Required	Description
`PROXIMA_BACKEND_URL`	Yes	Backend API URL for enrollment and renewal
`PROXIMA_INSTALL_TOKEN`	First boot	Install token — only needed if `state.json` doesn't exist. Also used for automatic re-enrollment if JWT expires.
`PROXIMA_LOG_LEVEL`	No	`debug`, `info` (default), `warn`, `error`
`PROXIMA_AGENT_DATA_DIR`	No	Default: `/var/lib/proxima-agent`
`PROXIMA_NATS_CA_FILE`	No	Override CA path (normally from enrollment)
`PROXIMA_NATS_URL`	No	Override NATS URL (normally from enrollment)

Recovery Decision Tree

Agent not sending data
├── systemctl is-active proxima-agent → inactive
│   └── Check: journalctl -u proxima-agent -e
│       ├── "no state.json and no install token" → Set PROXIMA_INSTALL_TOKEN, restart
│       ├── "enrollment failed, retrying" → Token issue or backend down, check retry_in
│       └── Other error → Check specific error message above
├── active, but no data in Console
│   ├── Check: journalctl | grep "outbox full" → NATS backpressure
│   ├── Check: journalctl | grep "permission" → JWT scope mismatch, re-enroll
│   └── Check: journalctl | grep "tls" → Certificate expired, reissue
└── active, data flowing, but wrong host
    └── Duplicate host_id → Deactivate old host via API

Quick Diagnostics​

Enrollment Issues​

Agent crash-loops on first start​

Re-enrollment is rejected with 409 Conflict​

Expired install token​

State & Credential Issues​

Corrupt state.json​

JWT expired — agent can't communicate​

NATS TLS certificate expired​

Agent has ca.pem but NATS CA was rotated​

NATS Connectivity Issues​

Agent connected but not sending data​

Permission violation on NATS publish​

Update Issues​

Self-update command failed​

Agent running old version after update​

Data & Configuration Issues​

Config not applied after push from backend​

Agent collecting stale collector data​

File Layout Reference​

Environment Variables​

Recovery Decision Tree​

Quick Diagnostics

Enrollment Issues

Agent crash-loops on first start

Re-enrollment is rejected with 409 Conflict

Expired install token

State & Credential Issues

Corrupt state.json

JWT expired — agent can't communicate

NATS TLS certificate expired

Agent has ca.pem but NATS CA was rotated

NATS Connectivity Issues

Agent connected but not sending data

Permission violation on NATS publish

Update Issues

Self-update command failed

Agent running old version after update

Data & Configuration Issues

Config not applied after push from backend

Agent collecting stale collector data

File Layout Reference

Environment Variables

Recovery Decision Tree