NATS Security

Proxima Console uses a production-grade JWT/operator model for NATS authentication, issuing per-agent credentials with scoped permissions.

Why Per-Agent JWT?

Each agent receives a unique JWT with permissions scoped to its own subjects. This ensures that if one agent is compromised, the attacker cannot publish or subscribe to any other agent's NATS subjects.

Key security properties:

Property	Per-Agent JWT
Authentication	Unique nkey + JWT per agent
Authorization	Scoped to agent-specific subjects only
Revocation	Revoke a single agent (instant disconnect)
Blast radius	Single agent compromised
Key storage	Nkey seed in agent state.json (0600); operator/account seeds in Vault
Transport	Mandatory mutual TLS 1.2+

Architecture

NATS Account Model

JWT mode uses the NATS operator/account/user hierarchy:

Level	Name	Purpose
Operator	proxima-operator	Root of trust. Signs account JWTs. Nkey seed stored in Vault.
Account	proxima	Application traffic. All agent and backend users belong here.
Account	SYS	NATS system account. Used for admin operations (JWT revocation push).
User	backend	Backend service user with full proxima.> permissions.
User	sys-admin	System admin user for $SYS.> operations.
User	agent-{uuid}	Per-agent user with scoped permissions (one per enrolled agent).

Per-Agent User Permissions

Each agent JWT is scoped to subjects containing its own client, environment, and agent identifiers:

Direction	Subjects	Purpose
Publish	proxima.{client}.{env}.{agent_id}.>	Telemetry (inventory, metrics, heartbeat, processes, changes), plus tunnel/SSH outbound bytes
Publish	proxima.system.agent.renew	JWT renewal requests
Publish	proxima.system.agent.renew.nonce	Fetch a server-issued single-use renewal challenge nonce (AUTH F2 — replay freshness)
Publish	proxima.system.agent.config.{agent_id}	Config fetch request-reply at startup (per-agent subject — the broker enforces identity; the worker derives the requesting agent from the subject's last token, so an agent can only fetch its own config)
Publish	proxima.system.config.apply.{agent_id}	Config apply ACK (success/failure per config type)
Subscribe	proxima.system.commands.{agent_id}	Commands from backend (config updates, terminal, runbooks)
Subscribe	proxima.system.file.{agent_id}.>	File transfer data (upload chunks, download acks)
Subscribe	proxima.{client}.{env}.{agent_id}.tunnel.in.>	Inbound tunnel data from backend (port forwarding)
Subscribe	proxima.{client}.{env}.{agent_id}.ssh.in.>	Inbound SSH bytes from the backend relay (OpenSSH ProxySSH)
Subscribe	_INBOX.{agent_id}.>	Per-agent reply inbox for request-reply. The agent sets a matching custom inbox prefix (nats.CustomInboxPrefix) on its connection, so a compromised agent cannot subscribe to (or eavesdrop on) another agent's reply inbox — the shared _INBOX.> wildcard is not granted.
Response	Max 1 message, 30s expiry	Limits reply scope

All other subjects are denied by default. An agent cannot publish to or subscribe to another agent's subjects.

Infrastructure Bootstrap

Bootstrap is a one-time process that creates the operator, accounts, and backend credentials.

Prerequisites

• HashiCorp Vault running and unsealed
• NATS server not yet started in JWT mode

Step 1: Initialize Vault

./scripts/vault-init.sh

This script configures Vault with:

Component	Details
PKI engine	Root CA (EC P-256), nats-server role for server certificates (30-day max TTL)
KV v2 engine	At secret/, stores nkey seeds under proxima/nats/ prefix
AppRole auth	proxima-backend role with 1h token TTL, 4h max TTL
Policy	proxima policy granting KV read/write access

The script outputs VAULT_ROLE_ID and VAULT_SECRET_ID for the backend.

Step 2: Generate NATS JWTs

make nats-jwt-init

Runs backend/cmd/nats-jwt-init/main.go, which:

1. Generates (or loads from Vault) nkey pairs for operator, proxima account, and SYS account
2. Creates self-signed operator JWT
3. Creates account JWTs signed by the operator (with JetStream enabled)
4. Creates backend user credentials with full proxima.> permissions
5. Creates sys-admin user credentials with $SYS.> permissions
6. Generates resolver_preload.conf with account public key → JWT mappings for NATS bootstrap
7. Templates the system_account value in nats-server-jwt.conf with the current SYS public key

Output files:

File	Purpose	Used by
infra/nats/jwt/operator.jwt	Operator identity	NATS server
infra/nats/jwt/proxima.jwt	Proxima account identity	NATS server (resolver preload)
infra/nats/jwt/sys.jwt	SYS account identity	NATS server (resolver preload)
infra/nats/jwt/resolver_preload.conf	Account JWT preload config	NATS server config include
infra/nats/creds/backend.creds	Backend user JWT + nkey seed	Backend service
infra/nats/creds/sys-admin.creds	Sys-admin JWT + nkey seed	Backend (revocation operations)

Vault KV paths:

Path	Contents
secret/data/proxima/nats/operator	Operator nkey seed
secret/data/proxima/nats/account-proxima	Proxima account nkey seed
secret/data/proxima/nats/account-SYS	SYS account nkey seed

Step 3: Issue NATS Server TLS Certificate

./scripts/nats-cert-issue.sh [common_name] [ttl]
# Defaults: CN="nats.proxima.local", TTL="720h" (30 days)

Output files in infra/nats/certs/:

File	Permissions	Purpose
server.pem	644	Server TLS certificate
server-key.pem	600	Server private key
ca.pem	644	CA certificate (agents use this for verification)

Step 4: Start NATS

The NATS server loads the operator JWT, preloads account JWTs, and enables TLS with mutual authentication using the nats-server-jwt.conf configuration.

Step 5: Start the Backend

Configure the backend with:

PROXIMA_NATS_CREDS_FILE=infra/nats/creds/backend.creds
PROXIMA_NATS_SYS_CREDS_FILE=infra/nats/creds/sys-admin.creds
PROXIMA_NATS_CA_FILE=infra/nats/certs/ca.pem
VAULT_ADDR=http://localhost:8200
VAULT_ROLE_ID=<from vault-init output>
VAULT_SECRET_ID=<from vault-init output>

Deployment: Local Dev / Bootstrap

In local development the bootstrap writes files under the repo's infra/nats/ directory and Docker Compose mounts them into the NATS container:

Host path	Container path
infra/nats/nats-server-jwt.conf	/etc/nats/nats-server-jwt.conf
infra/nats/jwt/	/etc/nats/jwt/
infra/nats/creds/	/etc/nats/creds/
infra/nats/certs/	/etc/nats/certs/

Deployment: Production (Kubernetes / ArgoCD)

Production runs on the proxima-production Kubernetes cluster (GitOps via ArgoCD). NATS JWT seeds and the backend/sys-admin credentials are not copied as files: they live in Vault and are injected into the NATS and backend pods by the External-Secrets Operator (ESO). Nothing sensitive lives in manifests, and there is no /opt/proxima box, no Docker Compose, and no systemctl. Rotation flows through Vault + ArgoCD (re-sync the affected workloads), not file copies.

Regenerating JWTs

If you need to regenerate JWTs (e.g., after key rotation or accidental deletion), follow this procedure. The nkey seeds in Vault are preserved, so account public keys remain the same and existing agent credentials continue to work.

warning

Regenerating JWTs requires restarting NATS. All connected agents will briefly disconnect and automatically reconnect. Plan for a short maintenance window.

Local dev / bootstrap:

# 1. Ensure Vault is unsealed and accessible
export VAULT_ADDR=http://127.0.0.1:8200 VAULT_TOKEN=<token>

# 2. Regenerate JWTs, creds, resolver_preload.conf, and nats-server-jwt.conf
make nats-jwt-init

# 3. Restart NATS (Docker Compose) to pick up new JWTs
docker restart proxima-nats

# 4. Restart the backend to use new credentials, then verify it reconnected

Production (k8s / ArgoCD): regeneration writes the refreshed JWTs/creds back to Vault; ESO syncs them into the NATS and backend pods, and an ArgoCD re-sync (or pod restart) of the affected workloads picks them up. There are no file copies, docker restart, or systemctl/journalctl steps.

What make nats-jwt-init does on re-run:

• Loads existing nkey seeds from Vault (does not generate new keys)
• Creates fresh JWTs with new timestamps (signed by the same keys)
• Regenerates resolver_preload.conf with updated account JWTs
• Updates system_account in nats-server-jwt.conf (handles both placeholder and existing values)
• Regenerates backend and sys-admin credential files with new user nkeys

What happens to connected agents:

• Agents disconnect briefly when NATS restarts
• Agents reconnect automatically (NATS client has built-in reconnection)
• Existing agent JWTs remain valid (same account keys, no revocations added)
• Agent JWT renewal continues to work (backend has new account signing keys from Vault)

Agent Enrollment Flow

When an agent starts for the first time, it performs a single-call enrollment with the backend over HTTPS:

Step-by-Step

1. Enroll: Agent sends POST /api/v1/agents/enroll with the install token and optionally agent_type ("host", "collector", or "k8s-node-monitor", default "host"). The backend validates the token (SHA-256 hash lookup, not expired), generates an nkey pair server-side, resolves client and environment from the token, upserts an agent record (not a host), creates a scoped JWT named agent-{uuid}, and returns everything the agent needs.

2. Persist: Agent writes state.json containing the JWT, nkey seed, NATS URL, agent_id, agent_type, and client/environment slugs. If a CA PEM is included in the response (private CA deployments), it writes ca.pem alongside.

3. Connect: Agent connects to NATS using nats.UserJWT() callbacks that re-read the current JWT + nkey seed from state.json on every (re)connect. This is deliberate: a JWT renewed in state is then picked up on the next connect, so the renewer can apply a fresh credential to the live connection (see JWT Renewal) instead of the connection being pinned to the JWT captured at process start.

4. Host record creation (agent_type=host only): For a host agent that supplies a hostname, the backend pre-links the hosts row at enrollment (best-effort INSERT … ON CONFLICT, agent_status='online') so per-host config resolves on the agent's very first config fetch; the InventoryWorker later upserts the same row (preserving the agent_id via COALESCE). When no hostname is supplied (or for collectors, which own no host row), the host record is instead created on the first inventory message.

5. Subsequent starts: If state.json exists, the agent skips enrollment entirely and connects directly; credential refresh is handled by the renewal loop, not by re-enrolling. A second enrollment of an already-enrolled identity (same environment_id, agent_type, name) that maps to a different agent is rejected with 409 Conflict — identity-takeover protection (AUTH H-1): a fresh install token cannot rebind an existing agent's nkey. Re-keying requires renewing the existing agent's credentials or deleting it first.

Install Token Properties

Property	Value
Format	32 cryptographically random bytes, hex-encoded
Storage	SHA-256 hash in database (plaintext shown once at creation)
Usage	Single-use by default (max_uses=1). Reuse is an explicit opt-in at creation: a positive max_uses bounds it to N enrollments, and max_uses=0 opts into unlimited use (e.g. a fleet/reusable token). The single-use default ensures one leaked token cannot enroll a whole fleet.
Default expiry	24 hours
Maximum expiry	30 days
Enrollment tracking	enrolled_by column on agents table
API	POST /api/v1/clients/{slug}/install-tokens (create), GET (list), DELETE (revoke)

JWT Renewal

Agent JWTs have a 7-day default TTL. A background renewal loop ensures uninterrupted connectivity:

Renewal Details

Parameter	Value
Check interval	Every 1 hour
Renewal threshold	When remaining TTL < 50% of original (e.g., < 3.5 days for a 7-day JWT)
Challenge nonce	Before renewing, the agent fetches a server-issued single-use nonce (NATS proxima.system.agent.renew.nonce, or HTTPS POST /api/v1/agents/credentials/renew/nonce) — gives each renewal replay freshness (AUTH F2). Older backends without the endpoint fall back to a legacy nonce-less path.
Primary channel	NATS request-reply on proxima.system.agent.renew (30s timeout)
Fallback channel	HTTPS POST /api/v1/agents/credentials/renew
Proof of identity	Agent signs the server-issued nonce with its nkey seed from state; the backend verifies the signature and cross-checks the nkey against the enrolled agent
Identity lookup	Backend extracts agent_id from JWT name (agent-{uuid}) and validates against the agents table
Legacy JWT rejection	JWTs with host-{uuid} name prefix are rejected — agents must re-enroll
On success	state.json atomically updated, then the agent calls ForceReconnect() so the live connection re-reads state and presents the new JWT immediately (the nats.UserJWT callbacks read from state on each connect). Without this the connection would stay pinned to the old JWT until it expired and the server dropped it.
On failure	Logged and retried at the next hourly check

Agent Lifecycle Management

Administrators can manage agent connectivity through three actions. These operations act on the agent identity (the agents table), not the host record.

Action	API Endpoint	DB Effect	JWT Effect	NATS Effect
Deactivate	POST /api/v1/hosts/{id}/deactivate	status = 'inactive' on agent	Renewal blocked	JWT expires naturally (up to 7 days)
Activate	POST /api/v1/hosts/{id}/activate	status = 'active' on agent	Renewal restored	Agent reconnects normally on next renewal
Revoke	POST /api/v1/hosts/{id}/revoke	status = 'inactive' on agent	Revocation entry added to account JWT	Immediate disconnect

Revocation

When a host is revoked, the backend:

1. Adds a revocation entry (the host's nkey public key + timestamp) to the proxima account JWT
2. Signs the updated account JWT with the account nkey
3. Pushes the updated JWT to the NATS server via $SYS.REQ.CLAIMS.UPDATE (using sys-admin credentials)
4. The NATS server immediately rejects the agent's connection and any reconnection attempts

This provides instant, server-enforced disconnection without waiting for the JWT to expire.

File Layout Reference

Agent Files

File	Path	Permissions	Purpose	Created
state.json	{data_dir}/state.json	0600	All agent state: JWT, nkey seed, agent_id, agent_type, client/environment slugs, NATS URL, version	After enrollment
ca.pem	{data_dir}/ca.pem	0644	NATS server CA certificate (only with private CA)	After enrollment (if private CA)

Default data_dir: /var/lib/proxima-agent

Infrastructure Files

File	Path	Purpose
operator.jwt	infra/nats/jwt/operator.jwt	NATS operator identity
proxima.jwt	infra/nats/jwt/proxima.jwt	Proxima account JWT
sys.jwt	infra/nats/jwt/sys.jwt	SYS account JWT
resolver_preload.conf	infra/nats/jwt/resolver_preload.conf	Account JWT preload for NATS config
backend.creds	infra/nats/creds/backend.creds	Backend user credentials
sys-admin.creds	infra/nats/creds/sys-admin.creds	Sys-admin user credentials
server.pem	infra/nats/certs/server.pem	NATS server TLS certificate
server-key.pem	infra/nats/certs/server-key.pem	NATS server TLS private key
ca.pem	infra/nats/certs/ca.pem	TLS CA certificate

Security Properties

• Server-side key generation: The nkey pair is generated on the backend. The seed is sent once over TLS during enrollment, which is equivalent security to sending the JWT over the same connection.
• Single-use install tokens by default: A token is consumed by one enrollment unless created with a higher max_uses bound (or max_uses=0 for an unlimited fleet/reusable token). Authorization is the install token itself — possession of a valid, unconsumed token grants enrollment — so the single-use default limits the blast radius of a leaked token.
• Default-deny permissions: Each agent JWT allows only its own agent-scoped subjects. Publishing to another agent's subjects is rejected by the NATS server.
• Identity-takeover protection (AUTH H-1): The first enrollment binds the agent's nkey. A second enrollment of an already-enrolled (environment_id, agent_type, name) mapping to a different agent is rejected with 409 Conflict — a fresh install token cannot rebind (hijack) an existing agent's NATS identity. Pod restarts and re-deployments are safe because the agent reuses its persisted state.json and skips enrollment entirely (credential refresh is handled by renewal). Re-keying requires renewing the existing agent's credentials or deleting it first.
• Mandatory TLS: TLS 1.2+ with strong cipher suites is required for all NATS connections. The agent verifies the server certificate against the Vault-issued CA.
• Vault seed storage: Operator and account nkey seeds are stored in Vault KV v2, not on the filesystem. Only the bootstrap tool and backend access them.
• Rate limiting: Credential endpoints are rate-limited per IP (configurable via PROXIMA_CREDENTIAL_RATE_LIMIT, default 10/min) to prevent brute-force attacks on install tokens.
• Immediate revocation: Compromised agents can be instantly disconnected by pushing a revocation entry to the NATS server, without waiting for JWT expiry.

Configuration Reference

Backend Environment Variables

Variable	Default	Description
PROXIMA_NATS_CREDS_FILE	—	Path to backend NATS credentials file
PROXIMA_NATS_SYS_CREDS_FILE	—	Path to sys-admin NATS credentials file (for JWT revocation)
PROXIMA_NATS_CA_FILE	—	Path to NATS TLS CA certificate
VAULT_ADDR	—	Vault server address
VAULT_TOKEN	—	Vault root token (from vault-init.sh output or infra/vault/.vault-keys)
VAULT_ROLE_ID	—	Vault AppRole role ID (production — use instead of VAULT_TOKEN)
VAULT_SECRET_ID	—	Vault AppRole secret ID (production — use instead of VAULT_TOKEN)

Agent Environment Variables

Variable	Default	Description
PROXIMA_BACKEND_URL	—	Backend HTTPS URL for enrollment and renewal.
PROXIMA_INSTALL_TOKEN	—	Install token for enrollment.
PROXIMA_NATS_CA_FILE	—	Path to NATS TLS CA certificate
PROXIMA_AGENT_DATA_DIR	/var/lib/proxima-agent	Directory for persistent agent data (state.json)

note

PROXIMA_CLIENT_SLUG and PROXIMA_ENVIRONMENT_SLUG are not required. The client and environment are determined during enrollment from the install token's associated client.