Rotating Agent Credentials

Each agent and collector connects to NATS with a per-agent JWT whose permissions are issued from a grant template at enrollment time (see NATS Security). When that template changes, an already-enrolled agent keeps running on its old grant until its next scheduled renewal — which can be hours away. Rotating credentials forces an agent to renew its JWT in place, immediately, so it picks up the current template without a re-enroll or any on-host surgery.

When and why to rotate

An agent renews its own JWT automatically — the renewal goroutine checks expiry every hour and renews when the remaining TTL drops below 50% (see NATS Security — JWT Renewal). On a successful renewal it forces a NATS reconnect so the live connection picks up the new credential immediately.

That auto-renewal is enough for normal operation, but it only happens on the agent's own ~50%-TTL schedule. When the grant template itself changes, an already-enrolled agent will not pick up the new permissions until that next scheduled renewal. Common triggers:

Change	New grant added	Symptom until rotated
Enabling pc kube on a cluster	kube.in.> subscribe	The kube relay subscription is denied; every kubectl call through the collector hangs.
Enabling ProxySSH on a host	ssh.in.> subscribe	The SSH relay subscription is denied; connections through the agent fail.

Newly enrolled agents always get the current template automatically. Rotation is for the agents that were already enrolled before the change.

Force the change now instead of waiting

You could wait for each agent to renew on its own ~50%-TTL cycle, but for a grant change you usually want the new permission live now — that is exactly what rotation does.

How to rotate

Rotating credentials requires the agents:write permission (the same operator-class permission used for fleet rollouts). It is tenant-scoped: you can only rotate agents for clients you have access to.

Fleet → Agents (UI)

The actions live on the Agents page (Assets → Agents, /agents):

• One agent — open the row's actions menu (⋯) and choose Rotate credentials. Confirm in the dialog.
• A whole fleet — tick the checkbox for each agent you want to rotate, then click Rotate N selected in the toolbar and confirm. This fires the rotation for every selected agent (bounded-concurrency) and reports a summary, e.g. Rotated 8 · 2 unreachable.

API

POST /api/v1/fleet/agents/{agentID}/rotate-credentials

A successful (online) rotation returns:

{ "status": "rotation_initiated" }

The endpoint is admin-gated by agents:write on the agent's owning client (checked in-handler, per the RBAC union-check rule).

What to expect

Rotation is asynchronous from the operator's point of view. The backend delivers a renew_credentials command to the agent over NATS request/reply; the agent then runs its normal renewal and forces a reconnect to apply the new JWT.

• Brief reconnect — the agent drops and re-establishes its NATS connection to pick up the new credential. This is the same reconnect a scheduled renewal performs; it is quick and self-healing.
• rotation_initiated — a 200 with {"status":"rotation_initiated"} means the command was delivered and acknowledged, not that the new JWT is already live. Confirm completion by checking the agent's updated last-seen / last renewal time (the jwt_issued_at on the agent record advances after a successful renewal) — it should refresh shortly after you rotate.
• For pc kube, confirm the collector logs kube access proxy enabled after it reconnects.

Offline and revoked agents

Situation	Response	What happens
Offline / unreachable agent	503 (agent_unreachable)	The command can't be delivered now. The agent will pick up the new template on its next scheduled ~50%-TTL renewal once it is back online — no further action needed. In a bulk rotation these are counted as unreachable, not failures.
Revoked / inactive agent	409 (agent_not_active)	Rotation is refused. A revoked agent's JWT is intentionally dead; re-enroll the agent instead of rotating.

Offline is not an error

A 503 for an offline agent is expected and safe — it simply means "deliver later." Only a rotation_failed (the agent responded but rejected the renewal) is a genuine failure to investigate.

Permissions

Permission	Grants
agents:read	View the Agents page.
agents:write	Rotate credentials (per-agent and bulk), and push/control fleet rollouts.

agents:write is held by the operator-class system roles — Super Admin, Admin, Engineer, and Team Lead. See Roles & Permissions for the full matrix.

See also

• NATS Security — JWT Renewal — how the per-agent JWT and its grants work.
• Updating Agents (Fleet Management) — the Agents page and version rollouts.
• How It Works — agent startup, command channel, and the renewal goroutine.