Node Agent on Kubernetes

The Proxima node agent can run as a Kubernetes DaemonSet to collect per-node kubelet metrics (CPU, memory, network, filesystem) and cAdvisor metrics (throttling, OOM, disk I/O).

Architecture

When deployed as a DaemonSet, the agent enrolls as agent_type=k8s-node-monitor and only collects Kubernetes-specific metrics:

• Kubelet /stats/summary — node, pod, and container CPU/memory/network/filesystem metrics
• Kubelet /metrics/cadvisor — CPU throttling, OOM events, and disk I/O metrics

It does not collect gopsutil system metrics, inventory, processes, file changes, or provide terminal/runbook access. Those are handled by the standalone host agent if installed.

Three-Agent Architecture

On a Kubernetes cluster, up to three agents work together:

Agent	Deployment	agent_type	Purpose
Node agent (DaemonSet)	Every node	k8s-node-monitor	Kubelet metrics only
Cluster agent (StatefulSet)	1+ replicas (leader-elected)	collector	K8s API inventory (what exists)
Host agent (systemd, optional)	Per node	host	Full host monitoring (inventory, metrics, terminal, runbooks)

Each agent enrolls independently with its own agent_id. Enrollment is idempotent by (environment_id, agent_type, name).

Dual-Agent Setup (DaemonSet + Host Agent)

The DaemonSet and host agent can run on the same node, provided each has its own data directory (see the box below — this is the one thing you must configure):

• Separate agent records — the two enroll independently and hold different agent_type values. This depends on the separate data directories, not on agent_type itself: an agent that finds an existing state.json adopts that identity and never enrolls, whatever type it is configured as.
• No duplicate metrics — DaemonSet collects kubelet data only, host agent collects gopsutil data only
• Shared host record — the host agent owns the host record (agent_id set), DaemonSet metrics resolve by hostname
• Independent stale detection — each agent has its own liveness tracking

Setup	Kubelet Metrics	System Metrics	Inventory	Terminal	Runbooks
DaemonSet only	Yes	No	Minimal (hostname)	No	No
Host agent only	No	Yes	Full	Yes	Yes
Both	Yes (DaemonSet)	Yes (host agent)	Full (host agent)	Yes (host agent)	Yes (host agent)

Data directory — whichever agent arrives second must be moved

Both default to /var/lib/proxima-agent, and the DaemonSet mounts it from the host. Whichever agent starts second finds the other's state.json and adopts its identity — same agent_id, same private nkey seed — because an agent only enrolls when no valid state file is present. The result is two processes reporting as one agent: versions that flap on every console refresh, and a single self_update delivered to both, where the pod fails to swap a binary on its read-only image layer and the rollout records that failure against a healthy host.

Since v0.7.2 the agent refuses to start on a mismatched directory rather than sharing an identity, so this shows up as a startup error instead of silent corruption. It never deletes the state file it found — that file belongs to the other agent.

If the DaemonSet is already there, give the host agent its own directory:

curl -sL https://install.prxm.uz/agent.sh | bash -s -- \
  --backend-url https://api-console.prxm.uz \
  --install-token <token> \
  --data-dir /var/lib/proxima-agent-host

If the host agent is already there — the common case, since it owns the default path — move the DaemonSet's host path instead:

helm upgrade --install proxima-agent proxima/proxima-agent \
  --namespace proxima-system \
  --set nodeAgent.hostDataDir=/var/lib/proxima-node-agent \
  ...

hostDataDir changes only the host side of the mount; the in-container path stays /var/lib/proxima-agent.

Changing it on a live cluster only works where the node's k8s-node-monitor name is still free. On a node running just the DaemonSet, that name is already taken by the record the pod is abandoning. Enrollment now re-binds the pod onto that record by itself, provided it has been quiet for five minutes and the pod can still present a signal it carries — which a host-root mount gives it, since the host's /etc/machine-id and DMI uuid do not move with the data directory. Where that does not hold the old 409 loop still applies (the agent retries rather than exiting, so the pod stays Ready while reporting nothing), and the record's name is freed with POST /api/v1/fleet/agents/{agentID}/release-identity. See Helm chart.

Collected Metrics

Every series below is collected once per metrics interval (PROXIMA_K8S_METRICS_INTERVAL). The Helm chart sets it to 60s (nodeAgent.metricsInterval); without the chart the agent's default is 30s.

When the node agent's host is linked to a Kubernetes node (Node-Host Linkage), the backend adds a cluster_id label to every metric it sends. That label is how a cluster's Overview charts find the node's metrics — see Kubernetes Clusters: Overview & List.

From /stats/summary (every metrics interval)

Metric	Level	Type	Description
node_cpu_usage_ratio	Node	gauge	CPU utilization, 0–1 of the node's logical CPUs
node_memory_used_bytes	Node	gauge	Memory usage
node_memory_available_bytes	Node	gauge	Memory available
node_memory_rss_bytes	Node	gauge	Resident set size
node_fs_used_bytes	Node	gauge	Root filesystem used
node_fs_available_bytes	Node	gauge	Root filesystem available
node_network_receive_bytes_total	Node	counter	Network received
node_network_transmit_bytes_total	Node	counter	Network transmitted
pod_cpu_usage_ratio	Pod	gauge	Pod CPU usage in cores (not a 0–1 ratio despite the name; 2.5 = two and a half cores)
pod_memory_used_bytes	Pod	gauge	Pod memory usage
pod_memory_rss_bytes	Pod	gauge	Pod RSS
pod_memory_working_set_bytes	Pod	gauge	Pod working set
pod_network_receive_bytes_total	Pod	counter	Pod network received
pod_network_transmit_bytes_total	Pod	counter	Pod network transmitted
container_cpu_usage_ratio	Container	gauge	Container CPU usage in cores (not a 0–1 ratio despite the name)
container_memory_used_bytes	Container	gauge	Container memory usage
container_memory_working_set_bytes	Container	gauge	Container working set
container_memory_rss_bytes	Container	gauge	Container RSS
container_rootfs_used_bytes	Container	gauge	Container rootfs usage
container_logs_used_bytes	Container	gauge	Container log disk usage

From /metrics/cadvisor (every metrics interval)

Metric	Level	Type	Description
container_cpu_throttled_ratio	Container	gauge	CPU throttle ratio (0-1)
container_oom_events_total	Container	counter	OOM kill count
container_fs_reads_bytes_total	Container	counter	Disk read bytes (delta)
container_fs_writes_bytes_total	Container	counter	Disk write bytes (delta)

From /pods (every metrics interval)

Metric	Level	Type	Description
kubelet_pods	Node	gauge	Pods on the node by phase (Running, Pending, Succeeded, Failed, Unknown); every phase is reported each tick, zeros included
container_restarts_total	Container	counter	The container's restart count. It resets to 0 when a pod is recreated. Count restarts with a per-step difference, not increase(): clamp_min(last_over_time(container_restarts_total[W]) - last_over_time(container_restarts_total[W] offset <step>), 0) with W = max(step, 5m), which is what the cluster restarts query does. increase() counts part of a new series' first sample, so an agent upgrade or a new pod would show as a restart spike. Init containers are included and carry init="true"
container_waiting	Container	gauge	1 while a container is Waiting, with the reason label (for example CrashLoopBackOff); a pod stuck in Init:CrashLoopBackOff reports its init container with init="true"

Workload labels

Pod and container metrics also carry workload_kind and workload — the Deployment, Argo Rollout, StatefulSet, DaemonSet, Job, ReplicaSet, static pod or bare pod each pod belongs to — so charts can group by workload. The node agent reads its own kubelet's /pods once per tick to resolve them; this uses the nodes/proxy permission the node agent's ClusterRole already grants.

A ReplicaSet owner becomes a Deployment when the pod's pod-template-hash label is the ReplicaSet name's -<hash> suffix, or an Argo Rollout (workload_kind="Rollout") when the pod's rollouts-pod-template-hash label is. Argo Rollouts do not set pod-template-hash, so node agents built before Rollout support label a Rollout's pods workload_kind="ReplicaSet", workload="<rollout>-<hash>", and the Rollout's workload page shows no per-pod charts until the node agent is upgraded. The node agent needs no new permission for this.

A pod's workload never changes, so once the agent has seen a pod it keeps labelling that pod's metrics even if /pods later becomes unavailable — the labels do not come and go, which would otherwise split a pod's chart into two series. A pod the agent has never seen (for example one created during a /pods outage) is sent without workload labels.

If the kubelet refuses /pods (for example a ClusterRole without nodes/proxy), the agent logs a warning when the outage starts (then at most one every 10 minutes while it lasts), and an info line when it recovers. A slow or hanging /pods delays a collection tick by at most 2 seconds. Every other metric keeps flowing; the three /pods series above stop after about two collection intervals and resume on recovery. Node agents older than v0.8.0 never send workload labels or these series: a cluster's Overview then shows "Workload breakdown and restart history need a newer node agent (v0.8.0 or later)" instead of an empty restarts chart, and in a mixed fleet its restart total covers only the upgraded nodes.

One-time series change on upgrade

Adding the workload labels changes the identity of every pod and container series once, when the node agent is upgraded. On a long-lived pod's chart you will see one series end and a new one begin at the upgrade time. This happens once per series and does not repeat.

Log Enrichment

Logs tailed from /var/log/pods/ are automatically enriched with:

• k8s_namespace — pod namespace
• k8s_pod — pod name
• k8s_container — container name
• k8s_pod_uid — pod UID

Queryable in VictoriaLogs: k8s_namespace:app AND k8s_pod:nginx*

Deployment

Recommended: Use the Helm Chart

The easiest way to deploy both the node agent and cluster agent is via the Proxima Helm chart. The manual steps below are for environments where Helm is not available.

Prerequisites

• Kubernetes cluster with kubelet API access
• Proxima Console backend running
• NATS server accessible from cluster
• Install token from Proxima Console
• The cluster agent should also be deployed for inventory data

1. Create RBAC

kubectl apply -f infra/k8s/node-agent/serviceaccount.yaml
kubectl apply -f infra/k8s/node-agent/clusterrole.yaml
kubectl apply -f infra/k8s/node-agent/clusterrolebinding.yaml

2. Create install token secret (if not already exists)

kubectl -n proxima-system create secret generic proxima-install-token \
  --from-literal=token=YOUR_INSTALL_TOKEN

3. Deploy DaemonSet

Edit infra/k8s/node-agent/daemonset.yaml:

• Set PROXIMA_BACKEND_URL to your backend URL
• Adjust resource limits if needed

The DaemonSet ships with the following security context:

spec:
  hostPID: true
  containers:
    - name: proxima-node-agent
      securityContext:
        runAsUser: 0
        allowPrivilegeEscalation: true
        readOnlyRootFilesystem: true
        capabilities:
          add:
            - SYS_PTRACE
            - SYS_ADMIN

Field	Why
runAsUser: 0	Reading /proc and /sys for node metrics, and writing the agent's state to its hostPath volume
readOnlyRootFilesystem: true	The agent pod's own filesystem stays read-only
No privileged: true	Privileged mode is not required and is a security audit red flag

hostPID, SYS_PTRACE, SYS_ADMIN and allowPrivilegeEscalation are no longer exercised

These four were added for the nsenter-based node terminal, and the agent no longer implements it — runbook_terminal.go skips terminal setup entirely when it detects DaemonSet mode, and NsenterCommander has no caller left. The chart still requests them (charts/proxima-agent/values.yaml), and the comments there still cite nsenter as the reason.

Metrics collection needs runAsUser: 0 and the hostPath mounts; it does not need the host PID namespace or either capability. If you are hardening a cluster, these are removable — test in a non-production cluster first, since kubelet access patterns vary by distribution.

kubectl apply -f infra/k8s/node-agent/daemonset.yaml

4. Verify

# Check pods are running on all nodes
kubectl -n proxima-system get pods -l app.kubernetes.io/name=proxima-node-agent -o wide

# Check logs for kubelet metrics collection
kubectl -n proxima-system logs daemonset/proxima-node-agent | grep "kubelet stats published"

Configuration

Environment Variable	Default	Description
PROXIMA_AGENT_TYPE	host	Agent type for enrollment. Set to k8s-node-monitor for DaemonSet agents (Helm chart sets this automatically).
PROXIMA_K8S_NODE_METRICS	false	Enable kubelet/cAdvisor metrics
PROXIMA_K8S_METRICS_INTERVAL	30s (Helm chart: 60s)	Kubelet scrape interval
PROXIMA_K8S_NODE_NAME	(downward API)	Fallback node name for enrollment. The agent prefers /host/etc/hostname (FQDN) when PROXIMA_HOST_ROOT is set.
PROXIMA_INSTALL_TOKEN	(required)	Enrollment token
PROXIMA_BACKEND_URL	(required)	Backend API URL
PROXIMA_LOG_LEVEL	info	Log verbosity

When PROXIMA_AGENT_TYPE=k8s-node-monitor, the agent only runs kubelet/cAdvisor collection and heartbeats. Inventory, system metrics, processes, file watch, terminal, and runbooks are disabled.

Terminal Access

The DaemonSet agent does not provide terminal access

An nsenter-based node terminal was built and then withdrawn — a PTY over nsenter proved unreliable on the minimal node operating systems these agents run on. The agent now detects DaemonSet mode and skips terminal setup entirely, so no terminal handler is registered and a terminal session to a node served only by the DaemonSet cannot be opened. The agent logs this at startup:

k8s DaemonSet mode — terminal access disabled (install host agent for terminal)

To get terminal access on a Kubernetes node, run the standalone host agent on the node alongside the DaemonSet.

Dual-Agent Setup

Component	Purpose
DaemonSet agent	Kubelet/cAdvisor metrics, K8s node monitoring
Host agent	Terminal access, inventory, system metrics, user creation, runbooks

Keep the DaemonSet on PROXIMA_AGENT_TYPE=k8s-node-monitor, and give the two agents separate data directories — the agent type alone does not keep them apart, because an agent that finds an existing state.json adopts that identity instead of enrolling. See Data directory above. Note that minimal distributions (RKE2/SLE Micro, k3s, Talos, Flatcar) ship without useradd/groupadd/passwd, so the host agent detects their absence and disables host user creation — sessions are limited to existing users, typically root.

Install the host agent on each node using the standard install script:

curl -sL https://install.prxm.uz/agent.sh | bash -s -- \
  --backend-url https://api-console.prxm.uz \
  --install-token <token>

Both agents enroll independently and coexist without conflict.

Integration-to-Asset Bridge

When the node agent runs with integration collectors enabled (PostgreSQL, Redis, Nginx, Docker), the agent heartbeat includes collector statuses. The backend heartbeat worker automatically creates CMDB assets for each detected integration. For example, a K8s node running PostgreSQL will produce both a host asset (from inventory) and a database asset (from the heartbeat integration bridge).

This works identically whether the agent runs as a DaemonSet on K8s or as a standalone service on a bare-metal host. See Assets & CMDB for details on the asset types and metadata created.