Node Agent on Kubernetes

The Proxima node agent can run as a Kubernetes DaemonSet to collect per-node kubelet metrics (CPU, memory, network, filesystem) and cAdvisor metrics (throttling, OOM, disk I/O).

Architecture

When deployed as a DaemonSet, the agent enrolls as agent_type=k8s-node-monitor and only collects Kubernetes-specific metrics:

• Kubelet /stats/summary — node, pod, and container CPU/memory/network/filesystem metrics
• Kubelet /metrics/cadvisor — CPU throttling, OOM events, and disk I/O metrics

It does not collect gopsutil system metrics, inventory, processes, file changes, or provide terminal/runbook access. Those are handled by the standalone host agent if installed.

Three-Agent Architecture

On a Kubernetes cluster, up to three agents work together:

Agent	Deployment	agent_type	Purpose
Node agent (DaemonSet)	Every node	k8s-node-monitor	Kubelet metrics only
Cluster agent (StatefulSet)	1+ replicas (leader-elected)	collector	K8s API inventory (what exists)
Host agent (systemd, optional)	Per node	host	Full host monitoring (inventory, metrics, terminal, runbooks)

Each agent enrolls independently with its own agent_id. Enrollment is idempotent by (environment_id, agent_type, name).

Dual-Agent Setup (DaemonSet + Host Agent)

The DaemonSet and host agent can run on the same node without conflict:

• No enrollment collision — different agent_type values create separate agent records
• No duplicate metrics — DaemonSet collects kubelet data only, host agent collects gopsutil data only
• Shared host record — the host agent owns the host record (agent_id set), DaemonSet metrics resolve by hostname
• Independent stale detection — each agent has its own liveness tracking

Setup	Kubelet Metrics	System Metrics	Inventory	Terminal	Runbooks
DaemonSet only	Yes	No	Minimal (hostname)	No	No
Host agent only	No	Yes	Full	Yes	Yes
Both	Yes (DaemonSet)	Yes (host agent)	Full (host agent)	Yes (host agent)	Yes (host agent)

Data directory

The DaemonSet uses /var/lib/proxima-agent (hostPath). If you install a host agent alongside it, give it a different data directory so it does not load the DaemonSet's state.json and adopt its k8s-node-monitor identity. Pass --data-dir /var/lib/proxima-agent-host to the installer (or set PROXIMA_AGENT_DATA_DIR directly):

curl -sL https://install.prxm.uz/agent.sh | bash -s -- \
  --backend-url https://api-console.prxm.uz \
  --install-token <token> \
  --data-dir /var/lib/proxima-agent-host

Collected Metrics

From /stats/summary (every 30s)

Metric	Level	Type	Description
node_cpu_usage_ratio	Node	gauge	CPU utilization (cores)
node_memory_used_bytes	Node	gauge	Memory usage
node_memory_available_bytes	Node	gauge	Memory available
node_memory_rss_bytes	Node	gauge	Resident set size
node_fs_used_bytes	Node	gauge	Root filesystem used
node_fs_available_bytes	Node	gauge	Root filesystem available
node_network_receive_bytes_total	Node	counter	Network received
node_network_transmit_bytes_total	Node	counter	Network transmitted
pod_cpu_usage_ratio	Pod	gauge	Pod CPU utilization
pod_memory_used_bytes	Pod	gauge	Pod memory usage
pod_memory_rss_bytes	Pod	gauge	Pod RSS
pod_memory_working_set_bytes	Pod	gauge	Pod working set
pod_network_receive_bytes_total	Pod	counter	Pod network received
pod_network_transmit_bytes_total	Pod	counter	Pod network transmitted
container_cpu_usage_ratio	Container	gauge	Container CPU utilization
container_memory_used_bytes	Container	gauge	Container memory usage
container_memory_working_set_bytes	Container	gauge	Container working set
container_memory_rss_bytes	Container	gauge	Container RSS
container_rootfs_used_bytes	Container	gauge	Container rootfs usage
container_logs_used_bytes	Container	gauge	Container log disk usage

From /metrics/cadvisor (every 30s)

Metric	Level	Type	Description
container_cpu_throttled_ratio	Container	gauge	CPU throttle ratio (0-1)
container_oom_events_total	Container	counter	OOM kill count
container_fs_reads_bytes_total	Container	counter	Disk read bytes (delta)
container_fs_writes_bytes_total	Container	counter	Disk write bytes (delta)

Log Enrichment

Logs tailed from /var/log/pods/ are automatically enriched with:

• k8s_namespace — pod namespace
• k8s_pod — pod name
• k8s_container — container name
• k8s_pod_uid — pod UID

Queryable in VictoriaLogs: k8s_namespace:app AND k8s_pod:nginx*

Deployment

Recommended: Use the Helm Chart

The easiest way to deploy both the node agent and cluster agent is via the Proxima Helm chart. The manual steps below are for environments where Helm is not available.

Prerequisites

• Kubernetes cluster with kubelet API access
• Proxima Console backend running
• NATS server accessible from cluster
• Install token from Proxima Console
• The cluster agent should also be deployed for inventory data

1. Create RBAC

kubectl apply -f infra/k8s/node-agent/serviceaccount.yaml
kubectl apply -f infra/k8s/node-agent/clusterrole.yaml
kubectl apply -f infra/k8s/node-agent/clusterrolebinding.yaml

2. Create install token secret (if not already exists)

kubectl -n proxima-system create secret generic proxima-install-token \
  --from-literal=token=YOUR_INSTALL_TOKEN

3. Deploy DaemonSet

Edit infra/k8s/node-agent/daemonset.yaml:

• Set PROXIMA_BACKEND_URL to your backend URL
• Adjust resource limits if needed

The DaemonSet requires the following security context to support both kubelet metric collection and interactive terminal access via nsenter:

spec:
  hostPID: true          # required for nsenter -t 1 (enter host PID namespace)
  containers:
    - name: proxima-node-agent
      securityContext:
        allowPrivilegeEscalation: true   # required for nsenter to enter namespaces
        readOnlyRootFilesystem: true     # pod filesystem stays read-only
        capabilities:
          add:
            - SYS_PTRACE   # allows ptrace of PID 1 to enter namespaces
            - SYS_ADMIN    # allows mount namespace operations

Why each field is needed:

Field	Why
hostPID: true	Shares the host PID namespace so nsenter -t 1 can target the host's init process
SYS_PTRACE	Required by nsenter to attach to PID 1 and enter its namespaces
SYS_ADMIN	Required to enter the host mount namespace (nsenter --mount)
allowPrivilegeEscalation: true	nsenter raises its effective capabilities when entering namespaces; this must not be blocked
readOnlyRootFilesystem: true	The agent pod's own filesystem remains read-only — writes go to the host via nsenter
No privileged: true	Privileged mode is not required and is a security audit red flag — targeted capabilities are sufficient

kubectl apply -f infra/k8s/node-agent/daemonset.yaml

4. Verify

# Check pods are running on all nodes
kubectl -n proxima-system get pods -l app.kubernetes.io/name=proxima-node-agent -o wide

# Check logs for kubelet metrics collection
kubectl -n proxima-system logs daemonset/proxima-node-agent | grep "kubelet stats published"

Configuration

Environment Variable	Default	Description
PROXIMA_AGENT_TYPE	host	Agent type for enrollment. Set to k8s-node-monitor for DaemonSet agents (Helm chart sets this automatically).
PROXIMA_K8S_NODE_METRICS	false	Enable kubelet/cAdvisor metrics
PROXIMA_K8S_METRICS_INTERVAL	30s	Kubelet scrape interval
PROXIMA_K8S_NODE_NAME	(downward API)	Fallback node name for enrollment. The agent prefers /host/etc/hostname (FQDN) when PROXIMA_HOST_ROOT is set.
PROXIMA_INSTALL_TOKEN	(required)	Enrollment token
PROXIMA_BACKEND_URL	(required)	Backend API URL
PROXIMA_LOG_LEVEL	info	Log verbosity

When PROXIMA_AGENT_TYPE=k8s-node-monitor, the agent only runs kubelet/cAdvisor collection and heartbeats. Inventory, system metrics, processes, file watch, terminal, and runbooks are disabled.

Terminal Access

DaemonSet agents support interactive terminal access to the underlying Kubernetes node via nsenter. When you open a terminal session to a K8s node in the Console UI, the agent runs nsenter -t 1 --pid --mount --uts --ipc --net -- su - <login> to enter the host's full namespace set and drop into a shell on the actual node — not inside the agent pod.

The security context described in the Deploy DaemonSet section above is required for this to work. Without hostPID: true and the SYS_PTRACE/SYS_ADMIN capabilities, nsenter will fail with a permission error.

See Terminal Security — K8s Node Terminal Security for the full security model.

Terminal Access on Minimal Nodes

Minimal K8s distributions (RKE2/SLE Micro, k3s, Talos, Flatcar) don't include user management binaries (useradd, groupadd, passwd). The DaemonSet agent automatically detects this and disables host user creation — terminal sessions are limited to existing users (typically root).

Dual-Agent Setup (Full Features)

For full terminal features (user creation, sudoers, custom shell) on K8s nodes, install the Proxima agent directly on the host alongside the DaemonSet:

Component	Purpose
DaemonSet agent	Metrics, inventory, K8s node monitoring
Host agent	Terminal access, user creation, runbooks

Install the host agent on each node using the standard install script:

curl -sL https://install.prxm.uz/agent.sh | bash -s -- \
  --backend-url https://api-console.prxm.uz \
  --install-token <token>

Both agents enroll independently and coexist without conflict.

Integration-to-Asset Bridge

When the node agent runs with integration collectors enabled (PostgreSQL, Redis, Nginx, Docker), the agent heartbeat includes collector statuses. The backend heartbeat worker automatically creates CMDB assets for each detected integration. For example, a K8s node running PostgreSQL will produce both a host asset (from inventory) and a database asset (from the heartbeat integration bridge).

This works identically whether the agent runs as a DaemonSet on K8s or as a standalone service on a bare-metal host. See Assets & CMDB for details on the asset types and metadata created.