Cluster Agent

The cluster agent is a standalone binary that runs inside Kubernetes clusters, queries the K8s API, and publishes inventory data to Proxima Console via NATS.

Architecture

• Workload: Kubernetes StatefulSet in the proxima-system namespace — one replica by default, with optional leader-elected high availability
• Data source: K8s API via client-go (in-cluster service account)
• Transport: NATS (same as the node agent, using JetStream)
• Collection: Full sync every 5 minutes (configurable via PROXIMA_K8S_SYNC_INTERVAL)
• Enrollment: Self-enrolls with the backend via install token on first startup as agent_type=collector
• Identity: Each replica enrolls under its pod name (a stable StatefulSet identity). Enrollment binds a NATS identity once; the backend refuses to re-bind it (identity-takeover protection), so each pod keeps a PVC-backed state.json to preserve its identity across restarts — see Storage
• No host record: Collector agents do not create host records. Inventory is published directly to the Kubernetes worker without a corresponding hosts table entry.
• Heartbeat: Publishes periodic heartbeats for liveness detection (only the leader heartbeats)

Collected Resources

Resource	K8s API	Data
Cluster info	Discovery API	Version, platform, cloud provider
Nodes	CoreV1	Status, capacity, allocatable, addresses
Namespaces	CoreV1	Status, labels, resource counts
Workloads	AppsV1	Deployments, StatefulSets, DaemonSets
Pods	CoreV1	Phase, containers, restarts, resources, owner
Services	CoreV1	Type, ports, selector, cluster IP, load balancer
Ingresses	NetworkingV1	Rules, TLS, ingress class, load balancer
PVCs	CoreV1	Phase, storage class, capacity, access modes
Events	CoreV1	Type, reason, involved object, count, timestamps
Jobs/CronJobs	BatchV1	Status, completions, schedule, containers
HPAs	AutoscalingV2	Target, min/max/current replicas, metrics
Network Policies	NetworkingV1	Pod selector, ingress/egress rules, policy types
Resource Quotas	CoreV1	Hard limits, current usage per namespace
Endpoint Slices	DiscoveryV1	Addresses, ports, service association

Not collected: Secrets and ConfigMaps are deliberately excluded. The ClusterRole does not grant access to these resources.

Deployment

Recommended: Use the Helm Chart

The easiest way to deploy the cluster agent (and node agent) is via the Proxima Helm chart — it ships the collector as a StatefulSet with per-pod persistence and optional leader-elected HA. The manual kubectl apply steps below are a minimal single-replica, no-persistence quickstart for environments where Helm is not available.

Prerequisites

• Kubernetes cluster with API access
• Proxima Console backend running and accessible
• NATS server accessible from the cluster
• Install token from Proxima Console

1. Create namespace and RBAC

kubectl apply -f infra/k8s/cluster-agent/namespace.yaml
kubectl apply -f infra/k8s/cluster-agent/serviceaccount.yaml
kubectl apply -f infra/k8s/cluster-agent/clusterrole.yaml
kubectl apply -f infra/k8s/cluster-agent/clusterrolebinding.yaml

2. Create install token secret

kubectl -n proxima-system create secret generic proxima-install-token \
  --from-literal=token=YOUR_INSTALL_TOKEN

3. Configure and deploy

Edit infra/k8s/cluster-agent/deployment.yaml:

• Set PROXIMA_CLUSTER_NAME to your cluster's display name
• Set PROXIMA_BACKEND_URL to your Proxima Console backend URL
• Update the image tag if needed

kubectl apply -f infra/k8s/cluster-agent/deployment.yaml

4. Verify

kubectl -n proxima-system logs deployment/proxima-cluster-agent

You should see enrollment and collection logs. The cluster will appear in the Proxima Console API within one sync interval.

Configuration

Environment Variable	Default	Description
PROXIMA_CLUSTER_NAME	(required)	Cluster display name
PROXIMA_CLUSTER_SLUG	(derived from name)	URL-safe cluster identifier
PROXIMA_BACKEND_URL	(required)	Backend API for enrollment
PROXIMA_INSTALL_TOKEN	(required)	Enrollment token
PROXIMA_K8S_SYNC_INTERVAL	5m	Full sync interval
PROXIMA_K8S_POD_LIMIT	5000	Max pods per sync. When exceeded, non-Running pods are prioritized (sorted first), then by restart count descending.
PROXIMA_K8S_EXCLUDE_NAMESPACES	(empty)	Comma-separated list of namespaces to exclude from collection. Resources in excluded namespaces are not collected or stored.
PROXIMA_NATS_URL	(from enrollment)	Override NATS URL (takes priority over enrolled URL)
PROXIMA_NATS_CA_FILE	(empty)	Path to NATS TLS CA certificate
PROXIMA_AGENT_DATA_DIR	/var/lib/proxima-agent	Directory for enrollment state and CA certificate
PROXIMA_LOG_LEVEL	info	Log verbosity (debug, info, warn, error)
PROXIMA_METRICS_ADDR	:9090	Prometheus metrics HTTP address

Event limit

Events are capped at 1000 per sync (sorted by LastTimestamp descending, most recent kept). This limit is hardcoded and not configurable via environment variable.

High Availability

Run multiple replicas (clusterAgent.replicas > 1 in Helm) for HA. Replicas elect a single leader through a coordination.k8s.io Lease; only the leader collects and publishes inventory and heartbeats, while standbys stay enrolled and connected, ready to take over within the lease window (~15s). This prevents duplicate inventory while giving much faster failover than waiting for a single pod to reschedule.

Each replica is a StatefulSet pod with its own PVC and a stable per-pod identity (its pod name), so no ReadWriteOnce volume is shared and every replica keeps a durable identity across restarts. Standbys appear as separate agents in the Console; exactly one — the leader — reports at a time.

Leader election is automatic and needs no configuration beyond replicas. With replicas: 1 (the default) the single pod is always the leader, so behavior is unchanged. Local/dev runs without the Kubernetes downward API (no POD_NAME) skip leader election and run as the sole collector.

Security Hardening

The deployment manifest includes security best practices:

securityContext:
  runAsNonRoot: true
  runAsUser: 65534          # nobody
  fsGroup: 65534
  seccompProfile:
    type: RuntimeDefault    # meets the restricted Pod Security Standard
containers:
  - securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      capabilities:
        drop: [ALL]

RBAC ClusterRole

The collector uses a read-only ClusterRole with minimal permissions:

rules:
  - apiGroups: [""]
    resources: [nodes, namespaces, pods, services, endpoints,
                resourcequotas, events]
    verbs: [get, list, watch]
  - apiGroups: [apps]
    resources: [deployments, statefulsets, daemonsets, replicasets]
    verbs: [get, list, watch]
  - apiGroups: [batch]
    resources: [jobs, cronjobs]
    verbs: [get, list, watch]
  - apiGroups: [networking.k8s.io]
    resources: [ingresses, networkpolicies]
    verbs: [get, list, watch]
  - apiGroups: [""]
    resources: [persistentvolumeclaims]
    verbs: [get, list, watch]
  - apiGroups: [autoscaling]
    resources: [horizontalpodautoscalers]
    verbs: [get, list, watch]
  - apiGroups: [discovery.k8s.io]
    resources: [endpointslices]
    verbs: [get, list, watch]

No access to secrets, configmaps, or write operations.

When deployed via Helm, the collector also gets a small namespaced Role granting get/create/update on its own coordination.k8s.io Lease — used only for leader election. Nothing cluster-wide, nothing else.

Graceful Shutdown

The collector handles SIGINT and SIGTERM signals for clean shutdown:

1. The signal cancels the main context.
2. The metrics HTTP server is shut down with a 5-second timeout.
3. The scheduler waits for any in-progress sync to complete.
4. The NATS connection is closed.

NATS max_payload

The entire K8s inventory is published as a single NATS message. For large clusters (5000+ pods) the payload can exceed the NATS default max_payload of 1MB.

The Proxima NATS server is configured with max_payload: 8MB to fit large inventories (see infra/nats/). If a payload still exceeds the server limit, the agent rejects it up front with a clear ErrPayloadTooLarge error (logged and counted) rather than silently dropping it — lower PROXIMA_K8S_POD_LIMIT or raise the server's max_payload if you hit this.

Storage

The collector binds a NATS identity on its first enrollment, and the backend refuses to re-bind it (identity-takeover protection). A collector that loses its state.json therefore cannot re-enroll under the same name — it fails fast with 409 already enrolled. The collector needs durable state.

• Helm (recommended): the StatefulSet gives each replica its own PVC via a volumeClaimTemplate (persistence.enabled: true, the default). State survives restarts, reschedules, node drains, and upgrades. Requires a (default) StorageClass.
• Manual / emptyDir: state is lost on pod restart, so the pod then crash-loops on re-enrollment. Use emptyDir only for short-lived testing; for anything real, back /var/lib/proxima-agent with a PVC.

If you must reset a collector's identity (e.g. migrating off emptyDir), delete its agent record in Proxima Console first, then let the fresh pod enroll.

Monitoring

The collector exposes a Prometheus-compatible /metrics endpoint on port 9090:

• proxima_collector_syncs_total — total sync count
• proxima_collector_sync_errors_total — error count
• proxima_collector_last_sync_duration_seconds — last sync duration
• proxima_collector_resources_collected{resource="..."} — per-type resource counts

The collector exposes two probe endpoints: /healthz backs the Kubernetes liveness probe (process alive, never gated on enrollment or NATS), and /readyz backs the readiness probe — it reports ready only once the collector is enrolled and connected to NATS.