Kubernetes Inventory

The Kubernetes inventory system collects and stores cluster infrastructure data — clusters, nodes, namespaces, workloads, pods, services, ingresses, PVCs, events, jobs, HPAs, network policies, resource quotas, and endpoint slices — and exposes it through REST API endpoints with RBAC client scoping. Clusters are also registered as CMDB assets, enabling unified inventory views alongside hosts.

Phase

CMDB-2a is complete. The system collects 13 Kubernetes resource types via the cluster agent, stores them in PostgreSQL, and exposes read-only API endpoints. The cluster agent binary runs as a single-replica Deployment inside each K8s cluster.

Overview

Kubernetes inventory data flows from the cluster agent (deployed inside each K8s cluster) through NATS into the backend, where the KubernetesWorker processes it into PostgreSQL tables. Each cluster also gets an entry in the assets table for unified CMDB views.

Collected Resource Types

The cluster agent gathers 13 resource types from the K8s API:

#	Resource	K8s API Group	Table	Description
1	Nodes	CoreV1	k8s_nodes	Status, capacity, allocatable, usage, addresses
2	Namespaces	CoreV1	k8s_namespaces	Status, labels, workload/pod/service counts
3	Workloads	AppsV1	k8s_workloads	Deployments, StatefulSets, DaemonSets
4	Pods	CoreV1	k8s_pods	Phase, containers, restarts, resources, owner
5	Services	CoreV1	k8s_services	Type, ports, selector, cluster IP
6	Ingresses	NetworkingV1	k8s_ingresses	Rules, TLS, ingress class, load balancer
7	PVCs	CoreV1	k8s_pvcs	Phase, storage class, capacity, access modes
8	Events	CoreV1	k8s_events	Type, reason, involved object, count, timestamps
9	Jobs/CronJobs	BatchV1	k8s_jobs	Status, completions, schedule, containers
10	HPAs	AutoscalingV2	k8s_hpas	Target, min/max/current replicas, metrics
11	Network Policies	NetworkingV1	k8s_network_policies	Pod selector, ingress/egress rules, policy types
12	Resource Quotas	CoreV1	k8s_resource_quotas	Hard limits, current usage per namespace
13	Endpoint Slices	DiscoveryV1	k8s_endpoint_slices	Addresses, ports, service association

Data Flow

NATS Message Path

Kubernetes inventory messages use the standard telemetry subject pattern with kubernetes as the message type:

proxima.{client_slug}.{env_slug}.{host_id}.kubernetes

The message is routed through the EVENTS JetStream stream to the EventsWorker, which dispatches kubernetes message types to the KubernetesWorker.

Upsert-Then-Prune Sync Pattern

The KubernetesWorker uses an upsert-then-prune pattern rather than delete-reinsert. This is important for data consistency and referential integrity:

1. Upsert phase — Each resource is inserted or updated using ON CONFLICT ... DO UPDATE. The worker tracks all seen entity IDs during the sync.
2. Prune phase — After all upserts, the worker deletes any rows for the cluster that were not seen in this sync cycle. This removes resources that no longer exist in the cluster.

The prune follows FK dependency order (children before parents):

k8s_endpoint_slices → k8s_resource_quotas → k8s_network_policies →
k8s_hpas → k8s_jobs → k8s_events → k8s_pvcs → k8s_ingresses →
k8s_services → k8s_pods → k8s_workloads → k8s_namespaces → k8s_nodes

All upserts and prunes run inside a single database transaction with a 60-second timeout.

Processing Steps

1. Asset upsert — Creates or updates an assets record with asset_type = 'cluster' and maps the cluster status to an asset status.
2. Cluster upsert — Inserts or updates the clusters record with identity, K8s metadata, cloud context, and status fields.
3. Node sync — Upserts all nodes for the cluster. For each node, attempts to link it to an existing host via hostname or IP address matching.
4. Namespace sync — Upserts all namespaces with workload, pod, and service counts.
5. Workload sync — Upserts all workloads (Deployments, StatefulSets, DaemonSets) with replica status and Helm metadata.
6. Pod sync — Upserts all pods with phase, container specs, resources, restarts, and owner references.
7. Service sync — Upserts all services with type, ports, selectors, and IPs.
8. Ingress sync — Upserts all ingresses with rules, TLS, and class metadata.
9. PVC sync — Upserts all PVCs with phase, storage class, and capacity.
10. Event sync — Upserts all events with type, reason, and involved object.
11. Job sync — Upserts all jobs and cron jobs with schedule, completions, and status.
12. HPA sync — Upserts all HPAs with target reference, scaling bounds, and current metrics.
13. Network policy sync — Upserts all network policies with selectors and rules.
14. Resource quota sync — Upserts all quotas with hard limits and current usage.
15. Endpoint slice sync — Upserts all endpoint slices with addresses and ports.
16. Prune — Removes entities not seen in this sync cycle.
17. Count update — Updates denormalized counts on the cluster record.

Collection Limits and Filtering

Namespace exclusion: The collector skips namespaces listed in PROXIMA_K8S_EXCLUDE_NAMESPACES (comma-separated). Excluded namespaces and their resources are not collected or stored.

Pod limit: The collector enforces a configurable pod limit (PROXIMA_K8S_POD_LIMIT, default: 5000). When the limit is exceeded, pods are sorted with non-Running pods first (then by restart count descending), and the list is truncated. This ensures problematic pods are always included.

Event limit: Events are capped at 1000 per sync (hardcoded maxEvents). Events are sorted by LastTimestamp descending, keeping only the most recent.

Annotation stripping: The annotation kubectl.kubernetes.io/last-applied-configuration is stripped from all resources before transmission. This annotation often contains the full resource manifest and can be very large. The stripping is applied to workloads, services, ingresses, PVCs, jobs, and network policies.

Security

The cluster agent follows a strict security posture:

• No secrets or configmaps collected — These resource types are not in the ClusterRole permissions.
• Read-only RBAC — The ClusterRole only grants get, list, watch verbs.
• Annotation stripping — kubectl.kubernetes.io/last-applied-configuration is removed to avoid leaking full resource manifests.

Table Schemas

clusters

The primary cluster identity table. Linked to assets via asset_id.

Column	Type	Description
id	UUID	Primary key
asset_id	UUID	FK to assets.id (ON DELETE CASCADE)
client_id	UUID	FK to clients.id
environment_id	UUID	FK to environments.id
name	TEXT	Cluster name (e.g., k8s-prod-01)
display_name	TEXT	Human-readable display name
slug	TEXT	URL-safe identifier (unique per environment)
version	TEXT	Kubernetes version (e.g., v1.29.3)
platform	TEXT	Distribution platform (rke2, eks, gke, aks, k3s, kubeadm)
distribution	TEXT	Distribution detail
api_server_url	TEXT	API server endpoint
cluster_cidr	TEXT	Pod CIDR range
service_cidr	TEXT	Service CIDR range
dns_domain	TEXT	Cluster DNS domain (default: cluster.local)
cloud_provider	TEXT	Cloud provider (aws, gcp, azure, hetzner)
cloud_region	TEXT	Cloud region
cloud_account_id	TEXT	Cloud account identifier
status	TEXT	Cluster health status (healthy, degraded, unreachable, unknown)
agent_status	TEXT	Agent connection status (connected, disconnected)
agent_version	TEXT	Reporting agent version
node_count	INTEGER	Denormalized node count
namespace_count	INTEGER	Denormalized namespace count
workload_count	INTEGER	Denormalized workload count
last_seen_at	TIMESTAMPTZ	Last inventory message timestamp
created_at	TIMESTAMPTZ	Record creation time
updated_at	TIMESTAMPTZ	Last modification time

Indexes: client_id, environment_id, asset_id, status, agent_status, last_seen_at, unique on (environment_id, slug).

k8s_nodes

Kubernetes nodes with capacity, allocatable resources, and usage data.

Column	Type	Description
id	UUID	Primary key
cluster_id	UUID	FK to clusters.id (ON DELETE CASCADE)
host_id	UUID	FK to hosts.id (nullable, set by host linkage)
name	TEXT	Node name
uid	TEXT	Kubernetes UID
roles	TEXT[]	Node roles (e.g., control-plane, worker)
labels	JSONB	Kubernetes labels
taints	JSONB	Node taints
status	TEXT	Node condition status (Ready, NotReady, Unknown)
conditions	JSONB	Full conditions array
unschedulable	BOOLEAN	Whether scheduling is disabled
kubelet_version	TEXT	Kubelet version
container_runtime	TEXT	Container runtime (e.g., containerd://1.7.2)
os_image	TEXT	OS image (e.g., Ubuntu 22.04.3 LTS)
kernel_version	TEXT	Kernel version
capacity_cpu_millicores	INTEGER	Total CPU capacity
capacity_memory_bytes	BIGINT	Total memory capacity
capacity_pods	INTEGER	Max pod count
allocatable_cpu_millicores	INTEGER	Allocatable CPU
allocatable_memory_bytes	BIGINT	Allocatable memory
allocatable_pods	INTEGER	Allocatable pod count
usage_cpu_millicores	INTEGER	Current CPU usage
usage_memory_bytes	BIGINT	Current memory usage
pod_count	INTEGER	Current pod count
provider_id	TEXT	Cloud provider instance ID
instance_type	TEXT	Cloud instance type
zone	TEXT	Availability zone

Indexes: cluster_id, host_id, status, unique on (cluster_id, name) and (cluster_id, uid).

k8s_namespaces

Kubernetes namespaces with resource counts.

Column	Type	Description
id	UUID	Primary key
cluster_id	UUID	FK to clusters.id (ON DELETE CASCADE)
name	TEXT	Namespace name
uid	TEXT	Kubernetes UID
status	TEXT	Namespace phase (Active, Terminating)
labels	JSONB	Kubernetes labels
workload_count	INTEGER	Workloads in this namespace
pod_count	INTEGER	Pods in this namespace
service_count	INTEGER	Services in this namespace

Indexes: cluster_id, GIN on labels, unique on (cluster_id, name).

k8s_workloads

Kubernetes workload resources (Deployments, StatefulSets, DaemonSets, Jobs, CronJobs).

Column	Type	Description
id	UUID	Primary key
cluster_id	UUID	FK to clusters.id (ON DELETE CASCADE)
namespace_id	UUID	FK to k8s_namespaces.id (ON DELETE CASCADE)
kind	TEXT	Resource kind (Deployment, StatefulSet, etc.)
name	TEXT	Workload name
uid	TEXT	Kubernetes UID
namespace	TEXT	Namespace name (denormalized)
replicas_desired	INTEGER	Desired replica count
replicas_ready	INTEGER	Ready replica count
replicas_available	INTEGER	Available replica count
conditions	JSONB	Workload conditions
strategy	TEXT	Deployment strategy
service_account	TEXT	Service account name
containers	JSONB	Container specs (image, resources, ports)
labels	JSONB	Kubernetes labels
annotations	JSONB	Kubernetes annotations
helm_release	TEXT	Helm release name
helm_chart	TEXT	Helm chart name
helm_chart_version	TEXT	Helm chart version

Indexes: cluster_id, namespace_id, kind, GIN on labels, conditional index on helm_release (non-null only), unique on (cluster_id, kind, namespace, name).

Table Relationships

API Endpoints

All endpoints require authentication and the assets:read permission. Responses are scoped to the caller's allowed clients.

List Clusters

GET /api/v1/clusters

Query parameters:

Parameter	Type	Description
client_id	UUID	Filter by client
environment_id	UUID	Filter by environment
status	string	Filter by cluster status
search	string	Name search (case-insensitive substring)
page	int	Page number (default: 1)
per_page	int	Items per page (default: 50)

Response:

{
  "data": [
    {
      "id": "a1b2c3d4-...",
      "asset_id": "e5f6g7h8-...",
      "client_id": "c1d2e3f4-...",
      "environment_id": "d4e5f6a7-...",
      "name": "k8s-prod-01",
      "display_name": "Production Cluster",
      "slug": "k8s-prod-01",
      "version": "v1.29.3",
      "platform": "rke2",
      "status": "healthy",
      "node_count": 5,
      "namespace_count": 12,
      "workload_count": 47,
      "last_seen_at": "2026-04-05T12:00:00Z"
    }
  ],
  "meta": {
    "total": 3,
    "page": 1,
    "per_page": 50
  }
}

Get Cluster

GET /api/v1/clusters/:clusterID

Returns full cluster details including all metadata, cloud context, and counts.

List Cluster Nodes

GET /api/v1/clusters/:clusterID/nodes

Returns all nodes in the cluster with capacity, allocatable resources, usage, and host linkage.

List Cluster Namespaces

GET /api/v1/clusters/:clusterID/namespaces

Returns all namespaces with resource counts.

List Cluster Workloads

GET /api/v1/clusters/:clusterID/workloads

Query parameters:

Parameter	Type	Description
namespace	string	Filter by namespace name
kind	string	Filter by resource kind
page	int	Page number (default: 1)
per_page	int	Items per page (default: 50)

List Cluster Pods

GET /api/v1/clusters/:clusterID/pods

Query parameters:

Parameter	Type	Description
namespace	string	Filter by namespace name
node_name	string	Filter by node name
phase	string	Filter by pod phase
page	int	Page number (default: 1)
per_page	int	Items per page (default: 50)

List Node Pods

GET /api/v1/clusters/:clusterID/nodes/:nodeID/pods

Returns pods running on a specific node.

List Workload Pods

GET /api/v1/clusters/:clusterID/workloads/:workloadID/pods

Returns pods belonging to a specific workload.

List Cluster Services

GET /api/v1/clusters/:clusterID/services

List Cluster Ingresses

GET /api/v1/clusters/:clusterID/ingresses

List Cluster PVCs

GET /api/v1/clusters/:clusterID/pvcs

List Cluster Events

GET /api/v1/clusters/:clusterID/events

Query parameters:

Parameter	Type	Description
type	string	Filter by event type (Normal, Warning)
page	int	Page number (default: 1)
per_page	int	Items per page (default: 50)

List Cluster Jobs

GET /api/v1/clusters/:clusterID/jobs

List Cluster HPAs

GET /api/v1/clusters/:clusterID/hpas

List Cluster Network Policies

GET /api/v1/clusters/:clusterID/network-policies

List Cluster Resource Quotas

GET /api/v1/clusters/:clusterID/resource-quotas

List Cluster Endpoint Slices

GET /api/v1/clusters/:clusterID/endpoint-slices

Client-Scoped Clusters

GET /api/v1/clients/:id/clusters

Returns clusters belonging to the specified client.

Environment-Scoped Clusters

GET /api/v1/environments/:envID/clusters

Returns clusters in the specified environment.

Node-Host Linkage

The KubernetesWorker automatically links K8s nodes to existing Proxima hosts when possible. This enables cross-referencing between Kubernetes node data and host-level inventory, metrics, and compliance data.

How It Works

For each node in a cluster inventory message, the worker attempts to find a matching host in the same environment by:

1. Hostname match — Compares the node name against hosts.hostname
2. IP match — Compares node addresses against hosts.ip_address

If a match is found, k8s_nodes.host_id is set to the matching host's ID. This linkage is tracked by the proxima_k8s_node_host_linkage_total Prometheus metric with a result label (linked or unlinked).

PQL Integration

The cluster.* PQL prefix uses this linkage to find hosts that are K8s nodes:

cluster.name = "k8s-prod-01" AND host.os = "Ubuntu"

This query joins hosts -> k8s_nodes -> clusters, so only hosts that are linked as K8s nodes will match cluster.* filters.

Stale Cluster Detection

The K8sMaintenanceWorker runs periodically and detects clusters whose collector has stopped reporting:

• Threshold: If clusters.last_seen_at is older than 15 minutes (configurable), the cluster's agent_status is set to disconnected.
• Event cleanup: The maintenance worker also deletes stale K8s events from PostgreSQL (older than the event retention period, default: 24 hours).
• Metric: proxima_k8s_stale_clusters_marked_total tracks how many clusters have been marked disconnected.

Cluster Status and Asset Status Mapping

Cluster health status is mapped to the CMDB asset status when upserting the asset record:

Cluster Status	Asset Status
healthy	active
degraded	degraded
unreachable	unreachable
unknown	active

CLI Seeder

The scripts/k8s-seed.sh script publishes a test Kubernetes NATS message for development and E2E testing. If kubectl is available and connected to a cluster, it reads real cluster data. Otherwise, it uses hardcoded test data.

# Default: internal / development / test-cluster
./scripts/k8s-seed.sh

# Custom slugs
./scripts/k8s-seed.sh acme production k8s-prod

# Custom NATS server
NATS_URL=nats://remote:4222 ./scripts/k8s-seed.sh

Requirements: nats CLI, jq. Optionally kubectl for real cluster data.

Observability

Prometheus Metrics

Metric	Type	Description
proxima_k8s_clusters_synced_total	counter	Total cluster inventory syncs processed
proxima_k8s_sync_duration_seconds	histogram	Time to process a kubernetes message
proxima_k8s_nodes_synced_total	counter	Total node upserts across all syncs
proxima_k8s_workloads_synced_total	counter	Total workload upserts across all syncs
proxima_k8s_node_host_linkage_total	counter	Node-host linkage attempts and results (label: result)
proxima_k8s_stale_clusters_marked_total	counter	Clusters marked disconnected by maintenance worker

OTel Spans

The KubernetesWorker creates spans for each processing step:

• k8s.process_message — Top-level span for the entire message
• k8s.upsert_cluster — Cluster and asset upsert
• k8s.sync_nodes — Node sync with host linkage
• k8s.sync_namespaces — Namespace sync
• k8s.sync_workloads — Workload sync
• worker.KubernetesPruneUnseen — Prune phase

Permissions

Cluster endpoints reuse the existing CMDB permissions:

Permission	Granted To	Description
assets:read	All system roles	View clusters and sub-resources
assets:write	Super Admin, Admin	Future: create and modify clusters

PQL Search

The cluster.* prefix is available in PQL for searching hosts by their associated cluster:

Field	Type	Description	Example
cluster.name	string	Cluster name	cluster.name = "k8s-prod-01"
cluster.version	string	K8s version	cluster.version ~ "v1.29*"
cluster.platform	string	Distribution	cluster.platform = "rke2"
cluster.status	string	Health status	cluster.status = "healthy"

The cluster.* prefix uses a two-table JOIN chain (k8s_nodes -> clusters), so only hosts that are linked as K8s nodes will match these filters.

See the PQL Reference for full syntax documentation.