How It Works

This page describes the internal architecture of the Proxima Agent, covering the startup sequence, enrollment flow, config fetching, command handling, change detection, scheduling, and message publishing.

Communication Architecture

The agent communicates with the backend over NATS using three distinct patterns:

Pattern	Direction	Subject	Purpose
Request-reply	Agent → Backend	proxima.system.agent.config.<agentID>	Startup config fetch (all config types)
Publish (JetStream)	Agent → Backend	proxima.{client}.{env}.{agent_id}.*	Telemetry: inventory, metrics, heartbeat, processes, changes
Publish-subscribe	Backend → Agent	proxima.system.commands.{agent_id}	Runtime command push (config updates, future: remote commands)

Startup Sequence

When the agent process starts, it executes the following steps in order:

Enrollment creates an agent record (in the agents table) and returns an agent_id. There is no separate NATS registration step. The host record (in the hosts table) is created later by the InventoryWorker when the first inventory message arrives. See NATS Security -- Agent Enrollment for the full enrollment protocol.

Steps

1. Load configuration -- reads startup configuration from environment variables (including PROXIMA_AGENT_LABELS JSON merge)
2. Enroll or load credentials -- on first run (no state.json), the agent enrolls over HTTPS using an install token. The backend generates the nkey pair server-side, upserts an agent record, issues a scoped NATS JWT, and returns agent_id, the nkey seed, credentials, and slugs in a single response. The agent persists all of this to state.json — it does not generate or store a separate agent.id or nkey.seed file. On subsequent runs, it loads state.json from disk. Re-enrolling a name that is already enrolled is rejected with 409 Conflict (AUTH H-1 identity-takeover protection); re-keying requires deleting the existing agent or using credential renewal.
3. Connect to NATS -- connects with nats.UserJWT() callbacks that read the current JWT + nkey seed from state.json on every (re)connect, plus TLS CA verification. Tolerates temporary NATS outages with auto-reconnect, and a credential renewed in state is applied on the next reconnect (see step 9).
4. Fetch remote configs -- sends a config fetch request via NATS request-reply with a 5-second timeout. If the backend responds with config overrides, they are applied immediately: labels (static + dynamic), intervals, collectors, and change detection configs are all applied before starting schedulers. Healthcheck and watch_files configs are also applied. If the fetch times out or fails, the agent gracefully falls back to its cached/default config only.
5. Start label manager -- initializes with local static labels, then applies any remote label config (replacing static labels and starting dynamic label goroutines). Launches goroutines for each dynamic label that run commands immediately, then on configured periods.
6. Start self-monitoring collector -- initializes the SelfMonCollector that tracks the agent's own CPU, memory, Go runtime, and NATS transport metrics
7. Start command dispatcher -- subscribes to proxima.system.commands.{agent_id} for runtime config updates and commands from the backend
8. Start schedulers -- begins periodic inventory collection, metrics collection, process snapshot collection, heartbeat publishing, and file change scanning (all include current labels from the label manager)
9. Start credential renewal -- launches a background goroutine that checks JWT expiry every hour and renews when the remaining TTL drops below 50%. On a successful renewal it forces a NATS reconnect so the live connection picks up the new JWT immediately (rather than staying pinned to the one captured at connect). See NATS Security -- JWT Renewal.

Enrollment

Enrollment is a single HTTPS call (POST /api/v1/agents/enroll) that creates an agent record in the agents table and returns the NATS credentials. The agent optionally sends agent_type ("host" or "collector") and a hostname. If no hostname is provided, the backend derives one from the nkey public key (agent-{first_12_chars}).

The state.json file written after enrollment contains:

Field	Description
agent_id	UUID from the agents table — the agent's identity
agent_type	"host" or "collector"
client_slug	Client identifier for NATS subject routing
environment_slug	Environment identifier for NATS subject routing
nats_url	NATS server URL
nkey_seed	Ed25519 nkey seed for signing (0600 permissions)
user_jwt	Scoped NATS user JWT
jwt_expires_at	JWT expiry timestamp

The host record in the hosts table is not created during enrollment. It is created by the InventoryWorker on the first inventory message. See NATS Security -- Agent Enrollment for the complete enrollment protocol.

Config Fetch at Startup

Immediately after enrollment, the agent fetches remote configuration from the backend using a NATS request-reply exchange. This allows operators to centrally manage agent configuration from the backend API without redeploying agents. See Config Sync for the full config sync system.

Request payload:

{
  "agent_id": "550e8400-e29b-41d4-a716-446655440000"
}

Response payload:

{
  "configs": {
    "watch_files": {
      "paths": ["/etc", "/opt/myapp/config"],
      "exclude_patterns": ["*.swp", "*.bak"],
      "sensitive_files": ["*shadow*", "*.key"],
      "max_file_size": 1048576
    },
    "healthcheck": {
      "interval_seconds": 60,
      "timeout_seconds": 5
    }
  },
  "versions": {
    "watch_files": 3,
    "healthcheck": 1
  }
}

Key design decisions:

• Two-level resolution -- the backend resolves effective configs by merging environment-level defaults with host-level overrides (shallow merge, host keys win). The agent receives the already-merged result.
• Disk cache for resilience -- on success, all configs are persisted to {data_dir}/config.cache.json. On failure, the agent falls back to cached configs, then to local YAML.
• Best-effort with graceful degradation -- the fetch has a 5-second timeout. If the backend is unavailable, the agent uses cached or local config.
• Generic config map -- the response returns all config types (keyed by config_type). Currently watch_files and healthcheck are supported.
• JWT authentication -- the request is authenticated via the agent's NATS JWT credentials (scoped to the host's subjects).

Command Channel (Runtime Push)

After startup, the agent subscribes to a per-host command channel for receiving runtime updates from the backend. This enables operators to push configuration changes via the API without restarting agents. See Config Sync for the full config sync system.

Command Envelope

All commands use a generic envelope format, making it easy to add new command types:

{
  "type": "config_update",
  "config_type": "watch_files",
  "version": 3,
  "payload": {
    "paths": ["/etc", "/opt/myapp/config"],
    "exclude_patterns": ["*.swp"],
    "sensitive_files": ["*shadow*"],
    "max_file_size": 1048576
  }
}

Field	Type	Description
type	string	Command type (e.g., config_update). Routes to the appropriate handler.
config_type	string	Sub-type for config updates (e.g., watch_files).
version	int	Config version (auto-incremented by the backend on each update).
payload	object	Command-specific data (JSON).

Dispatcher Architecture

The agent uses a generic command dispatcher that decouples command routing from command handling:

The Dispatcher subscribes to proxima.system.commands.{agentID} and routes incoming commands to registered handlers based on the type field:

1. Handler interface -- each handler implements CommandType() string and Handle(ctx, cmd) error
2. Registration -- handlers are registered at startup via dispatcher.Register(handler)
3. Routing -- the dispatcher unmarshals the command envelope, looks up the handler by type, and calls Handle()
4. Unknown types -- commands with no registered handler are logged and discarded (forward compatibility)

ConfigHandler

The ConfigHandler replaces the earlier single-purpose FileWatchHandler with a generic handler that routes by config_type:

1. Persists to cache -- every config update is saved to the disk cache before calling callbacks
2. Callback routing -- per-config-type callbacks are registered at startup (e.g., watch_files triggers scanner.UpdateConfig())
3. Resilience -- if the callback fails, the config is still cached for next restart

Thread-Safe Config Updates

Config update callbacks call Scanner.UpdateConfig() for watch_files. The scanner uses a sync.RWMutex to safely update configuration while scans may be running concurrently:

• UpdateConfig() acquires a write lock -- replaces paths, exclude patterns, sensitive file patterns, max file size, and recomputes derived exclude/sensitive lists
• Scan() acquires a read lock -- ensures it reads a consistent configuration snapshot throughout the entire scan cycle
• GetConfig() acquires a read lock -- returns a copy of the current configuration for inspection

Scheduler

The agent uses a ticker-based scheduler for periodic tasks. Each task runs immediately on startup and then repeats at the configured interval:

Task	Default Interval	Description
Inventory collection	Every 15 minutes	Gathers host info (uptime, timezone, virtualization, cloud provider/instance type), services, packages, ports, interfaces, users, labels, CPU/disk hardware, filesystems, containers, and security/config data (swap, SELinux/AppArmor, DNS, NTP, kernel params, locale, failed units, firewall rules, cron jobs, kernel modules, certificates, security items, GPU devices)
Metrics collection	Every 60 seconds	Runs all enabled collectors (system, PostgreSQL, etc.) and publishes aggregated metrics
Process snapshot	Every 30 seconds	Collects a full process list snapshot (PID, name, command, user, state, CPU%, memory%, RSS) via gopsutil
Heartbeat	Every 30 seconds	Sends a lightweight status update including uptime tracking, current labels, and config hash/version
File change scanner	Every 60 seconds	Scans configured paths for file changes (created, modified, deleted, metadata-only), publishes change events
Log collection	Every 10 seconds	Drains buffered log entries from journald + file tailers, publishes to NATS
Dynamic labels	Per-label (configurable)	Each dynamic label runs its command on its own period (e.g., kernel every 1h, load_avg every 30s)

The immediate-execution-on-startup behavior ensures the backend has current data as soon as the agent comes online, without waiting for the first interval to elapse.

Collectors

The agent uses a Collector interface for extensible metric gathering. Each collector implements Name() and Collect(ctx), returning a slice of metric items.

Built-in collectors:

Collector	Enabled	Metrics
System	Always	CPU, memory, disk, network (via gopsutil)
PostgreSQL	When postgresql.dsn is configured	Connections, replication lag, database sizes, cache hit ratio, transactions, dead tuples, pg_stat_statements
Nginx	When nginx.url is configured	Active connections, accepted/handled connections, request rate, reading/writing/waiting states

The metrics task aggregates results from all enabled collectors into a single message and sends it via the Publisher interface. If a collector fails (e.g., the PostgreSQL server is unreachable or the Nginx stub_status endpoint returns an error), it is skipped for that cycle and the remaining collectors still run. This graceful degradation ensures a temporary subsystem outage does not block other metrics.

Future collectors (Redis) will follow the same pattern.

Process Collector

The process collector is a separate scheduled task (not a metric collector), similar to the inventory task. It uses gopsutil to enumerate all running processes and publishes the full list to the backend on a configurable interval (default 30 seconds).

For each process, the collector gathers:

• PID and PPID (parent process ID)
• Name and command line
• Username of the process owner
• State (running, sleeping, zombie, etc.)
• Thread count
• CPU percent and memory percent
• RSS and VMS memory in bytes
• Start time (used with PID for unique process identity across PID reuse)

Per-process errors are expected and handled gracefully -- zombie processes, kernel threads, and permission-denied errors are logged at debug level and skipped. The collector only returns an error if the initial process enumeration fails entirely.

The backend's ProcessesWorker receives the full process list, stores it in PostgreSQL, and exports top-N process metrics to VictoriaMetrics.

Change Detection

When enabled, the agent monitors configured file paths for changes. The filewatch scanner uses SHA-256 hashing to detect modifications without reading file content on every scan.

Architecture

Scan Modes

• Baseline -- first scan after startup (or after hash database reset). All watched files are published. The backend creates watched_files records but no change events.
• Delta -- subsequent scans. Only files with changed SHA-256 hashes are published, along with their content for diff computation.

File Processing

For each watched path, the scanner:

1. Resolves directories recursively and applies exclude patterns
2. Computes the SHA-256 hash of each file
3. Compares against the stored hash from the previous scan (loaded from the JSON hash DB)
4. For changed/new files: reads content (if not binary and under max_file_size), base64-encodes it
5. For deleted files: reports the path for soft-deletion on the backend

Hash Database

File hashes are persisted as a JSON file at {data_dir}/file_hashes.json. This provides:

• Crash resilience -- the agent can restart and resume delta scanning without re-sending a baseline
• Efficient comparison -- only a hash lookup is needed per file, not a full content comparison
• Automatic baseline -- if the database is missing or corrupted, the scanner falls back to baseline mode

NATS Subject

Change detection messages are published to:

proxima.{client}.{env}.{host}.changes

This subject is part of the EVENTS JetStream stream (same as inventory and heartbeat), ensuring reliable delivery with at-least-once semantics.

Version Reporting

Collectors can optionally implement the VersionReporter interface to report the version of the service they monitor:

type VersionReporter interface {
    Version() string
}

The PostgreSQL collector detects the server version via SELECT version() (e.g., "PostgreSQL 18.2"), and the Nginx collector parses it from the nginx -T output (e.g., "nginx/1.27.4"). Versions are included in the heartbeat collector status and displayed on the Integration Detail Page.

Change Detection (File Watch Scanner)

The file watch scanner monitors configured filesystem paths for changes. It runs on a periodic interval (default 60 seconds) and publishes change events to the backend.

How Scanning Works

Each scan cycle:

1. Acquires a file lock on the hash database to prevent concurrent scans
2. Loads the hash DB -- a JSON file (file_hashes.json) in the agent's data_dir that stores the last-known SHA-256 hash, size, mode, owner, and group for every scanned file
3. Walks all configured paths using filepath.WalkDir, processing regular files and symlinks
4. Compares against the hash DB to classify each file as created, modified, metadata_only, or unchanged
5. Detects deletions by finding paths in the hash DB that no longer exist on disk
6. Builds message batches -- if the total payload exceeds max_message_size (default 800 KB), it splits into multiple batches with sequence numbers
7. Saves the hash DB atomically and publishes change events to NATS

Scan Types

Scan Type	When	Behavior
Baseline	First scan (empty hash DB)	Records all files as created. File content is not included in the payload (too large).
Delta	All subsequent scans	Only changed files are reported. Non-sensitive, non-binary file content is gzip-compressed and base64-encoded in the payload for diff generation.

File Processing Rules

Rule	Behavior
Excluded patterns	Files matching exclude patterns (e.g., *.swp, *.log) are skipped. Default excludes include common temp files, lock files, and container-specific paths.
Sensitive files	Files matching sensitive patterns (e.g., *shadow*, *.key, *.pem) are tracked (hash, size, mode) but their content is never sent to the backend.
Binary files	Detected by checking for null bytes in the first 8 KB. Tracked but content is not sent.
Max file size	Files larger than max_file_size (default 1 MB) are skipped entirely.
Symlinks	Target path is recorded and hashed (not the target file content).

Configurable Watch Paths

The scanner's configuration can be managed from two sources:

1. Local YAML config -- agent.change_detection section in the agent config file (default paths, intervals, excludes)
2. Remote config from backend -- operators can update watch paths, exclude patterns, sensitive file lists, and max file size via the REST API (PUT /hosts/{id}/watch-config). Remote config overrides local YAML.

Config Priority and Override Behavior

The agent resolves its watch configuration in two phases with different merge strategies:

At startup (selective merge):

1. Agent loads the local YAML config file
2. Environment variables override specific YAML fields (PROXIMA_AGENT_CHANGE_DETECTION_ENABLED, _INTERVAL)
3. Agent fetches remote config from the backend via NATS request-reply (proxima.system.agent.config.<agentID>)
4. Remote fields are merged selectively -- only non-empty remote values replace the local config. If the remote config has no paths, the YAML paths are kept. If it has no exclude_patterns, the YAML excludes are kept. This means the YAML config acts as a fallback default for any fields not set remotely.
5. If the backend is unreachable or has no config for this host, the agent uses the YAML config as-is (5-second timeout, silent fallback).

At runtime (full overwrite):

When a config_update command arrives via the NATS command channel (proxima.system.commands.{agentID}), the behavior is different:

• paths -- replaced entirely if the update contains any paths (empty array is ignored)
• exclude_patterns -- replaced entirely (empty array clears all custom excludes)
• sensitive_files -- replaced entirely (empty array clears all custom sensitives)
• max_file_size -- replaced if the update value is non-zero

The YAML config is not re-read at runtime. Once a runtime update is applied, the scanner uses only the new values until the next update or restart.

Both paths feed into Scanner.UpdateConfig(), which applies changes under a read-write mutex so in-progress scans are not affected.

caution

A runtime update that sets only paths will clear any custom exclude_patterns and sensitive_files that were set by YAML or a previous update. Always include all fields you want to keep when pushing a runtime config update.

Default Patterns (Always Active)

Regardless of YAML or remote config, the scanner always includes built-in default patterns that cannot be removed:

• Default excludes -- temporary files, caches, package manager artifacts (e.g., *.swp, *.tmp, *~, .git/, __pycache__/)
• Default sensitives -- shadow files, SSH keys, TLS certificates, sudoers (e.g., *shadow*, *.key, *.pem)

Custom exclude and sensitive patterns from config are appended to these defaults, not replaced.

Log Collection

When enabled, the agent collects logs from journald and configured log files, buffering them and publishing to the backend on a periodic interval (default 10 seconds).

Architecture

Journald Collector

The journald collector spawns journalctl --output=json --follow --no-pager as a subprocess and reads stdout line-by-line:

• Parses JSON fields: MESSAGE, PRIORITY, _SYSTEMD_UNIT, __REALTIME_TIMESTAMP
• Maps syslog priority (0--7) to level strings (emerg/alert/crit/error/warn/notice/info/debug)
• Optionally filters to specific systemd units via --unit=X flags
• Saves cursor to {data_dir}/journald-cursor for resume after restart
• Thread-safe: background goroutine reads journal, Collect() drains buffer under mutex

File Tail Collector

The file tail collector monitors configured log files for new content:

• Opens file and seeks to end (or saved offset on restart)
• Polls for new lines every 500ms
• Detects file rotation by inode change -- reopens from beginning
• Persists offset and inode in {data_dir}/logs/ for crash recovery
• Truncates lines longer than max_line_len (default 8192 bytes)
• Auto-detects format: tries JSON first, then syslog RFC 3164/5424, falls back to plain text

Log Collector Orchestrator

The Collector type orchestrates all log sources:

1. Starts the journald collector (if enabled) and all configured file tailers
2. On each scheduler tick, calls Collect() to drain all sources into a single LogsPayload
3. If the payload has entries, publishes to NATS via the Publisher interface
4. Gracefully stops all sources on shutdown

NATS Subject

Log entries are published to:

proxima.{client}.{env}.{host}.logs

This subject is part of the TELEMETRY JetStream stream (same as metrics and processes), with 1-hour max age.

Cloud Detection

During inventory collection, the agent probes cloud metadata services to detect the hosting environment:

Provider	Endpoint	Timeout
AWS	IMDSv2 (169.254.169.254)	2s
GCP	Metadata API (metadata.google.internal)	2s
Hetzner	Metadata API (169.254.169.254/hetzner/)	2s
Azure	IMDS (169.254.169.254/metadata/instance)	2s
Oracle Cloud	OCI metadata (169.254.169.254/opc/v2/instance/)	2s
DigitalOcean	Metadata API (169.254.169.254/metadata/v1/)	2s

All six providers are tried in order (AWS, GCP, Hetzner, Azure, Oracle, DigitalOcean). On the first successful response, the agent extracts the provider name, region, and availability zone. On non-cloud hosts or if all probes fail, cloud fields are left empty -- no errors are raised.

Publisher Interface

The metrics task depends on a Publisher interface (defined in metrics/collector.go) rather than the NATS transport directly:

type Publisher interface {
    Publish(msgType string, payload interface{}) error
}

*transport.Transport satisfies this interface implicitly -- no adapter or wrapper is needed. In main.go, the transport is passed directly to NewTask(collectors, metas, tr).

This decoupling provides:

• Testability -- unit tests use a mockPublisher struct instead of a real NATS connection
• Single responsibility -- the metrics task only knows how to collect and publish, not how NATS works
• Extensibility -- the transport layer can be swapped (e.g., to a file-based publisher for debugging) without touching the metrics package

Outbound Channel Pattern

The transport layer uses a buffered channel with a capacity of 256 messages. A dedicated publisher goroutine reads from this channel and sends messages to NATS.

This design provides several benefits:

• Decouples collection from publishing -- collectors are not blocked by NATS latency
• Backpressure handling -- if the channel is full (e.g., NATS is down and messages are accumulating), new messages are dropped rather than blocking the collector goroutines. This prevents resource exhaustion on the host.
• Single writer -- only one goroutine interacts with the NATS connection for publishing, avoiding concurrency issues

NATS Subjects

The agent uses the following NATS subjects:

Outbound (Agent → Backend)

Subject	Purpose
proxima.{client}.{env}.{agent_id}.inventory	Full inventory snapshot (includes labels)
proxima.{client}.{env}.{agent_id}.metrics	System metrics batch (CPU, memory, disk, network)
proxima.{client}.{env}.{agent_id}.heartbeat	Periodic heartbeat with uptime and labels
proxima.{client}.{env}.{agent_id}.processes	Full process list snapshot + top-N metrics
proxima.{client}.{env}.{agent_id}.changes	File change events (baseline + delta scans)
proxima.{client}.{env}.{agent_id}.logs	Log entries (journald + file tailing)

System (Request-Reply)

Subject	Direction	Purpose
proxima.system.agent.config.{agent_id}	Agent → Backend	Fetch all remote configs for this agent

Inbound (Backend → Agent)

Subject	Purpose
proxima.system.commands.{agent_id}	Runtime commands (config updates, future: remote actions)

Where:

• {client} -- the client slug (e.g., bank-abc)
• {env} -- the environment slug (e.g., production)
• {agent_id} -- the UUID from the agents table, assigned during enrollment

The backend subscribes to outbound subjects (using wildcards) to process incoming data from all agents. The agent subscribes to its per-agent command subject for receiving runtime updates.