Config Sync

The agent config sync system enables operators to manage agent operational configuration centrally from the backend API, without redeploying or restarting agents. Configs are stored at two levels — environment defaults and per-host overrides — merged by the backend, and delivered to agents via both push and pull mechanisms.

Architecture Overview

Config Types

The backend maintains a type registry that validates and controls merge behavior for each config type. Only registered types are accepted by the API — requests with unknown types receive a 400 Bad Request response.

Type	Purpose	Data Shape
collectors	Metric collector definitions (e.g., MySQL, Redis, HTTP checks)	JSON array of collector objects
intervals	Polling intervals for metrics, inventory, heartbeat, processes	JSON object with duration strings
labels	Static key-value labels and dynamic label generators	JSON object with static and dynamic keys
change_detection	File change detection paths, intervals, and enable/disable toggle	JSON object
log_collection	Log file paths, journald units, and collection settings	JSON object with files and journald keys
agent_settings	Agent-level tuning: collector timeout, memory limits	JSON object
watch_files	File watch scanner paths, excludes, sensitive files, max file size	JSON object
healthcheck	Health check intervals and timeouts	JSON object
tracing	Distributed-tracing toggle and trace sample ratio (0–1)	JSON object
log_shipping	Agent self-log shipping toggle and minimum level (error/warn/info/debug)	JSON object

Source of truth

The type registry is defined in backend/internal/agentconfig/types.go. Each entry specifies a Validate function (called on API write) and a Merge function (called during config resolution).

Two-Level Config Model

Configs are stored at two levels in PostgreSQL:

Level	Table	Scope	Purpose
Environment	environment_configs	All hosts in an environment	Fleet-wide defaults
Host	host_configs	Single host	Per-host overrides

Both tables share the same structure: config_type, config_data (JSONB), and version (auto-incrementing on each update).

Config Resolution

When a config is needed for a specific host, the Resolver merges the environment-level default with any host-level override. The merge strategy depends on the config type.

Resolution Rules

Scenario	Result	Source Field
Environment config only	Use environment data as-is	"environment"
Host override only	Use host data as-is	"host_override"
Both exist	Type-specific merge — host keys take precedence	"merged"
Neither exists	No config (nil)	—

The version of the effective config is the maximum of the environment and host versions.

Type-Specific Merge Strategies

Each config type declares its own merge function. The backend applies the correct strategy automatically during resolution.

Shallow Object Merge

Applies to: intervals, agent_settings, change_detection, watch_files, healthcheck

Top-level JSON keys from the host override replace the corresponding environment keys. Keys present only in the environment config are preserved. Nested objects are replaced entirely, not recursively merged.

Collectors: Identity Merge by {type, name}

Applies to: collectors

Collectors are JSON arrays. Each entry is identified by its {type, name} pair. Merge rules:

1. Start with the environment collector list (preserving order).
2. For each host entry, find the environment entry with the same {type, name}.
3. If found, the host entry replaces the environment entry in-place.
4. If not found, the host entry is appended to the end.
5. If a host entry has "enabled": false, the matching environment entry is removed entirely.

// Environment: [{type: "mysql", name: "primary"}, {type: "redis", name: "cache"}]
// Host:        [{type: "mysql", name: "primary", interval: "30s"}]
// Result:      [{type: "mysql", name: "primary", interval: "30s"}, {type: "redis", name: "cache"}]

Disabling inherited collectors

To disable a collector defined at the environment level for a specific host, set "enabled": false in the host override:

{"type": "mysql", "name": "primary", "enabled": false}

This removes the collector from the merged result for that host only.

Labels: Nested Merge

Applies to: labels

Labels have two sub-sections with different merge behaviors:

• static — Shallow key-value merge (host keys override environment keys).
• dynamic — Array merge by name field (host entries with the same name override environment entries; new entries are appended).

Other top-level keys in the labels object use shallow merge.

Log Collection: Files by Name, Journald Shallow

Applies to: log_collection

• files — Array of file objects merged by name field (same identity-merge logic as labels dynamic).
• journald — Shallow object merge at the journald level. If both sides have units, the host value fully replaces the environment value.
• Other top-level keys use shallow merge.

Credential Resolution

Collector configs can reference stored credentials via a credential_id field. Before the backend pushes a resolved config to an agent, the Credential Resolver decrypts and injects secret fields inline.

Resolution Rules

Rule	Detail
Scope	Only applies to collectors config type. Other types are passed through unchanged.
All-or-nothing	If any single credential fails to resolve (not found, client mismatch, decryption error), the entire push for that host fails. No partial configs are delivered.
Client scoping	The credential's client_id must match the host's client. Cross-client credential references are rejected.
Environment scoping	If the credential has an environment_id, it must match the host's environment.
Field injection	Decrypted secret fields are injected directly into the collector config object.
ID removal	The credential_id key is removed from the payload before delivery.

Secrets never reach disk

Resolved secrets are injected into the NATS payload only. They are never written to the agent's disk cache. When the agent restarts and loads its cache from disk, collector entries that originally had credentials are flagged as redacted and skipped by the fallback resolver. The agent waits for the backend to re-deliver full credentials via NATS.

Delivery: Push + Pull

Configs reach agents through two complementary mechanisms:

Pull: Startup Config Fetch

On startup, the agent fetches all configs from the backend via NATS request-reply:

Request payload:

{"agent_id": "550e8400-e29b-41d4-a716-446655440000"}

Response payload:

{
  "configs": {
    "watch_files": {"paths": ["/etc", "/opt"], "exclude_patterns": ["*.swp"]},
    "healthcheck": {"interval_seconds": 60, "timeout_seconds": 5}
  },
  "versions": {
    "watch_files": 3,
    "healthcheck": 1
  }
}

Push: Runtime Config Updates

When an operator updates a config via the API, the backend pushes the resolved config to affected agents immediately:

Push is best-effort — errors are logged but do not fail the API request. If an agent is offline, it will receive the updated config on its next startup via the pull mechanism.

Fallback Chain

When the agent needs a config at startup, the Fallback Resolver walks a priority chain to find the best available value:

Priority	Source	Notes
1 (highest)	NATS push/pull	Fresh config from backend with resolved credentials
2	Disk cache	Persisted from last successful NATS delivery
3 (lowest)	nil	No config for this type; agent uses local YAML or built-in defaults

Redacted collector cache

Collector configs that were resolved with credentials are cached with redacted secret values after a restart. The fallback resolver skips these entries because redacted credentials cannot connect to real services. The agent waits for the backend to re-send full credentials via NATS.

Apply Acknowledgement

After applying (or failing to apply) a config update, the agent publishes an acknowledgement message to NATS. The backend persists the result for observability.

Ack payload:

{
  "agent_id": "550e8400-e29b-41d4-a716-446655440000",
  "config_type": "collectors",
  "version": 5,
  "status": "applied"
}

On failure:

{
  "agent_id": "550e8400-e29b-41d4-a716-446655440000",
  "config_type": "collectors",
  "version": 5,
  "status": "failed",
  "error": "dial tcp: connection refused"
}

The host_config_apply_status table uses a UNIQUE(host_id, config_type) constraint, so each UPSERT replaces the previous status for that host+type pair. This gives operators a per-type view of which config version each host last applied and whether it succeeded.

Boot Jitter

On startup, the agent sleeps for a random duration between 0 and 5 seconds before issuing its NATS config fetch request.

jitter := time.Duration(rand.Int63n(5000)) * time.Millisecond
time.Sleep(jitter)

Why jitter?

When many agents restart simultaneously (e.g., after a fleet-wide update or infrastructure event), they would all hit the backend and Vault at the same instant. The random jitter spreads these requests over a 5-second window, preventing a thundering herd on config resolution and credential decryption.

Agent Config Cache

The agent maintains a persistent disk cache at {data_dir}/config.cache.json for restart resilience:

[
  {"config_type": "watch_files", "config_data": {"paths": ["/etc"]}, "version": 3},
  {"config_type": "healthcheck", "config_data": {"interval_seconds": 60}, "version": 1}
]

Cache Behavior

Operation	Behavior
Startup	Load cache from disk (missing/corrupt file starts empty)
Remote fetch success	Populate cache with all received configs
Remote fetch failure	Fall back to cached configs
Command push received	Update cache entry, persist to disk
Writes	Atomic via temp file + rename (no partial writes)
Permissions	File: 0600, directory: 0700

Cache Fingerprinting

The cache computes two values reported in every heartbeat:

• config_hash — SHA-256 hash of all cached config data (sorted by type for determinism). Enables drift detection.
• config_version — Maximum version across all cached configs. Enables staleness detection.

Heartbeat Reporting

Every heartbeat includes the agent's current config state:

{
  "agent_version": "1.0.0",
  "config_hash": "a1b2c3d4e5f6...",
  "config_version": 3,
  "config_versions": {
    "collectors": 5,
    "intervals": 2,
    "labels": 1,
    "watch_files": 3
  }
}

Field	Type	Purpose
config_hash	string	SHA-256 of all cached config data. Enables drift detection across the fleet.
config_version	int	Max version across all cached configs. Quick staleness check.
config_versions	map[string]int	Per-type version map. Allows operators to verify that each individual config type is at the expected version.

The backend persists these values in the hosts table on each heartbeat, enabling operators to query which hosts are running outdated or mismatched configurations.

ConfigHandler (Agent)

The ConfigHandler is a generic command handler that routes config_update commands by config_type:

Handle Flow

1. Validate config_type is non-empty
2. Call cache.Put(configType, payload, version) to persist
3. Look up registered callback for the config type
4. Call callback (if registered) for hot-reload
5. Publish apply acknowledgement (success or failure) to NATS
6. If no callback is registered, the config is still cached for next restart

Hot-Reload Capabilities

All 10 config types have registered callbacks:

Config Type	Hot-Reload Behavior
labels	Replaces static labels and restarts dynamic label goroutines via labels.Manager.Update()
intervals	Sends new duration to running scheduler goroutines via channels (scheduler.UpdateInterval())
collectors	Swaps plugin collectors while preserving system collectors (metrics.Task.ReplacePluginCollectors())
change_detection	Forwards to scanner.UpdateConfig() and updates scan interval
watch_files	Forwards to file watch scanner.UpdateConfig()
healthcheck	Forwards to healthcheck runner.UpdateConfig()
log_collection	Logs a restart-required warning (Fluent Bit requires process restart)
agent_settings	Applies dynamic log level change by updating a slog.LevelVar that backs the active handler
tracing	Applies the new tracing toggle and sample ratio to the live OTel provider (tracing.Provider.Apply())
log_shipping	Toggles agent self-log shipping and adjusts the minimum level on the running controller

Startup Config Application

Remote configs are also applied at agent startup (not just on subsequent reloads). After the agent fetches its initial config from NATS/cache, it applies labels, intervals, collectors, and change_detection configs before starting the scheduler. This ensures console-managed configs take effect immediately on boot.

Registering Callbacks

configHandler.RegisterCallback("labels", func(ctx context.Context, configType string, data json.RawMessage, version int) error {
    return configsync.ReloadLabels(ctx, labelMgr)(ctx, configType, data, version)
})

Callbacks enable live config application without agent restart. Components use sync.RWMutex or channel-based updates to safely apply new configuration while tasks may be running concurrently.

API Endpoints

Environment-Level Configs

Method	Path	Description
GET	/api/v1/environments/{envID}/agent-configs	List all configs for an environment
GET	/api/v1/environments/{envID}/agent-configs/{configType}	Get a specific config
PUT	/api/v1/environments/{envID}/agent-configs/{configType}	Create or update a config (pushes to all online hosts)
DELETE	/api/v1/environments/{envID}/agent-configs/{configType}	Delete a config

Host-Level Overrides

Method	Path	Description
GET	/api/v1/hosts/{hostID}/agent-configs/{configType}	Get effective (resolved) config for a host
PUT	/api/v1/hosts/{hostID}/agent-configs/{configType}	Create or update a host override (pushes to agent)
DELETE	/api/v1/hosts/{hostID}/agent-configs/{configType}	Delete a host override (reverts to environment config)

Request/Response Examples

PUT environment config:

curl -X PUT /api/v1/environments/{envID}/agent-configs/watch_files \
  -d '{"config_data": {"paths": ["/etc", "/opt"], "exclude_patterns": ["*.log"]}}'

Response (200):

{
  "data": {
    "id": "...",
    "environment_id": "...",
    "config_type": "watch_files",
    "config_data": {"paths": ["/etc", "/opt"], "exclude_patterns": ["*.log"]},
    "version": 1
  }
}

GET effective config for a host:

{
  "data": {
    "config_type": "watch_files",
    "config_data": {"paths": ["/etc", "/opt", "/var/log"], "exclude_patterns": ["*.log"]},
    "version": 3,
    "source": "merged"
  }
}

Resilience

Scenario	Behavior
Agent offline during push	Config cached in DB; agent pulls on next startup
Backend unreachable at startup	Agent falls back to disk cache (via fallback chain)
Disk cache missing or corrupt	Agent starts with no cached configs; uses local YAML
Disk cache has redacted collectors	Fallback resolver skips; agent waits for NATS re-delivery
NATS connection lost	Agent reconnects automatically; dispatcher re-subscribes
Push publish fails	Logged but does not fail the API request
Credential resolution fails	All-or-nothing: entire push for that host is skipped
Callback error	Logged; config is still cached; apply ack reports "failed"

Key Files

File	Description
backend/internal/agentconfig/types.go	Config type registry with validation and merge functions
backend/internal/agentconfig/resolver.go	Two-level config resolution with type-specific merge
backend/internal/agentconfig/credential_resolver.go	Credential resolution for collector configs
backend/internal/domain/environment_config.go	Domain types (EnvironmentConfig, EffectiveConfig)
backend/internal/store/environment_config_store.go	PostgreSQL CRUD for environment configs
backend/internal/store/config_apply_store.go	CRUD for host_config_apply_status table
backend/internal/api/agent_config_handler.go	REST API handler for all config endpoints
backend/internal/api/command_push.go	NATS publish helper for config commands
backend/internal/worker/config_fetch.go	Request-reply worker for agent config pulls
backend/internal/worker/config_apply.go	NATS subscriber for apply acknowledgements
agent/internal/configcache/cache.go	Persistent disk cache with hash/version
agent/internal/command/config_handler.go	Generic config command handler with callbacks and apply ack
agent/internal/configsync/fallback.go	Fallback resolver (NATS > cache > nil)