Runbook Overview

Runbooks are structured operational procedures that execute typed actions on managed hosts. They replace ad-hoc shell access with a controlled, auditable execution model.

Why Runbooks

Traditional infrastructure remediation relies on operators SSH-ing into hosts and running arbitrary commands. This creates several problems:

• No audit trail of what was run or why
• No risk assessment before execution
• No way for AI assistants to safely interact with infrastructure
• No parameter validation or safety checks

Runbooks solve these by defining procedures as YAML with typed actions, parameters, and conditional logic. Every execution is recorded, every action is validated, and risk is assessed before anything runs.

How It Works

1. Define -- Create a runbook with parameters, steps, and conditions in the two-column editor (form + YAML).
2. Execute -- Trigger a runbook on a target host. The backend validates permissions, risk tier, and rate limits, then publishes an execute command to the agent via NATS JetStream.
3. Run -- The agent executes each step sequentially. After each step, it publishes a step result back through NATS.
4. Monitor -- The backend worker persists step results to PostgreSQL. The frontend polls for updates and displays per-step streaming progress.
5. Complete -- When all steps finish (or one aborts), the agent publishes a completion message. The watchdog catches executions that go silent.

Risk Model

Every action has a risk tier. The runbook's risk tier must be at least as high as its most dangerous action.

Risk Tier	Example Actions
read_only	read_file_tail, refresh_inventory
operational	restart_systemd_unit, run_healthcheck
destructive	kill_process, stop_systemd_unit

No tier auto-executes

The risk tier does not gate confirmation. Every execution from the UI goes through ExecuteRunbookDialog regardless of tier — the tier only selects the banner and button styling. In AI chat, confirmation is required for run_runbook unconditionally, not by tier. Risk tier is a labelling and authoring constraint (a runbook's tier must be at least its most dangerous action), not an execution gate.

Safety Layers

Runbooks enforce safety at four levels:

1. Frontend validation -- The editor validates name, slug, risk tier, timeout and step presence before saving. Step references, on_failure values, duplicate step ids and unknown actions are caught server-side.
2. Backend API validation -- The handler validates the definition against the action registry, checks risk tier consistency, enforces RBAC permissions, and applies per-host rate limiting (default: 5 executions per host per 10 minutes).
3. Agent validation -- The executor validates each step's parameters against the action's registered constraints (type, pattern, min/max, enum values) and enforces file path allowlists.
4. Full audit trail -- Every execution, every step result, and every parameter value is recorded in PostgreSQL with the triggering user, timestamp, and trigger source (manual, api, ai_chat, l1-remediation).

Investigation Flow

Runbooks are designed to support the alert-to-resolution workflow:

Alert fires
    |
    v
AI Chat investigates (MCP tools: list_runbooks, preview_runbook)
    |
    v
AI suggests a runbook with parameters
    |
    v
Operator confirms (always — no tier auto-executes)
    |
    v
run_runbook executes on target host
    |
    v
get_runbook_status tracks per-step progress
    |
    v
Results feed back into investigation

This keeps humans in the loop for risky operations while allowing AI to discover, preview, and recommend procedures.

Key Concepts

Concept	Description
Action	A single typed operation (e.g., restart_systemd_unit). 10 built-in actions.
Step	One action invocation within a runbook, with params, conditions, and failure policy.
Runbook	A reusable procedure: metadata + parameters + ordered steps.
Version	Every save creates an immutable version snapshot. Executions reference the version used.
Execution	A single run of a runbook on a specific host. Tracks status and per-step results.
Client access	Runbooks can be global (all clients) or scoped to specific clients.