Runbook Overview

Runbooks are structured operational procedures that execute typed actions on managed hosts. They replace ad-hoc shell access with a controlled, auditable execution model.

Why Runbooks

Traditional infrastructure remediation relies on operators SSH-ing into hosts and running arbitrary commands. This creates several problems:

• No audit trail of what was run or why
• No risk assessment before execution
• No way for AI assistants to safely interact with infrastructure
• No parameter validation or safety checks

Runbooks solve these by defining procedures as YAML with typed actions, parameters, and conditional logic. Every execution is recorded, every action is validated, and risk is assessed before anything runs.

How It Works

1. Define -- Create a runbook with parameters, steps, and conditions in the two-column editor (form + YAML).
2. Execute -- Trigger a runbook on a target host. The backend validates permissions, risk tier, and rate limits, then publishes an execute command to the agent via NATS JetStream.
3. Run -- The agent executes each step sequentially. After each step, it publishes a step result back through NATS.
4. Monitor -- The backend worker persists step results to PostgreSQL. The frontend polls for updates and displays per-step streaming progress.
5. Complete -- When all steps finish (or one aborts), the agent publishes a completion message. The watchdog catches executions that go silent.

Risk Model

Every action has a risk tier. The runbook's risk tier must be at least as high as its most dangerous action.

Risk Tier	Behavior	Example Actions
read_only	Auto-execute without confirmation	read_file_tail, refresh_inventory
operational	Requires user confirmation	restart_systemd_unit, run_healthcheck
destructive	Requires admin role approval	kill_process, stop_systemd_unit

Safety Layers

Runbooks enforce safety at four levels:

1. Frontend validation -- The editor validates YAML structure, parameter types, and step references before saving.
2. Backend API validation -- The handler validates the definition against the action registry, checks risk tier consistency, enforces RBAC permissions, and applies per-host rate limiting (default: 5 executions per host per 10 minutes).
3. Agent validation -- The executor validates each step's parameters against the action's registered constraints (type, pattern, min/max, enum values) and enforces file path allowlists.
4. Full audit trail -- Every execution, every step result, and every parameter value is recorded in PostgreSQL with the triggering user, timestamp, and trigger source (manual, api, ai_chat).

Investigation Flow

Runbooks are designed to support the alert-to-resolution workflow:

Alert fires
    |
    v
AI Chat investigates (MCP tools: list_runbooks, preview_runbook)
    |
    v
AI suggests a runbook with parameters
    |
    v
Operator confirms (or auto-execute for read_only)
    |
    v
run_runbook executes on target host
    |
    v
get_runbook_status tracks per-step progress
    |
    v
Results feed back into investigation

This keeps humans in the loop for risky operations while allowing AI to discover, preview, and recommend procedures.

Key Concepts

Concept	Description
Action	A single typed operation (e.g., restart_systemd_unit). 10 built-in actions.
Step	One action invocation within a runbook, with params, conditions, and failure policy.
Runbook	A reusable procedure: metadata + parameters + ordered steps.
Version	Every save creates an immutable version snapshot. Executions reference the version used.
Execution	A single run of a runbook on a specific host. Tracks status and per-step results.
Client access	Runbooks can be global (all clients) or scoped to specific clients.