Execution Monitoring

When a runbook is triggered, an execution record tracks its progress from start to finish.

Execution Lifecycle

Execution Statuses

Status	Description
pending	Created and dispatched to the agent. Waiting for the first step to start.
running	At least one step has started executing.
completed	All steps finished successfully (or with warn/skip policies).
failed	A step with on_failure: abort failed, or steps failed and the overall result is failure.
timed_out	Execution exceeded the configured timeout.
cancelled	User or API cancelled the execution.

Step Statuses

Status	Description
pending	Step has not started yet.
running	Step is currently executing.
completed	Step finished successfully.
failed	Step execution returned an error.
skipped	Step was skipped (condition was false, or on_failure: skip).

Per-Step Streaming

Step results stream back through NATS JetStream as each step completes:

1. Agent finishes a step and publishes a runbook.step message with the result.
2. The backend RunbookWorker consumes the message and persists the step result to PostgreSQL.
3. The frontend polls GET /api/v1/runbook-executions/{id} and displays updated step statuses.

Each step result includes:

• Input -- The resolved parameters that were passed to the action.
• Output -- The structured result map returned by the action (varies by action type).
• Error -- Error message if the step failed.
• Timestamps -- When the step started and completed.

Timeout Behavior

Per-Execution Timeout

Each runbook has a configurable timeout_sec (default: 300 seconds / 5 minutes). The agent enforces this with a context deadline. If the timeout expires mid-execution:

• The current step's context is cancelled.
• Remaining steps are skipped.
• The agent publishes a completion message with status timed_out.

Watchdog

The backend runs a RunbookWatchdog that checks for stale executions every 60 seconds. An execution is considered stale if:

• Its status is pending or running.
• It has exceeded its timeout_sec duration since creation.

Stale executions are automatically transitioned to timed_out. This catches cases where the agent crashes or loses connectivity before publishing a completion message.

Rate Limiting

Executions are rate-limited per host to prevent runaway automation:

• Default limit: 5 executions per host per 10 minutes.
• The backend checks recent execution count for the target host before dispatching.
• If the limit is exceeded, the API returns a 429 Too Many Requests error.
• Rate limit parameters are configurable on the backend.

Cancellation

Running executions can be cancelled:

• Frontend: Click the cancel button on the execution detail view.
• API: POST /api/v1/runbook-executions/{id}/cancel
• Permission required: runbooks:execute

Cancellation sends a cancel command to the agent via NATS. The agent closes the execution's cancel channel, which cancels the context for any in-flight action handler.

Viewing Execution History

Runbooks Page -- Executions Tab

Navigate to Automation > Runbooks and select the Executions tab. This shows all executions across all runbooks, sorted by most recent.

Each row shows:

• Runbook name and slug
• Target hostname
• Status badge
• Trigger source (manual, API, AI chat)
• Timestamps

Execution Detail

Click an execution to see the full detail view:

• Execution metadata (runbook version, parameters, trigger source)
• Per-step status list with input/output/error details
• Timeline showing step durations

Filtering

Executions can be filtered by:

• Runbook name
• Status (pending, running, completed, failed, timed_out, cancelled)
• Host
• Trigger source

Trigger Sources

Source	Description
manual	User clicked "Execute" in the frontend.
api	Triggered via REST API call.
ai_chat	Triggered by an LLM via the MCP run_runbook tool.

Monitoring in the UI

This section describes how to monitor runbook executions through the frontend.

Execution Detail Page

Click any execution (from the Runbooks page or host detail page) to open the execution detail view. The page displays:

• Header -- Runbook name, execution ID, target host, trigger source, and overall status badge.
• Parameters -- The parameter values that were passed when the execution was triggered.
• Step-by-Step Timeline -- A vertical timeline showing each step in execution order. Each step displays its name, action type, status badge, duration, and expandable input/output/error details. Completed steps show a checkmark, failed steps show an error icon, and pending steps are dimmed.

Live Polling

While an execution is in pending or running status, the detail page automatically polls the backend every 3 seconds for updated step results. This means you can watch steps complete in near real time without refreshing. Polling stops automatically once the execution reaches a terminal status (completed, failed, timed_out, or cancelled).

Cancelling a Running Execution

To cancel an execution that is still in progress:

1. Open the execution detail page.
2. Click the Cancel button in the header area.
3. Confirm the cancellation in the dialog.

The cancel command is sent to the agent via NATS, which terminates the in-flight action. The execution status changes to cancelled. You need the runbooks:execute permission to cancel executions.

Viewing Execution History from the Runbooks Page

1. Navigate to Automation > Runbooks in the sidebar.
2. Select the Executions tab at the top of the page.
3. The table shows all executions across all runbooks, sorted by most recent first.
4. Use the filter controls to narrow results by runbook name, status, host, or trigger source.
5. Click any row to navigate to the execution detail page.

Viewing Host-Specific Execution History

1. Navigate to any host's detail page (via Assets > Hosts, then click a host).
2. Select the Runbooks tab in the host detail sidebar.
3. This tab shows all runbook executions that targeted this specific host, sorted by most recent.
4. Click any execution to open its detail page.