Escalation Dead-Man's Switch

The native escalation engine (On-call & Escalation Management) drives paging from a Postgres timer: a background worker ticks, finds escalations whose next step is due, and pages the on-call user. If that worker's goroutine ever wedges — a deadlock, a stuck DB connection, a panic-restart loop — escalations silently stop advancing and nobody gets paged. Because paging is the one thing that must not fail quietly, Console runs a second, independent guard: the escalation auditor, a dead-man's switch.

The auditor does two jobs on every tick:

1. Backup executor — it independently detects escalations whose step has fallen behind and, while they are still recoverable, re-plans and fires the page itself. A wedged timer worker no longer means a missed page.
2. External heartbeat — on a clean sweep it pings an outside URL (a cron monitor). If the whole Console process dies, or an escalation is truly stuck, the ping stops, and the outside monitor pages you about the pager.

This page is the operator guide: how to wire the heartbeat, what to alert on, and how to validate the whole safety net end-to-end.

Same enablement as the timer worker

The auditor is constructed inside the same ops-bot-token gate as the escalation timer worker. If Telegram paging is not configured (PROXIMA_OPS_TELEGRAM_BOT_TOKEN unset), there is no channel to page through, so both the timer worker and the auditor are disabled. There is no separate on/off switch.

How it decides: overdue, healing, stuck

Every tick the auditor takes a Postgres advisory lock (so exactly one replica sweeps per tick), then classifies each live-team escalation whose next_escalation_at has fallen behind:

Band	Condition	What the auditor does
On time	next_escalation_at in the future	Nothing — the timer worker owns it.
Healing	overdue by [overdue, dirty)	Recovers it: re-plans the step (rotating the fence token) and fires the page through the timer worker's own dispatch path — the backup executor. Counts as healed.
Stuck	overdue by ≥ dirty	Declared unrecoverable — not re-planned. Counts as stuck.

The [overdue, dirty) window is the healing zone: it gives the backup executor a few ticks to recover a merely-behind escalation before declaring it stuck. The backup page runs under the identical live-owner gate, idempotency ledger, and escalation token fence as the primary path, so it can never double-page.

Two classes are always suppressed, never healed and never counted stuck:

• Shadowed escalations — a team that has not been cut over to Console still pages via Grafana OnCall, so Console withholds.
• Un-owned escalations — a snapshot that froze no team (team_id absent/legacy). Never our page, by design.

The heartbeat rule

• Clean sweep (nothing stuck) → stamp proxima_escalation_auditor_last_success_timestamp, reset proxima_escalation_auditor_stuck to 0, and GET the heartbeat URL.
• Dirty sweep (≥1 stuck escalation) → set proxima_escalation_auditor_stuck to the stuck count and withhold the heartbeat.
• Sweep DB error → fail closed: withhold the heartbeat (a broken auditor must look dead, not healthy).

Withholding the ping is the signal. The outside monitor is configured to expect a ping on a schedule; when the pings stop — because the process died, an escalation is stuck, or the DB is unreachable — the monitor's own grace period elapses and it pages.

Configuration

All knobs are environment variables (Twelve-Factor; defaults in code):

Variable	Default	Meaning
PROXIMA_ESCALATION_AUDITOR_INTERVAL	60s	How often the auditor sweeps.
PROXIMA_ESCALATION_AUDITOR_OVERDUE_THRESHOLD	5m	A step overdue by this much enters the healing band.
PROXIMA_ESCALATION_AUDITOR_DIRTY_THRESHOLD	15m	A step still overdue by this much is unrecoverable (must exceed the overdue threshold, else every overdue escalation is treated as stuck immediately — the worker logs a WARN and proceeds).
PROXIMA_ESCALATION_AUDITOR_HEARTBEAT_URL	(empty)	The external dead-man's-switch URL. Empty disables the ping — the auditor still detects + heals + emits metrics, but nothing pings externally (detection-only).

Pick the interval shorter than the cron monitor's grace period (below), so a healthy auditor pings several times inside each grace window.

Wiring the heartbeat to a cron monitor (recommended)

Any "expect a ping on a schedule, alert on its absence" service works — Healthchecks.io, Better Uptime / Better Stack Heartbeats, Cronitor, Grafana Cloud Synthetics, Dead Man's Snitch. The pattern is identical.

1. Create a heartbeat/check in the monitor. Set its period to your sweep interval (e.g. 1 min) and its grace to a few multiples of that (e.g. 5–10 min) so a single missed tick doesn't false-alarm, but a genuinely dead auditor pages fast.
2. Copy the ping URL the monitor gives you (e.g. https://hc-ping.com/<uuid>).
3. Set it on the backend and restart:

   PROXIMA_ESCALATION_AUDITOR_HEARTBEAT_URL="https://hc-ping.com/<uuid>"

4. Point the monitor's own alert channel at a different medium than Console pages through. The whole value of a dead-man's switch is that it survives Console being down — so route the cron-monitor alert to email/SMS/PagerDuty/a second Telegram bot, not the same ops bot the escalation engine uses. If Console's Telegram path is what failed, a Telegram-only cron alert dies with it.

That is the entire external dependency: one outbound GET on a clean sweep. The auditor treats any non-2xx or transport error as a ping failure (logged + proxima_escalation_auditor_heartbeat_failures_total), and — importantly — a ping failure does not fail the sweep; the internal metrics are still authoritative.

Alternative: Grafana staleness alert (no external monitor)

If you cannot depend on an outside service, alert on the freshness of the last-success gauge in your own Grafana instead. This catches a dead/wedged auditor but not a total Console+Grafana outage, so the external cron monitor is still preferred as the primary.

Alert A — auditor stopped sweeping (staleness):

# Seconds since the last clean sweep. Fire when the auditor hasn't
# succeeded in > 5 minutes (tune to a few × the sweep interval).
time() - proxima_escalation_auditor_last_success_timestamp > 300

Alert B — an escalation is stuck (this is a real, active missed-page risk):

# Any live-team escalation overdue past the dirty threshold.
proxima_escalation_auditor_stuck > 0

Alert B is the high-signal one: it means the backup executor could not recover a page and a human on-call may already be missing an alert. Route it to your out-of-band channel and treat it as an incident. Alert A means the safety net itself is down.

Supporting panels (for the dashboard, not necessarily alerts):

sum(rate(proxima_escalation_auditor_sweeps_total[5m])) by (result)   # clean vs dirty vs error
sum(increase(proxima_escalation_auditor_healed_total[1h]))           # backup-executor recoveries
sum(increase(proxima_escalation_auditor_heartbeat_failures_total[1h]))

A rising healed_total is itself a signal worth watching: the backup executor should rarely have to act. If it fires regularly, the primary timer worker is intermittently falling behind — investigate that, don't just lean on the net.

Metrics reference

Metric	Type	Meaning
proxima_escalation_auditor_last_success_timestamp	gauge (unix s)	Wall-clock of the last clean sweep. Staleness = auditor down.
proxima_escalation_auditor_stuck	gauge	Live-team escalations overdue past the dirty threshold right now. > 0 = active missed-page risk.
proxima_escalation_auditor_healed_total	counter	Escalations the backup executor advanced/paged.
proxima_escalation_auditor_sweeps_total{result}	counter	Sweeps by result = clean | dirty | error.
proxima_escalation_auditor_heartbeat_failures_total	counter	Failed external heartbeat pings (non-2xx / transport error).

Query them live via the observability helper:

make obs-metrics PATTERN=proxima_escalation_auditor

Live-validation procedure

Run this once after enabling the heartbeat on the dev/staging server, to prove the whole net works before you rely on it. This is an operator step against a deployed server — it is not part of the automated test suite.

Prerequisites: a team that is cut over to Console (live), a schedule with a real on-call user, and PROXIMA_ESCALATION_AUDITOR_HEARTBEAT_URL set to a test cron monitor (create a throwaway Healthchecks.io check for this).

Phase 1 — backup executor pages, heartbeat stays green

1. Author a live-team escalation. Fire (or synthesize) a page-worthy alert for a source that routes into the live team's policy, so an escalation arms and its timer starts advancing.
2. Simulate a wedged timer worker. Pause only the escalation timer worker so its ticks stop while the auditor keeps running — the cleanest way is to let a step go overdue without the primary advancing it (e.g. stop the timer worker's goroutine / temporarily set PROXIMA_ESCALATION_POLL_INTERVAL very high and restart, or pause the process the timer worker runs in if it is isolated). The step's next_escalation_at will slide into the healing band.
3. Confirm the auditor healed it. Within a sweep interval or two:
   • the on-call user receives the page (from the backup executor, not the primary);
   • make obs-metrics PATTERN=proxima_escalation_auditor shows proxima_escalation_auditor_healed_total incremented and proxima_escalation_auditor_stuck at 0;
   • the cron monitor is still green (clean sweeps kept pinging).

This proves a wedged primary no longer drops a page.

Phase 2 — unrecoverable stuck: heartbeat stops, alert fires

4. Force an unrecoverable stuck group. Let an overdue live-team escalation age past PROXIMA_ESCALATION_AUDITOR_DIRTY_THRESHOLD without recovery (keep the primary paused and let the healing band elapse), so the auditor classifies it stuck rather than healing it.
5. Confirm the switch tripped. Within a sweep interval:
   • proxima_escalation_auditor_stuck reads ≥ 1 and proxima_escalation_auditor_sweeps_total{result="dirty"} is incrementing (make obs-metrics PATTERN=proxima_escalation_auditor);
   • the auditor stops pinging the heartbeat URL, so after its grace period the cron monitor pages your out-of-band channel;
   • your Grafana proxima_escalation_auditor_stuck > 0 alert (if configured) fires.

This proves that when the engine genuinely can't page, the absence of the heartbeat pages a human out-of-band.

Cleanup

6. Un-pause the timer worker (restore PROXIMA_ESCALATION_POLL_INTERVAL / restart), resolve or acknowledge the test alert so its escalation halts, and — once a clean sweep pings again — confirm the cron monitor returns to green and proxima_escalation_auditor_stuck returns to 0. Delete the throwaway cron monitor if it was temporary.

Troubleshooting

• Heartbeat never pings, even when healthy. Check PROXIMA_ESCALATION_AUDITOR_HEARTBEAT_URL is set (empty = detection-only, no ping) and that the ops bot token is set (an unset token disables the whole escalation stack). Look for escalation auditor started at boot and heartbeat-failure WARNs in the logs.
• stuck > 0 but no page went out. That is the point: the stuck group is unrecoverable by the backup executor (overdue past dirty). Treat it as an incident — inspect the group's escalation_snapshot/next_escalation_at, and check the primary timer worker's health.
• healed_total climbing steadily. The backup executor is compensating for a primary that keeps falling behind. Investigate the timer worker (DB latency, dispatch errors) — the auditor is a net, not a fix.
• Every overdue escalation is flagged stuck immediately. The dirty threshold was set at or below the overdue threshold, collapsing the healing band. Set DIRTY_THRESHOLD > OVERDUE_THRESHOLD; the worker logs a WARN when it detects this.