On-call Resolution Safety

Even a well-authored on-call setup can have gaps that prevent a page from reaching anyone: an alert matches no route, a schedule resolves to nobody, or a rotation is left empty. This page describes the four observability signals that surface these gaps — so operators can find and fix them before an incident proves the gap the hard way.

Observability only — these signals never page

All four gaps are captured as metrics + structured logs. None of them trigger a page. An unrouted alert is not the engine's job to escalate — it is a configuration problem to detect and fix. Wire the metrics below into Grafana alerts and you will find gaps ahead of a real incident.

G1 — Unrouted alert (default fallback)

An alert that fires for a client with no matching escalation route is a silent loss: nobody is paged, no record is written, and the alert continues to fire indefinitely with no on-call response.

Console now captures every such alert with:

• Metric: proxima_escalation_unrouted_total{severity} — incremented once per unrouted alert group. Labels the severity tier (P1–P5) so you can distinguish a missed P1 from a routine P5.
• Log (WARN): structured, carries alert_group_id, client_id, grouping_key, and severity.

The alert itself is not paged — an unrouted alert means the routing table has no policy to send it to, so there is nowhere to page to. The intended response is to author the missing route.

Recommended Grafana alert

# Any unrouted alert in the last 15 minutes — treat a spike as a routing gap.
increase(proxima_escalation_unrouted_total[15m]) > 0

To drill into which client is affected, filter logs by client_id:

./scripts/obs.sh logs '"escalation unrouted"' 1h

G2 — Reactive no-on-call signal

An escalation policy can reach a NOTIFY_SCHEDULE step whose schedule resolves to nobody at that moment: the schedule may have no rotation covering the current time, or no user may be assigned to the active slot. Previously this was a silent WARN-skip; the step advanced without any signal.

Console now emits:

• Metric: proxima_escalation_no_oncall_total{reason} — reason is one of:
  • missing_schedule — the schedule referenced by the step does not exist.
  • no_oncall — the schedule exists but resolves to nobody right now.
• Log (ERROR): structured, carries schedule_id, team_id, alert_group_id, and client_id (plus error on a schedule-load failure) so you can tie the no-on-call event to the specific firing incident.

The step still advances after emitting the signal — the escalation does not stall. But the no-oncall event is now observable instead of silent.

Recommended Grafana alert

# Any no-oncall resolution in the last 15 minutes — means a live page hit a gap.
increase(proxima_escalation_no_oncall_total[15m]) > 0

A no_oncall event during an active incident is high-priority: a page was attempted and nobody received it.

G3 — Schedule Auditor (proactive gap detection)

The ScheduleAuditorWorker sweeps all live-team schedules ahead of time, looking for coverage gaps and empty rotations before they cause a missed page. It runs on a configurable interval and emits metrics and structured WARN logs for every issue found.

The auditor holds a Postgres advisory lock (one replica sweeps per tick) and is enabled automatically when the ops-bot token is configured — the same gate as the escalation timer worker.

Observability only

The auditor never pages and never pings the dead-man's-switch heartbeat. A coverage gap is a configuration problem, not an engine failure.

Metrics emitted

Metric	Type	Meaning
proxima_oncall_schedule_gap{schedule,team}	counter	A coverage gap was found in the look-ahead window for this schedule.
proxima_oncall_empty_rotation{schedule,team}	counter	A rotation in this schedule has no users assigned.
proxima_schedule_auditor_sweeps_total{result}	counter	Sweep outcomes: clean | error.
proxima_schedule_auditor_last_success_timestamp	gauge (unix s)	Wall-clock of the last successful sweep. Staleness = auditor down.

The schedule and team labels carry the schedule id and team id (UUIDs), so you can identify the affected schedule directly from the metric.

Structured log fields (WARN)

Each gap emits a structured WARN log with:

• schedule_id — which schedule has the gap (UUID).
• team_id — which team owns it (UUID).
• gap_start, gap_end, leading_open, trailing_open — the uncovered window and whether it is unbounded at either edge (coverage gaps only).
• rotation_id — the empty rotation (empty-rotation warnings only).

Query gaps in the look-ahead window:

./scripts/obs.sh logs '"schedule gap"' 24h
./scripts/obs.sh logs '"empty rotation"' 24h

Recommended Grafana alerts

# Any coverage gap found in the last sweep window.
increase(proxima_oncall_schedule_gap[1h]) > 0

# Any empty rotation found in the last sweep window.
increase(proxima_oncall_empty_rotation[1h]) > 0

# Auditor stopped running (staleness guard).
time() - proxima_schedule_auditor_last_success_timestamp > 7200

Environment variables

All three knobs are optional — the defaults are safe for production:

Variable	Default	Meaning
PROXIMA_SCHEDULE_AUDITOR_INTERVAL	1h	How often the auditor sweeps.
PROXIMA_SCHEDULE_AUDITOR_WINDOW	336h (14 days)	How far ahead the auditor looks for gaps.
PROXIMA_SCHEDULE_AUDITOR_GAP_TOLERANCE	60s	Gaps shorter than this are ignored (handoff jitter / sub-minute transitions).

Querying live

make obs-metrics PATTERN=proxima_oncall_schedule
make obs-metrics PATTERN=proxima_schedule_auditor

G4 — Schedule guards (configuration-time enforcement)

Two guards prevent the most common misconfiguration classes at write time, before they cause a silent resolution failure.

by_day is required for daily and weekly rotations

For frequency=daily or frequency=weekly, the by_day field is now required. This routes these rotations through the DST-correct masked resolver, which ensures handoffs stay at the correct local wall-clock time across daylight-saving transitions.

For a 24/7 rotation that covers every day, set by_day to all seven weekdays:

"by_day": ["MO", "TU", "WE", "TH", "FR", "SA", "SU"]

A backfill migration (000149) added an all-days mask to existing daily/weekly rotations that had no by_day set, so all pre-existing schedules are already compliant.

Attempting to create or update a daily/weekly rotation without by_day returns 400.

You cannot empty a rotation of an enabled schedule

Removing all users from a rotation that belongs to an enabled schedule is now rejected with 409. An empty rotation resolves to nobody — the same state as the no_oncall signal — so the guard prevents this class of silent coverage gap from being authored.

To empty a rotation:

1. Disable the schedule (PATCH /api/v1/oncall/schedules/{id} with "enabled": false).
2. Remove the users from the rotation.
3. Re-enable (or delete) the schedule as appropriate.

G5 — Dispatch delivery completeness (orphaned claim auditor)

Every notification Console sends goes through an idempotency ledger: it claims the send slot (INSERT … status='sending'), calls the external channel (Telegram, voice), then marks the row terminal ('sent' or 'failed'). If the process crashes between the external call and the terminal write — a host OOM, a JVM kill, a kernel panic — the row is left permanently 'sending'. The next dispatch on the same key hits the ON CONFLICT DO NOTHING dedup fence and silently no-ops: the page is not re-sent, no error is logged, and the on-call engineer never hears about it. This is the orphaned dispatch claim problem.

What the auditor does

The DispatchCompletenessAuditor (a background singleton, held on a Postgres advisory lock so exactly one replica sweeps per tick) periodically scans notification_dispatch for rows that are still status='sending' after a configurable cutoff age. For each orphan it:

1. Marks the row terminal 'failed' — keeping it as the completeness record so the ledger is never left dangling.
2. Emits metrics and a structured ERROR log (dispatch_id, alert_group_id, epoch, channel, target_ref, step, path, age_seconds) so the event is observable.
3. Does not re-send.

No re-send — and why

A stuck 'sending' row is ambiguous: the crash may have happened after the external call succeeded — the Telegram message was delivered, the Twilio call went out — and marking it failed does not change that. Re-sending would double-page the on-call engineer for the same alert, which is worse than a late page.

The actual re-page guarantee lives elsewhere: the escalation ladder (P0-A) fires its REPEAT step on a live, fresh key that is not blocked by the orphan, and the notification chain worker (P0-B1) re-arms a user's next channel on a fresh step cursor. Both paths produce a new dispatch claim with a new key, so they are never blocked by an orphaned 'sending' row on the old key.

This replaces the previous ReapStuckSending reaper, which deleted orphaned rows with no alert — a silent discard that erased the evidence and produced no metric.

Metrics emitted

Metric	Type	Meaning
proxima_dispatch_orphans_total{channel,path}	counter	An orphaned claim was found and marked terminal. channel is the delivery channel (telegram, voice); path identifies the ledger step origin (escalation, memory, chain, or initial).
proxima_dispatch_auditor_sweeps_total{result}	counter	Sweep outcomes: clean (no orphans, no errors) | error (the sweep itself failed).
proxima_dispatch_auditor_last_success_timestamp	gauge (unix s)	Wall-clock of the last sweep that completed without error. Staleness means the auditor is down.

A rising proxima_dispatch_orphans_total rate is the key signal: it means the process is crashing between Send and Mark, and pages are losing their terminal write. The path label tells you which part of the paging stack the orphan came from.

Recommended Grafana alerts

# Any orphaned dispatch claim found in the last 15 minutes.
# A single orphan is a crash artifact. A sustained rate is a systemic delivery gap.
increase(proxima_dispatch_orphans_total[15m]) > 0

# Auditor stopped sweeping (staleness guard). Separate from findings.
time() - proxima_dispatch_auditor_last_success_timestamp > 300

Drill into which channel and path are affected:

make obs-metrics PATTERN=proxima_dispatch_orphans
make obs-metrics PATTERN=proxima_dispatch_auditor
./scripts/obs.sh logs '"orphaned dispatch claim"' 1h

Environment variables

Variable	Default	Meaning
PROXIMA_DISPATCH_AUDITOR_INTERVAL	1m	How often the auditor sweeps.
PROXIMA_DISPATCH_AUDITOR_CUTOFF	5m	A 'sending' row older than this is treated as an orphan. Must exceed the longest realistic Send latency.

The auditor is enabled under the same ops-bot-token gate as the escalation timer worker and notification chain worker. If Telegram paging is not configured, it is disabled.

Ack confirm-loop (after a page is acknowledged)

The gaps above (G1–G5) all concern getting a page out to someone. This section covers the other side: what happens after a human acknowledges the page. An ack is a promise ("I've got this"), and a promise can go quiet — the acker gets pulled onto something else, goes off shift, or simply forgets. The ack confirm-loop makes an acked-but-unresolved incident that goes silent a loud, escalating signal instead of a silent assumption that "someone's on it."

This loop does page — unlike G1–G5

G1–G5 are observability-only and never page. The ack confirm-loop is different: on silence it re-escalates (re-pages the chain), and in its terminal state it keeps nagging the acker. It replaced the old silent ack_watch auto-un-ack, which quietly un-acked and re-paged the whole chain when a timer fired — never first asking the acker whether they were still on it.

What replaced the old behavior

Previously, an ack with ack_timeout_minutes set entered an ack_watch state and, when the timer fired, the engine silently un-acked and re-paged the whole chain — it never asked the acker "still on it?" first. Worse, once the chain's repeat_count was exhausted the escalation halted with next_escalation_at = NULL: an acked, unresolved incident with nobody paged and nothing signalling it. Both are gone. An ack now converts the escalation timeline into a reminder loop that re-escalates only on silence and never ends in silence.

State machine

An ack arms a small state machine on the escalation timer (the escalation_snapshot.mode column — no new table, no migration):

State	What happens
ack_reminder	Every ack arms this (including a policy whose ack_timeout_minutes is null). On the reminder cadence, a "still on it?" DM/badge appears and the state moves to ack_confirm.
ack_confirm	A ⏰ Still on it? reminder has been sent; the loop waits for a confirm tap. If the acker confirms → back to ack_reminder (reminder re-armed, no re-page). If silent past the confirm window → un-ack + re-escalate (re-page the chain), bounded by repeat_count.
ack_abandoned	Re-escalations spent and still unresolved. A never-silent terminal state: it keeps nagging the acker (🚨 Unresolved and unconfirmed — please resolve or hand off) and holds an alertable metric until the incident resolves. next_escalation_at is never NULL. A confirm tap from here pulls the incident back into the gentle ack_reminder loop.

Resolving the incident anywhere clears the loop.

Confirm affordances (web + Telegram parity)

The acker confirms they're still on it from either channel — both call the same backend path (AlertService.ConfirmAck), which re-arms the reminder and rotates the escalation fence token so a concurrent stale silence-timeout can't also fire:

• Telegram: the ✅ Still on it inline button on the reminder DM.
• Web: POST /api/v1/alerts/{alertGroupID}/confirm (alerts:write, tenant-checked). The Alert and Incident detail pages render a "still on it?" confirm badge with a ✅ Still on it button (and a loud Abandoned banner in ack_abandoned). This is the load-bearing confirm surface.

Cadence env vars

Variable	Default	Meaning
PROXIMA_ACK_REMINDER_DEFAULT	30m	Reminder cadence used when the policy's ack_timeout_minutes is null (ack → first "still on it?").
PROXIMA_ACK_CONFIRM_WINDOW	10m	Silence window after a reminder is sent before it counts as no-response → re-escalate.
PROXIMA_ACK_ABANDONED_INTERVAL	60m	Nag cadence while a group is in ack_abandoned (re-nudge + re-emit the metric).

ack_timeout_minutes is the per-policy reminder cadence

When a policy sets ack_timeout_minutes (5–240), that value is the reminder cadence for its groups (frozen at arm time). Its meaning shifted from "silently un-ack after N minutes" to "ping 'still on it?' after N minutes." A null value falls back to the global PROXIMA_ACK_REMINDER_DEFAULT.

Metrics to alert on

Metric	Type	Meaning
proxima_escalation_ack_abandoned_total	counter	Alertable — a live abandoned incident. Incremented on entry to ack_abandoned and on each nag tick. A nonzero rate means an acked incident has gone unresolved and unconfirmed right now.
proxima_escalation_ack_reescalations_total	counter	A silence re-escalation: the acker went quiet past the confirm window and the chain re-paged. A rising rate means acks are not being followed through.
proxima_escalation_ack_reminders_total	counter	"Still on it?" reminder DMs planned/sent.
proxima_escalation_ack_reminder_failures_total	counter	Best-effort reminder-send failures (self-healed next tick).
proxima_escalation_ack_confirms_total{source}	counter	Confirm taps, source ∈ {telegram, web}.

Recommended Grafana alerts

# A live abandoned incident — acked, unresolved, nobody confirmed, re-escalations spent.
# This is the loudest ack-loop signal: page on it.
increase(proxima_escalation_ack_abandoned_total[15m]) > 0

# Acks that went silent and forced a re-escalation — a page reached someone who then dropped it.
increase(proxima_escalation_ack_reescalations_total[15m]) > 0

Drill into the loop live:

make obs-metrics PATTERN=proxima_escalation_ack_
./scripts/obs.sh logs '"ack_abandoned"' 24h

Liveness — P0-A covers the loop

The ack confirm-loop rides the same next_escalation_at timeline that the escalation dead-man's switch already watchdogs — its overdue sweep processes acknowledged rows through the shared plan body, so the ack-loop modes inherit that liveness guarantee. There is no separate ack auditor: an abandoned incident is a human-response failure (surfaced by proxima_escalation_ack_abandoned_total), while a wedged timer is an engine failure (surfaced by the dead-man's-switch heartbeat) — the two signals stay distinct.

Summary: signal matrix

Gap	When it fires	Metric(s)	Log level	Action
G1 Unrouted alert	Alert fires with no matching route	proxima_escalation_unrouted_total{severity}	WARN	Author the missing escalation route
G2 No on-call	NOTIFY_SCHEDULE step resolves to nobody	proxima_escalation_no_oncall_total{reason}	ERROR	Fix the schedule or add an override
G3 Schedule gap	Auditor finds uncovered window ahead of time	proxima_oncall_schedule_gap{schedule,team}	WARN	Add coverage (rotation or override)
G3 Empty rotation	Auditor finds a rotation with no users	proxima_oncall_empty_rotation{schedule,team}	WARN	Add users to the rotation
G5 Orphaned dispatch claim	Claim stuck 'sending' after crash between Send and Mark	proxima_dispatch_orphans_total{channel,path}	ERROR	Investigate process-crash source; rising rate = delivery gap
Ack abandoned	Acked incident goes silent past all re-escalations, still unresolved	proxima_escalation_ack_abandoned_total	ERROR	Re-claim (✅ Still on it) or resolve — a live abandoned incident

Related

• On-call & Escalation Management — schedules, rotations, policies, routes, covering teams, cutover.
• Paging Control — how routes decide whether and whom to page.
• Escalation Dead-Man's Switch — guard against a wedged escalation engine.
• Notification Chains — per-user paging channel order and fallback.