On-call & Escalation Management

Proxima Console has a native on-call and escalation engine — schedules, rotations, overrides, escalation policies, and routes — that decides who gets paged and in what order when an alert is page-worthy. This guide covers operating it from the admin console.

On-call is team-owned. Schedules and escalation policies belong to a team, not to a customer. Customers are alert sources that route by severity into a covering team's policies. This is the model Grafana OnCall uses (an integration receives alerts, an escalation chain pages a team's schedule), and Console maps onto it directly.

All of the pages and APIs below are super-admin only, enforced on every request. There is no per-client permission for them: schedules, teams, and the cutover span every tenant.

Where paging authority lives today

For teams still on Grafana OnCall, Console runs its scheduling engine in shadow and compares its answer to OnCall's on every poll. Console only becomes the paging authority for a team after you flip that team through the cutover panel. Until then, editing schedules here is safe rehearsal — nothing pages off them.

Concepts

The owner: a team

Every schedule and every escalation policy has an owner, expressed as owner_type + owner_id. In v1 the only owner axis is team (owner_id is a team id) — the API rejects any other owner_type with a 400 ("only team-owned schedules/policies are supported in v1"). The client and org owner axes are reserved in the schema for a future release; they are not authorable today.

Owning a schedule or policy by team means:

• You create and list them against a team: ?owner_type=team&owner_id=<teamId>.
• A rotation's users must be members of the schedule's owner team (team_members) — not merely users with access to some customer. Super-admins are assignable to any team.
• When an alert arms an escalation, the page-owner is the team that owns the armed policy. That team is frozen into the escalation snapshot at fire time, so who gets paged is a property of the team, independent of which customer's alert triggered it.

How a customer's alert reaches a team (the Grafana OnCall mapping)

Grafana OnCall	Proxima Console
Integration (an inbound alert source)	A customer (client) — the alert's client_id
Route (matches an alert, picks a chain)	An escalation route (matches by severity/labels, picks a policy)
Escalation chain	An escalation policy (the team's ordered step chain)
Schedule referenced by a chain step	A schedule referenced by a NOTIFY_SCHEDULE step
Team that owns the chain + schedule	The owner team (owner_type=team)

An alert enters as a customer's alert (it keeps its client_id), matches one of that customer's routes by severity/labels, and that route points at a team-owned policy. The policy's steps page the team — including resolving the team's schedule for NOTIFY_SCHEDULE. So the customer is the source; the team is the owner that actually gets paged.

Schedules, rotations, layers, overrides

A schedule is a named, timezone-aware on-call calendar owned by a team. Resolving "who is on call at time T" walks its parts in priority order:

• Rotations are the recurring backbone. A rotation has an ordered list of team-member users and a recurrence: a frequency (hourly / daily / weekly), an interval_count (every N periods), an optional by_day weekday mask (e.g. Mon–Fri), a week_start anchor, a handoff time-of-day, and a shift duration. The engine steps through the ordered users one hand-off per cycle.
• Layers stack rotations. A rotation's layer (0 = base) lets a higher layer take precedence when it has coverage and fall through to a lower layer when it doesn't — e.g. a Mon–Fri business-hours layer on top of a 24×7 catch-all layer. On a masked-off day (a weekday not in by_day) the higher layer yields no shift and the lower layer covers.
• Overrides are one-off manual coverage windows (user, start, end). An override always wins the resolver for its window, on top of every rotation and layer. Use them for swaps, sick days, and holiday coverage.

Everything is evaluated in the schedule's timezone. A shift that spans a DST change keeps its local start and end times.

Escalation policies, roles, and the 5 step types

An escalation policy is a named, ordered chain of steps, owned by a team, that runs when an armed alert isn't acknowledged. The supported step types are:

Step	What it does
WAIT	Pause for N minutes before the next step. The only delay step.
NOTIFY_USERS	Fan out — page every user in the list.
NOTIFY_USERS_QUEUE	Round-robin — page the next single user in the list.
NOTIFY_SCHEDULE	Resolve who is on call now for a schedule and page them. The schedule must belong to the same owner team as the policy.
REPEAT	Loop back to the start of the policy, capped by the policy's repeat count.

A policy also carries a repeat count (0–5, how many times a REPEAT may loop) and an ack-timeout (5–240 minutes, or empty for a permanent halt after the chain finishes).

A policy may additionally carry a role — critical or default — which marks it as a covering-team materialization target (see below). A team has at most one policy per role. Ordinary policies leave the role unset.

Roadmap step types

The engine reserves additional step types (notify-team, trigger-webhook, notify-if-time, notify-if-num-alerts). They appear greyed out in the step builder and are not selectable yet — they are on the roadmap, not shipped.

Routes (first match wins) and the covering team

A route maps a matching alert to a target policy. Routes belong to the customer whose alerts they match (the route keeps its client_id — it is the alert source), but they point at a team-owned policy. Routes are evaluated top to bottom in position order, and the first match wins — so order matters and the table is reorderable. A route matches on any combination of environment, severity (P1–P5), host, and label selectors; a route marked default matches everything and belongs at the bottom as the catch-all.

Each route carries a source:

• manual — a hand-authored route you created on the escalation page. Never touched by materialization.
• covering_team — a route auto-generated from a customer's covering-team assignment.

Assigning a covering team

Rather than hand-author routes for every customer, you can assign a customer a covering team — the team that fields that customer's pages. Assigning it materializes exactly two covering_team routes on the customer:

• P1 → the team's critical-role policy (a severity-matched route, not the catch-all).
• catch-all (any other severity) → the team's default-role policy (the is_default fallback, placed last).

Assign via PUT /api/v1/clients/{id}/covering-team with {"team_id": "<teamId>"}; clear via DELETE /api/v1/clients/{id}/covering-team. If the covering team lacks a critical or default role policy, the assignment fails with 409 naming the missing role and writes nothing. Re-assigning replaces the pair; your hand-authored manual routes are always left intact.

covering_team_id is authoring-only

Assigning a covering team sets clients.covering_team_id and (re)materializes the two routes. That column is authoring convenience only — the paging engine never reads it at fire time. The page-owner is always the team that owns the armed policy (frozen into the snapshot), resolved through the materialized routes like any other route. Clearing a covering team removes only its covering_team routes.

The editors

All editors live under the admin section and are gated on super-admin.

On-call page (/admin/oncall)

The read-first view. Pick a team to see, per schedule:

• Now / next — who is on call now (and until when) and who is next (and from when).
• Calendar preview — a 1–8 week grid of resolved shifts. Rotation shifts and override shifts are styled distinctly and the layer is indicated, so you can see exactly what the engine will resolve before anyone is paged.
• A schedule list linking into each schedule editor.

Schedule editor

1. Set the meta — name, timezone, enabled. The schedule is owned by the team selected on the on-call page.
2. Add rotation layers. Per rotation: frequency, shift duration, the by_day day-picker (shown for daily/weekly), week_start, handoff time, priority, and layer.
3. Give each rotation its ordered user list — add from the team's members, remove, and move up/down. Order is the hand-off order.
4. Add overrides — pick a user and a start/end window.

Every successful edit re-fetches the calendar preview, so you author, then see the resulting coverage immediately.

Escalation page (/admin/escalation)

• Pick a team to author its policies.
• The policies list, each opening a step builder: add steps of the 5 supported types with type-appropriate inputs (WAIT → minutes; NOTIFY_USERS → user multi-picker; QUEUE → ordered user list; NOTIFY_SCHEDULE → a schedule of the same owner team; REPEAT → loops to start), plus the policy's repeat count, ack-timeout, and optional role (None / Critical / Default).
• The routes table (per customer) in first-match position order — each row shows its match criteria, target policy, source (manual / covering_team), enabled state, and the default badge. Reorder with move up/down (persisted via the reorder endpoint).

Covering-team control (client detail page)

On a customer's detail page, the Covering team control assigns (or clears) that customer's covering team. Assigning shows that it materializes the two default covering_team routes, and surfaces the 409 missing-role message if the team has no critical/default policy.

Notification chains (admin user page)

Each user has two ordered channel chains — Default and Important — of telegram / voice entries, edited on the admin user detail page. The engine walks a chain in order when paging that user. An empty chain falls back to telegram → voice.

Two semantics you must know

These two rules are stated in the API descriptions and are load-bearing for how edits behave:

1. In-flight escalations are snapshot-frozen. When an alert arms an escalation, the engine snapshots the matched route, the policy, and the page-owner team at that instant. Editing (or deleting) that policy or route afterwards — or reassigning the customer's covering team — does not change the escalation already in flight; it runs to completion against the frozen snapshot. Edits apply to the next alert that arms.
2. Schedule edits take effect at the next resolution. Changing a schedule, rotation, ordered users, or overrides does not retroactively rewrite history or interrupt a resolution already computed — it is picked up the next time the engine resolves who is on call.

Cutover: Grafana OnCall → Console

The cutover panel (/admin/oncall-cutover) is where you hand paging authority from Grafana OnCall to Console — and where you take it back. Cutover is per team: each row is one team that owns an on-call schedule.

The procedure

1. Shadow. With the team still paged by OnCall, Console's shadow poller resolves who it thinks is on call on every tick and compares against OnCall's answer, recording drift and clean-poll counts per schedule (see the proxima_escalation_drift_total metric).
2. Watch the gate. The panel shows, per team: live?, clean-poll count, drift count, clean-window days, and poller-heartbeat freshness. A team becomes flip-eligible only when every criterion below passes.
3. Flip. When the team is eligible, the Flip button enables — flipping makes Console the paging authority for that team: POST /api/v1/admin/oncall-cutover/{teamID}.
4. Revert. Revert-to-shadow is always available; it returns paging authority to OnCall for that team immediately: DELETE /api/v1/admin/oncall-cutover/{teamID}.

Gate criteria

A team is flip-eligible only when all of these hold — the panel lists the exact failing ones inline when it isn't:

• A fresh poller heartbeat (the shadow poller is actually running).
• Every active schedule of the team has been observed by the shadow poller (never-observed schedules fail).
• No schedule is currently drifting (any live drift resets that schedule's clean window).
• Each schedule's clean window ≥ 14 days AND ≥ one full rotation length.
• Each schedule has enough observed clean polls.

Force-flip and the audit trail

You can force a flip of a still-failing team — but it is deliberately friction-heavy: the panel puts force-flip behind a typed-confirm dialog, and the server writes a WARN audit entry naming you and the team when a forced flip goes through. The eligibility check and the panel's human-readable reasons share a single evaluation path, so the gate and the explanation can never disagree — if the panel says "eligible", the flip will be allowed for the same reason.

Permissions

Every on-call, escalation, cutover, and covering-team route requires super-admin. The check is the first line of every handler; a non-super-admin (even a client admin) gets 403, and an unauthenticated request gets 401.