Skip to main content

Notification Chains

When an escalation step decides who to page (an on-call user, or a named user), a per-user notification chain decides how that one person is reached — which channels, in which order, with what delays between them. This guide covers the chain model, how to author a chain, the failure behavior, and the built-in fallback.

Notification chains are executed by the NotificationChainWorker, a Postgres-timer worker that runs alongside the escalation engine. It is enabled together with the rest of the paging stack (it requires the ops Telegram bot token). Voice legs additionally require Twilio to be configured; without Twilio a voice step is skipped (see Voice not configured).

The timed chain model​

A chain is an ordered list of steps frozen at the moment the escalation fires for that user. There are two kinds of step:

  • NOTIFY — page one channel: telegram or voice.
  • WAIT — pause for wait_seconds before the next step.

When an escalation NOTIFY fires for a user, the escalation engine freezes that user's chain into a run keyed on (alert group, firing episode, user, important) and hands off; the chain worker then walks the frozen steps on its own timeline. Because the chain is frozen at fire time, editing a user's chain never disturbs an in-flight page — the change takes effect on the next episode.

A typical chain:

telegram          # page Telegram immediately
WAIT 300s         # give them 5 minutes to acknowledge
voice             # if still firing, call their phone

The WAIT is honored: the voice leg does not fire until the wait has elapsed. If the user acknowledges (or the alert resolves) during the wait, the fence halts the chain and no further channels are paged.

Default vs important​

Every user can have two chains:

  • the default chain — used for a normal page.
  • the important chain — used when the escalation step is marked important. An important page typically rings the phone sooner (shorter or no WAIT), because a critical incident should not wait out an ack window.

The escalation step's important flag selects which of the user's two chains is frozen.

Authoring a chain (API-first)​

Chains are authored through the API. (A web chain-editor is a planned follow-on; until then, author via the API.)

Set a user's default and important chains with:

PUT /api/v1/users/{id}/notification-policy

The body carries the two ordered chains as step lists. Each step is either a NOTIFY step (notify_by: "telegram" | "voice") or a WAIT step (wait_seconds: <n>):

{
  "default": [
    { "notify_by": "telegram" },
    { "wait_seconds": 300 },
    { "notify_by": "voice" }
  ],
  "important": [
    { "notify_by": "telegram" },
    { "notify_by": "voice" }
  ]
}
  • A NOTIFY step sets notify_by and omits wait_seconds.
  • A WAIT step sets wait_seconds and omits notify_by.
  • Read the exact request/response shape from the generated API types / Swagger; the fields above are the load-bearing ones.

This endpoint is super-admin gated, consistent with the rest of the on-call/escalation authoring surface.

Channel contacts​

A NOTIFY step only pages if the user has a usable contact for that channel:

  • telegram — resolved from the user's telegram_id trait.
  • voice — resolved from the client roster's phone contact for that user (tenant-scoped).

A step whose channel has no usable contact is treated as a permanent skip: the chain advances past it to the next step rather than stalling (see below).

Failure behavior: permanent vs transient​

The whole reason delivery runs through the chain worker (rather than paging every channel inline) is robustness of the handoff between channels. Each NOTIFY step's send outcome is classified:

  • Permanent failure — a dead/blocked contact, a 403, an unregistered channel, or an empty render. The chain advances to the next channel. A user whose Telegram is blocked still gets the voice call.
  • Transient failure — a temporary send error or a DB blip. The same step is re-armed with backoff, up to PROXIMA_NOTIFICATION_TRANSIENT_MAX_ATTEMPTS (default 3) attempts, after which it is treated as terminal and the chain advances.
  • Sent — success; the chain advances to the next step.
Why this matters

Before chains, on-call delivery paged a user's channels inline and early-returned on the first channel error — so a dead Telegram contact silently dropped the page and the voice call never happened. The chain executor fixes this: a permanent failure advances instead of aborting.

The fence overrides everything: on every step the worker rechecks that the alert group is still firing. An acked or resolved group halts the whole episode's chains for every user — no further pages fire.

The fallback (user has no authored chain)​

If a paged user has no authored notification policy, the engine synthesizes a timed default fallback so they are never silently un-paged:

  • Default (non-important): telegram → WAIT 300s → voice — page Telegram, give a 5-minute ack window, then call.
  • Important: telegram → voice — page Telegram and call back-to-back, with no wait.

Authoring a chain for the user overrides the fallback entirely.

Configuration​

Env varDefaultMeaning
PROXIMA_NOTIFICATION_CHAIN_INTERVAL2sHow often the chain worker ticks.
PROXIMA_NOTIFICATION_TRANSIENT_MAX_ATTEMPTS3Transient re-arm attempts per channel before advancing.

The worker is gated on the ops Telegram bot token (the same gate as the escalation timer and auditor): with no Telegram channel there is nothing to page through, and the whole paging stack — including chains — is off.

Voice not configured​

The voice channel is registered only when Twilio is fully configured (PROXIMA_TWILIO_ACCOUNT_SID, PROXIMA_TWILIO_AUTH_TOKEN, PROXIMA_VOICE_PUBLIC_BASE_URL). When Twilio is not configured, a voice NOTIFY step resolves no channel and is permanently advanced past (a config gap, not a stuck chain) — the chain still completes, just without the call.

Observability​

MetricLabelsMeaning
proxima_notification_chain_steps_totalchannelSuccessful chain sends per channel.
proxima_notification_chain_retries_totalchannelTransient re-arms per channel.
proxima_notification_delivery_failures_totalchannel, kindDelivery failures, split permanent vs transient.

Live validation procedure​

The chain worker only runs on a deployed backend (it needs Postgres, the ops bot, and — for voice — Twilio). Validate on a dev/staging deployment after the branch is deployed:

  1. Author a timed chain via the API for a test user (super-admin):

    PUT /api/v1/users/{testUserId}/notification-policy
    {
      "default":   [ { "notify_by": "telegram" }, { "wait_seconds": 60 }, { "notify_by": "voice" } ],
      "important": [ { "notify_by": "telegram" }, { "notify_by": "voice" } ]
    }

    (Use a short wait_seconds like 60 for the test so you don't wait 5 minutes.) Ensure the user has a telegram_id trait and a roster phone contact.

  2. Fire a test escalation that pages this user (e.g. a P1 alert routed to a policy whose step notifies the user / their schedule). Confirm the sequence with real timing:

    • the Telegram page arrives first;
    • after the WAIT elapses, the voice call is placed.

  3. Blocked-Telegram case — repeat with a user whose Telegram contact is dead/blocked (or delete the telegram_id trait). Confirm the voice call still lands — the permanent Telegram failure must not drop the page.

  4. Check the metrics in VictoriaMetrics:

    • proxima_notification_chain_steps_total{channel="telegram"} and {channel="voice"} both increment;
    • proxima_notification_delivery_failures_total{kind="permanent"} increments for the blocked-Telegram case, while voice still succeeds.

  5. Acknowledge the alert mid-chain and confirm the chain halts — no further pages.