Alerting & Correlation

Proxima Console ingests alerts from Alertmanager, enriches them with infrastructure context (metrics, changes, related alerts, compliance), and presents a correlated view for faster incident response.

Overview

The alerting system sits alongside your existing on-call tooling. Rather than replacing Alertmanager or Grafana OnCall, it acts as a parallel receiver that adds the infrastructure context operators need when they get paged at 3am:

• Metrics snapshot around the alert firing time (baseline vs. current values)
• Recent changes from both agent file monitoring and external webhooks (GitLab, GitHub, ArgoCD)
• Related alerts on the same host or environment
• Compliance state of the affected host
• Host context (hostname, OS, architecture)

VMAlert --> Alertmanager --+--> Grafana OnCall  (keeps handling on-call)
                           |
                           +--> Proxima Console  (adds infrastructure context)

Architecture

The pipeline is fully asynchronous. The webhook endpoint validates the token, parses the payload, publishes to NATS, and returns 202 Accepted immediately. Processing happens in two worker stages:

1. AlertWorker (fast path): resolves labels to Proxima entities, deduplicates alerts by fingerprint, finds or creates alert groups, handles resolve/reopen lifecycle
2. CorrelationWorker (enrichment): queries VictoriaMetrics for metrics, PostgreSQL for changes/related alerts/compliance, and writes the correlation context back to the alert group

Alert Source Setup

An alert source is a per-client integration endpoint that receives webhooks from Alertmanager.

Creating an Alert Source

curl -X POST https://api-console.prxm.uz/api/v1/clients/{client_id}/alert-sources \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Production Alertmanager",
    "source_type": "alertmanager",
    "enabled": true
  }'

Response (201):

{
  "data": {
    "id": "a1b2c3d4-...",
    "client_id": "...",
    "source_type": "alertmanager",
    "name": "Production Alertmanager",
    "enabled": true,
    "token": "pxm_as_7f3a8b2c1d...",
    "created_at": "2026-03-15T10:00:00Z",
    "updated_at": "2026-03-15T10:00:00Z"
  }
}

caution

The token is returned only once at creation time. Store it securely. If lost, delete and recreate the alert source.

Alertmanager Configuration

Add a webhook_configs entry to your Alertmanager configuration pointing at the Proxima webhook receiver:

# alertmanager.yml
receivers:
  - name: "proxima"
    webhook_configs:
      - url: "https://api-console.prxm.uz/api/v1/alerts/webhook/alertmanager/pxm_as_7f3a8b2c1d..."
        send_resolved: true

route:
  receiver: "default"
  routes:
    - receiver: "proxima"
      continue: true    # Important: continue to other receivers (OnCall, etc.)
      matchers: []      # Match all alerts (or restrict with matchers)

Key points:

• Set send_resolved: true so Proxima can track alert resolution
• Use continue: true if you want alerts to also reach other receivers (e.g., Grafana OnCall)
• The token in the URL authenticates the request (no additional headers needed)

Label Mapping

Label mappings define how Alertmanager labels resolve to Proxima infrastructure entities. Each client configures their own mappings based on their labeling conventions.

How It Works

1. When an alert arrives, the AlertWorker loads the client's label mappings (cached, refreshed every 60s)
2. For each mapping, it extracts the source_label value from the alert's labels
3. Applies the configured transform (exact match, strip port, or regex)
4. Looks up the transformed value in the corresponding Proxima entity table

Creating Label Mappings

curl -X POST https://api-console.prxm.uz/api/v1/clients/{client_id}/alert-label-mappings \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "source_label": "instance",
    "target_field": "host_address",
    "transform": "strip_port",
    "priority": 10
  }'

Target Fields

Target Field	Matches Against	Example
client_slug	clients.slug	cluster=acme-prod maps to client acme-prod
environment_slug	environments.slug	namespace=staging maps to environment staging
host_address	hosts.ip_address or hosts.hostname	instance=10.0.1.5:9100 maps to host 10.0.1.5
team_name	teams.name	team=platform maps to team platform

Transforms

Transform	Behavior	Example
exact	Use label value as-is	cluster=production matches production
strip_port	Remove :port suffix	instance=10.0.1.5:9100 becomes 10.0.1.5
regex	Regex extraction	Extract hostname from FQDN patterns

Common Mapping Configurations

Kubernetes cluster setup:

[
  {"source_label": "cluster", "target_field": "client_slug", "transform": "exact", "priority": 10},
  {"source_label": "namespace", "target_field": "environment_slug", "transform": "exact", "priority": 20},
  {"source_label": "instance", "target_field": "host_address", "transform": "strip_port", "priority": 30},
  {"source_label": "team", "target_field": "team_name", "transform": "exact", "priority": 40}
]

Bare-metal/VM setup:

[
  {"source_label": "datacenter", "target_field": "environment_slug", "transform": "exact", "priority": 10},
  {"source_label": "instance", "target_field": "host_address", "transform": "strip_port", "priority": 20}
]

Unresolved Alerts

If label resolution fails (no matching host, environment, or client), the alert is stored with host_id = NULL. It still appears in the global alerts list and client-scoped views.

Correlation

When the AlertWorker finishes processing an alert, it publishes a message to proxima.alerts.correlate. The CorrelationWorker gathers infrastructure context and writes it to the correlation JSONB column on the alert group.

What Gets Correlated

Context	Source	Window
Metrics snapshot	VictoriaMetrics	±30 minutes around first_fired_at
Recent changes	PostgreSQL (change_events)	Last 1 hour before alert
Related alerts	PostgreSQL (alert_groups)	Firing alerts on same host/environment (limit 10)
Compliance score	PostgreSQL (control_evaluations)	Average across all frameworks at correlation time
Host snapshot	PostgreSQL (hosts)	Hostname, OS, architecture

How Correlation Works

When the AlertWorker finishes processing an alert group, it publishes the alert_group_id to proxima.alerts.correlate. The CorrelationWorker picks it up and:

1. Loads the alert group from PostgreSQL
2. If host_id is set (label resolution succeeded):
   • Resolves the VM tenant via tenantCache (host → client → vm_account_id)
   • Queries VictoriaMetrics for cpu_usage_ratio, memory_used_ratio, and disk_used_ratio in a ±30-minute window around first_fired_at with 60-second step
   • Computes trends: separates data points into baseline (before alert) and at-alert (closest point to fire time). Trend = spike if at-alert > 1.5× baseline, drop if < 0.5× baseline, stable otherwise
   • Queries change events from the last hour (limit 10) — includes both agent-detected file changes and external webhook events (GitLab, GitHub, ArgoCD deployments)
   • Queries compliance score — averages across all frameworks for the host
   • Captures host snapshot — hostname, OS, architecture
3. Queries related alerts — other firing alert groups on the same host or environment (limit 10, excludes self), with relative time offsets
4. Stores the correlation JSON in alert_groups.correlation and logs a "correlated" action to the audit trail

If any individual enrichment step fails (e.g., VM query timeout, change store error), it logs a warning and continues with the other steps. Partial correlation data is better than none.

Correlation JSON Structure

The correlation column on alert_groups contains this JSON structure:

{
  "metrics_snapshot": {
    "window": {
      "from": "2026-03-15T10:16:00Z",
      "to": "2026-03-15T11:16:00Z"
    },
    "series": {
      "cpu_usage_ratio": {"baseline": 0.341, "at_alert": 0.872, "trend": "spike"},
      "memory_used_ratio": {"baseline": 0.623, "at_alert": 0.910, "trend": "spike"},
      "disk_used_ratio": {"baseline": 0.779, "at_alert": 0.785, "trend": "stable"}
    }
  },
  "recent_changes": [
    {
      "id": "d4e5f6a7-...",
      "source": "gitlab",
      "event_type": "deployment",
      "summary": "Merged MR !482: reduce memory limits",
      "file_path": "/etc/kubernetes/manifests/api-server.yaml",
      "severity": "high",
      "created_at": "2026-03-15T10:28:00Z"
    }
  ],
  "related_alerts": [
    {
      "id": "a1b2c3d4-...",
      "grouping_key": "DiskSpaceFull",
      "status": "firing",
      "severity": "P1",
      "first_fired_at": "2026-03-15T10:34:00Z",
      "relative_offset": "-12m"
    }
  ],
  "compliance_score": 0.72,
  "host_snapshot": {
    "id": "e3f4a5b6-...",
    "hostname": "prod-k8s-01",
    "os": "Ubuntu 24.04",
    "arch": "x86_64"
  }
}

Field reference:

Field	Type	Description
metrics_snapshot.window	object	Time range queried (from/to in RFC 3339)
metrics_snapshot.series	map	Metric name → trend object. Currently queries cpu_usage_ratio, memory_used_ratio, disk_used_ratio. Values are 0.0–1.0 ratios.
metrics_snapshot.series.*.baseline	float	Average value of data points before first_fired_at
metrics_snapshot.series.*.at_alert	float	Value of the data point closest to first_fired_at
metrics_snapshot.series.*.trend	string	"spike" (above 1.5× baseline), "drop" (below 0.5× baseline), or "stable"
recent_changes	array	Up to 10 most recent change events in the hour before alert
recent_changes[].source	string	"agent", "gitlab", "github", "argocd"
recent_changes[].file_path	string	File path (if applicable, omitted otherwise)
related_alerts	array	Up to 10 other firing alert groups on the same host/environment
related_alerts[].relative_offset	string	Time offset from this alert's first_fired_at (e.g., "-12m" = fired 12 min before, "+2m" = fired 2 min after)
compliance_score	float	Average compliance score across all frameworks (0.0–1.0). null if no evaluations exist.
host_snapshot	object	Host info at correlation time. null if host_id was not resolved.

Metric values are ratios

All metric values in metrics_snapshot.series are 0.0–1.0 ratios (not percentages). The frontend multiplies by 100 for display. This matches the OpenMetrics naming convention used throughout the agent.

Null vs empty

• metrics_snapshot is null when: the host has no VM data, the tenant cache is unavailable, or all metric queries failed.
• recent_changes and related_alerts are always arrays (empty [] when none found, never null).
• compliance_score is null when no compliance evaluations exist for the host.
• host_snapshot is null when host_id was not resolved by label mapping.

Alert Lifecycle

Alert groups follow a four-state lifecycle:

Status	Description
firing	One or more alerts are actively firing
acknowledged	An operator has acknowledged the alert group
resolved	All individual alerts in the group have resolved
silenced	Muted until a specified timestamp

State Transitions

• Acknowledge: Sets acknowledged_by and acknowledged_at. Requires alerts:write permission.
• Resolve: Manually resolves the alert group. Does not reach back to Alertmanager.
• Silence: Sets silenced_until to a future timestamp. Automatically reverts to firing when the silence expires.
• Reopen: When a new firing alert arrives for a resolved group, the group automatically reopens (status returns to firing, resolved_at is cleared).

REST API Reference

Alert Source Management

Requires alertsources:read, alertsources:write, or alertsources:delete permissions.

Method	Path	Description
POST	/api/v1/clients/{id}/alert-sources	Create alert source (returns token once)
GET	/api/v1/clients/{id}/alert-sources	List alert sources
GET	/api/v1/clients/{id}/alert-sources/{id}	Get alert source
PUT	/api/v1/clients/{id}/alert-sources/{id}	Update alert source
DELETE	/api/v1/clients/{id}/alert-sources/{id}	Delete alert source

Alert Label Mappings

Requires alertsources:read, alertsources:write, or alertsources:delete permissions.

Method	Path	Description
GET	/api/v1/clients/{id}/alert-label-mappings	List label mappings
POST	/api/v1/clients/{id}/alert-label-mappings	Create label mapping
PUT	/api/v1/clients/{id}/alert-label-mappings/{id}	Update label mapping
DELETE	/api/v1/clients/{id}/alert-label-mappings/{id}	Delete label mapping

Webhook Receiver

Public endpoint, authenticated via token in URL path. No JWT or API key required.

Method	Path	Description
POST	/api/v1/alerts/webhook/alertmanager/{token}	Receive Alertmanager webhook (returns 202)

Alert Browsing

Requires alerts:read permission.

Method	Path	Description
GET	/api/v1/alerts	List alert groups (filterable)
GET	/api/v1/alerts/{id}	Alert group detail with correlation, alerts, and log
GET	/api/v1/alerts/counts	Alert counts by status (scoped to user's clients)
GET	/api/v1/hosts/{id}/alerts	Alert groups for a host
GET	/api/v1/environments/{id}/alerts	Alert groups for an environment
GET	/api/v1/clients/{id}/alerts	Alert groups for a client

Query Parameters for /alerts

Parameter	Default	Description
page	1	Page number
per_page	20	Items per page (max 100)
client_id	—	Filter by client
environment_id	—	Filter by environment
host_id	—	Filter by host
status	—	Filter by status: firing, acknowledged, resolved, silenced
severity	—	Filter by severity: P1, P2, P3, P4, P5
from	—	Start time (RFC3339)
to	—	End time (RFC3339)

Alert Actions

Requires alerts:write permission. All actions are local to Proxima only — they do not reach back to Alertmanager or OnCall.

Method	Path	Description
POST	/api/v1/alerts/{id}/acknowledge	Acknowledge alert group
POST	/api/v1/alerts/{id}/resolve	Manually resolve alert group
POST	/api/v1/alerts/{id}/silence	Silence until specified timestamp
POST	/api/v1/alerts/{id}/unsilence	Remove silence

Silence Request Body

{
  "until": "2026-03-16T08:00:00Z"
}

Deduplication

Alerts are deduplicated using Alertmanager's fingerprint field:

• A partial unique index on (alert_group_id, fingerprint) WHERE status = 'firing' enforces at most one active alert per fingerprint per group
• If a firing alert with the same fingerprint already exists, updated_at is refreshed (no duplicate created)
• Resolved alerts free the fingerprint for future firing alerts
• Alert groups are capped at 1000 alerts each; additional alerts are rejected with a log warning

Permissions

Permission	Scope	Description
alerts:read	Browse	View alert groups, alert detail, correlation context
alerts:write	Actions	Acknowledge, resolve, silence, unsilence alert groups
alertsources:read	Config	View alert sources and label mappings
alertsources:write	Config	Create and update alert sources and label mappings
alertsources:delete	Config	Delete alert sources and label mappings

All endpoints are scoped to the authenticated user's accessible clients. Super admins can see all alerts across all clients.

Frontend

Alerts Page

Navigate to Alerts in the sidebar to view all alert groups. The page features:

• Hybrid expandable table — click any row to expand inline and see correlation preview, summary, and action buttons
• P1–P5 severity badges — color-coded priority levels (P1=Critical red, P2=High orange, P3=Medium amber, P4=Low blue, P5=Info gray)
• Sortable columns — Severity, Status, Alert, Client, Host, Count, Duration
• Filters — Status (Firing/Acknowledged/Resolved/Silenced), Severity (P1–P5), Search
• 30-second polling — alerts refresh automatically
• Sidebar badge — red count of firing alerts in the navigation

Alert Detail Page

Click "View Full Context →" on any alert to open the detail page (/alerts/:id). Two-column layout:

Left column — Correlation Data:

• Metric cards with sparkline charts and SPIKE/DROP/STABLE trend indicators
• Recent changes with source badges (GitLab, Agent, ArgoCD) and time offsets
• Related alerts in Before/After card layout
• Host information bar
• Individual alerts in group table

Right sidebar — Activity Timeline:

• Unified chronological feed mixing all event types (newest first):
  • Alert status changes (fired, acknowledged, resolved, silenced)
  • Correlation events (config changes detected, related alerts)
  • Operator notes
• Notes composer at bottom for adding investigation notes (Markdown supported)
• Notes are preserved as post-mortem evidence

Admin Pages

• Alert Sources (/admin/alert-sources) — manage Alertmanager webhook integrations per client. Token-based authentication with one-time display.
• Alert Label Mappings (/admin/alert-label-mappings) — configure how Alertmanager labels map to Proxima entities (client, environment, host, team).

Host Detail — Alerts Tab

The host detail page includes an Alerts tab showing all alert groups for that specific host.

Permissions

Permission	Access
alerts:read	View alerts, alert detail, host alerts tab
alerts:write	Acknowledge, resolve, silence alerts, add notes
alertsources:read	View alert source and mapping admin pages
alertsources:write	Create/edit alert sources and mappings
alertsources:delete	Delete alert sources and mappings

Incident Response

The full incident-response layer that originally sat behind Grafana OnCall now ships in Proxima Console as the L1 incident agent:

• Escalation policies — multi-step escalation with notify-users, round-robin queues, on-call-schedule resolution, and channel notifications (escalation_store, escalation_timer_worker)
• On-call schedules and rotations — per-client schedules that resolve who is on call at notification time (oncall_store)
• Telegram notifications — escalation steps fan notifications out to bound Telegram chats (notification worker)
• Automated triage — the L1 agent triages incoming alert groups against an incident memory loop (triage worker)

Console is mid-cutover from Grafana OnCall to this native stack; per-client cutover state is tracked in the backend.

What's Deferred

Feature	Timeline
Internal alert sources (healthcheck failures, host offline)	Post-launch
Inhibition and silence rules (label matchers)	Post-launch