Rate Limiting

Proxima Console applies several independent rate limits, each serving a different goal — brute-force defense, cost control, and abuse prevention. This page is the one place that documents them all: what each protects, its default, the environment variable that tunes it, and how it's enforced.

Every limit here is configurable and ships with a safe default, with exactly two stated exceptions — the voice-fetcher and push-heartbeat limits below, which are compile-time constants and each says why. None require tuning for a normal deployment.

At a glance

Limit	Protects	Default	Env var	Keyed on	Backing
Auth	Login / password-reset / MFA / SSO brute-force	5 / min	PROXIMA_AUTH_RATE_LIMIT	client IP	Valkey (cross-instance)
Credential	Agent enroll / credential-renew abuse	10 / min	PROXIMA_CREDENTIAL_RATE_LIMIT	client IP	Valkey (cross-instance)
API	General authenticated API flooding	1000 / min	PROXIMA_API_RATE_LIMIT	client IP	in-process (per instance)
Webhook	Inbound webhook (Alertmanager, CI) flooding	60 / min	PROXIMA_WEBHOOK_RATE_LIMIT	client IP	in-process (per instance)
Voice fetcher	Flooding the two sound-prefetch endpoints	300 / min	(none — see below)	client IP	in-process (per instance)
Push heartbeat	Flooding the two public push-monitor heartbeat routes	120 / min	(none — see below)	client IP	in-process (per instance)
AI chat	Per-user chat-request abuse / cost	30 / min	PROXIMA_CHAT_RATE_LIMIT	user ID	Valkey
LLM tokens	Per-client AI token spend (budget + burst)	unlimited	clients.features.llm	client ID	Valkey
Runbook exec	Runaway runbook executions on a host	5 / 10 min	PROXIMA_RUNBOOK_RATE_LIMIT_PER_HOST · …_WINDOW_SEC	host ID	database

Exceeding an HTTP limit returns 429 Too Many Requests.

HTTP request limits (auth, credential, API, webhook)

These protect the HTTP surface. The auth and credential limits are the brute-force / credential-stuffing defense; the API and webhook limits are flood protection. Each is a per-minute cap keyed on the caller's IP.

The voice-fetcher limit is a constant, deliberately

GET /api/v1/voice/pending-sounds and POST /api/v1/voice/pending-sounds/{name}/ready are called by the sound fetcher on the Asterisk box, which polls at 1 Hz — exactly 60 requests a minute, which is the entire webhook budget. Under the webhook limit the first confirmation of every minute would be the 61st request and would be refused, and a refused confirmation is not a slow-down: it makes the backend decline to play audio that is already on the box, so that page falls back to the content-free prompt. The symptom is one 429 a minute, which nobody notices until a page is silent.

So these routes carry their own budget: 300 / min, five times the poll rate.

It is voicePrefetchRateLimit in backend/internal/api/voice_prefetch_handler.go, not an environment variable, and that divergence from every other row is on purpose. The number is not a policy choice an operator should tune — it is derived from the fetcher's own poll interval, and the two must move together. An env var would let one side be changed without the other, which reintroduces exactly the failure above while looking like a deployment knob. If the poll interval ever changes, change both, in the same commit.

The routes are authenticated by PROXIMA_VOICE_FETCHER_TOKEN and by nothing else; a user session is rejected on them. With that variable unset the routes are not registered at all, so they answer 404 rather than 401.

The push-heartbeat limit is a constant too

GET /api/v1/monitors/heartbeat/{token} and GET /api/v1/monitors/heartbeat/{token}/fail are the public, session-less routes a push monitor's own script calls at the end of a cron run. They carry their own budget — 120 / min — for the same class of reason the voice-fetcher routes do: they are machine-called at a cadence the product itself sets, and the shared webhook budget is not sized for them.

pushHeartbeatRateLimit lives in backend/internal/api/push_heartbeat_handler.go and is not an environment variable.

The budget is per source IP, not per monitor

Like every HTTP limit on this page, it is keyed on the trusted client IP — so all of a customer's push monitors calling in from one cron host or one NAT egress share the single 120/min bucket. A monitor's own interval_seconds floor (20 s, so at most 3 calls a minute) bounds only one caller's share: roughly 40 monitors behind one IP reach the ceiling at an entirely ordinary cadence, with nothing misbehaving.

A 429 here is worse than a dropped request. A refused heartbeat is indistinguishable from silence, and silence past a monitor's grace window is exactly what the push auditor turns into a down — so the limit can in principle fabricate an outage for a dense enough tenant. That is latent rather than live today only because paging_enabled ships false on every monitor: a fabricated down currently shows on the monitor's own page and wakes nobody. It becomes real the first time paging is enabled for a client with many push monitors behind one IP.

If you hit that, the fix is to make this an env-configurable limit like every other row in the table above — not to raise the constant, which moves the same cliff.

IP keying is spoof-resistant

The IP a limit keys on is the trusted client IP, not a header a caller can forge. It is derived by skipping the trusted reverse-proxy hops from the right-most end of X-Forwarded-For (the entries your own proxies appended) — the spoofable True-Client-IP, X-Real-IP, and left-most X-Forwarded-For values are ignored. The number of trusted hops in front of the backend is set by PROXIMA_TRUSTED_PROXY_COUNT (default 1).

Deployment

PROXIMA_TRUSTED_PROXY_COUNT must match the real number of reverse-proxy hops in front of the backend (e.g. nginx = 1; a CDN in front of nginx = 2). If it's too low, a client can prepend a fake hop and rotate the keyed IP; if it's too high, every caller can collapse to one bucket. Your proxies must reliably append the real client IP to X-Forwarded-For.

Cross-instance vs per-instance

• Auth and credential limits are backed by Valkey, so the cap holds across all backend instances — 5 auth requests/min total, no matter how many replicas are running.
• API and webhook limits are in-process (per replica). With N backend instances the effective ceiling is roughly N × the configured value. These are flood-protection ceilings set high (1000/min, 60/min), so per-instance counting is acceptable; size them with your replica count in mind.

The search_fleet / fleet-create endpoint reuses the credential limit (10/min per IP).

AI chat rate limit

Interactive AI chat is capped per user at PROXIMA_CHAT_RATE_LIMIT (default 30) requests per minute (Valkey-backed, keyed chat:rate:<userID>). This bounds a single user's chat-request rate independently of the per-client token budget below. It is fail-open: if Valkey is unavailable (or the limit is set to 0), the check is skipped rather than blocking chat.

Per-client LLM token limits (budget + burst)

Distinct from the request-rate limits above, each client (project) can be given a monthly token budget and a per-minute token burst cap that apply across every AI surface — NL→PQL translate, AI chat, and L1 triage — so a client's total AI spend is bounded, not just its request rate. These are super-admin-only settings on the client page (clients.features.llm), Valkey-backed, and fail-open (a Valkey/config error allows the call rather than blocking AI).

See Per-Client LLM Budgets for the full details: configuration, per-surface degradation, usage reporting, and metrics.

Runbook execution limit

Runbook executions are capped per host to prevent a runaway loop from hammering a single machine: PROXIMA_RUNBOOK_RATE_LIMIT_PER_HOST (default 5) executions per PROXIMA_RUNBOOK_RATE_LIMIT_WINDOW_SEC (default 600 s = 10 minutes). This limit is tracked in the database, not Valkey.

Failure behavior summary

• Security limits (auth, credential) are the brute-force defense — keep them enabled and correctly-proxied (see the warning above).
• Cost / UX limits (AI chat, LLM token budget/burst) are fail-open: a Valkey or config error skips the check rather than breaking AI features — the durable signal is the metrics, not a hard block.
• HTTP limits reject with 429; the AI chat limit returns a 429 with a budget_exceeded/rate_limited code the UI surfaces as a toast.

Observability

The per-client LLM token limits export proxima_llm_budget_denied_total{surface,limit} and proxima_llm_tokens_consumed_total{surface} (see Per-Client LLM Budgets). HTTP rejections show up as 429 responses in the standard proxima_http_* request metrics.