AI Chat Configuration

Prerequisites

AI Chat requires:

LiteLLM gateway running (Docker container)
Vault Transit with a KEK key for message encryption
At least one LLM provider API key (DeepSeek, Kimi, or Google)
Valkey/Redis for chat rate limiting and write-tool confirmation flow

If either LiteLLM or Vault is unavailable, chat is automatically disabled and the backend logs a warning. Chat can be enabled without Valkey, but rate limiting fails and write-tool confirmations are auto-rejected (you'll see a WARN: Valkey unavailable, auto-rejecting confirmation log).

Environment Variables

LiteLLM Gateway

Variable	Default	Description
`PROXIMA_LITELLM_URL`	`http://litellm:4000`	LiteLLM gateway URL (Docker Compose service name)
`PROXIMA_LITELLM_MASTER_KEY`	—	LiteLLM admin API key (generate with `openssl rand -hex 24`)

LLM Provider Keys (Platform Fallback)

These keys are used when a client hasn't configured their own (BYOK). At least one is required.

Variable	Model Tier	Description
`PROXIMA_DEEPSEEK_API_KEY`	Standard	DeepSeek API key (platform.deepseek.com)
`PROXIMA_KIMI_API_KEY`	Complex	Moonshot/Kimi API key
`PROXIMA_GOOGLE_API_KEY`	Expert	Google AI API key (for Gemini)

Reliability & Fallback

The gateway retries transient failures and can fail over to a second DeepSeek endpoint (a redundant deployment — different region/key) when a primary tier fails. See Configure a fallback endpoint.

Variable	Default	Description
`PROXIMA_LITELLM_TIMEOUT`	`15m`	Backend HTTP client timeout for a single LLM call (Go duration). Kept above the gateway's `request_timeout` so the gateway's clean, retryable timeout fires first.
`PROXIMA_DEEPSEEK_FALLBACK_API_BASE`	— (empty = inert)	Base URL of the backup DeepSeek endpoint for the `-fallback` tiers. Must be defined* (even if empty) so the gateway starts cleanly.
`PROXIMA_DEEPSEEK_FALLBACK_API_KEY`	— (empty = inert)	API key for the backup DeepSeek endpoint.

The backend also retries connection-level and transient gateway failures (502/503/504) up to 3 attempts before any tokens stream — once streaming begins it never retries, since partial output can't be replayed. Application errors (4xx, 429, 500) are surfaced immediately.

Chat Behavior

Variable	Default	Description
`PROXIMA_CHAT_RATE_LIMIT`	`30`	Max messages per minute per user
`PROXIMA_CHAT_MAX_TOOL_CALLS`	`10`	Max tool calls per message (safety limit)
`PROXIMA_CHAT_MAX_CONVERSATION_MESSAGES`	`100`	Soft limit on messages per conversation
`PROXIMA_CHAT_CONTEXT_WINDOW`	`50`	Max messages sent to LLM as context (sliding window)

Encryption

Variable	Default	Description
`PROXIMA_CHAT_VAULT_KEK_NAME`	`proxima-chat-kek`	Vault Transit key name for DEK encryption

LiteLLM Setup

1. Create the Vault Transit KEK

vault write -f transit/keys/proxima-chat-kek type=aes256-gcm96

2. Add LiteLLM to Docker Compose

LiteLLM is included in the docker-compose.yml:

The image is pinned to a SHA256 digest (not a mutable tag) because tagged versions v1.82.7/v1.82.8 were compromised by a credential stealer. Use docker-compose.yml as the source of truth and do not swap in a mutable tag.

litellm:
  # Pinned to SHA256 digest — DO NOT upgrade until supply chain incident is resolved.
  # See: https://docs.litellm.ai/blog/security-update-march-2026
  # Compromised versions: v1.82.7, v1.82.8 (credential stealer via TeamPCP)
  image: ghcr.io/berriai/litellm@sha256:067aee932b8770ed42955ee802a04abdcd369d0995b5e696bb07d6520a231b1c
  container_name: proxima-litellm
  command: ["--config", "/app/config.yaml"]
  ports:
    - "${LITELLM_BIND:-127.0.0.1}:4000:4000"
  environment:
    LITELLM_MASTER_KEY: ${PROXIMA_LITELLM_MASTER_KEY:-sk-litellm-dev-key}
    DATABASE_URL: postgresql://${PROXIMA_DB_USER:-proxima}:${PROXIMA_DB_PASSWORD:-proxima_dev}@postgres:5432/proxima_litellm
    PROXIMA_DEEPSEEK_API_KEY: ${PROXIMA_DEEPSEEK_API_KEY:-}
    PROXIMA_KIMI_API_KEY: ${PROXIMA_KIMI_API_KEY:-}
    PROXIMA_GOOGLE_API_KEY: ${PROXIMA_GOOGLE_API_KEY:-}
    # Second DeepSeek endpoint for the *-fallback tiers (empty = inert).
    PROXIMA_DEEPSEEK_FALLBACK_API_BASE: ${PROXIMA_DEEPSEEK_FALLBACK_API_BASE:-}
    PROXIMA_DEEPSEEK_FALLBACK_API_KEY: ${PROXIMA_DEEPSEEK_FALLBACK_API_KEY:-}
  volumes:
    - ./infra/litellm/config.yaml:/app/config.yaml
  depends_on:
    postgres:
      condition: service_healthy
  restart: unless-stopped

3. Create the LiteLLM Database

PGPASSWORD="$PROXIMA_DB_PASSWORD" psql -h localhost -U "$PROXIMA_DB_USER" -d postgres \
  -c "CREATE DATABASE proxima_litellm;"

4. Configure Model Routing

Edit infra/litellm/config.yaml to configure which models handle each tier:

The current deployment maps all three tiers to DeepSeek models:

model_list:
  # Standard tier — default for chat, simple-to-moderate tasks
  - model_name: "proxima/standard"
    litellm_params:
      model: "deepseek/deepseek-chat"
      api_key: "os.environ/PROXIMA_DEEPSEEK_API_KEY"

  # Complex tier — multi-step investigation, 3+ tool calls
  # TODO: restore moonshot/kimi-2.5 when budget allows
  - model_name: "proxima/complex"
    litellm_params:
      model: "deepseek/deepseek-chat"
      api_key: "os.environ/PROXIMA_DEEPSEEK_API_KEY"

  # Expert tier — root cause analysis, post-mortems, fleet-wide
  - model_name: "proxima/expert"
    litellm_params:
      model: "deepseek/deepseek-reasoner"
      api_key: "os.environ/PROXIMA_DEEPSEEK_API_KEY"

You can point any tier at a different provider/model without code changes — just update the config (and the matching PROXIMA_*_API_KEY) and restart LiteLLM.

5. Start LiteLLM

docker compose up -d litellm

Verify it's running:

curl http://localhost:4000/health

6. (Optional) Configure a fallback endpoint

infra/litellm/config.yaml defines a *-fallback deployment per tier plus router_settings.fallbacks, so a primary tier failure transparently retries on a second DeepSeek endpoint. The primary is always tried first; the fallback is reached only when the primary fails.

router_settings:
  fallbacks:
    - {"proxima/standard": ["proxima/standard-fallback"]}
    - {"proxima/complex": ["proxima/complex-fallback"]}
    - {"proxima/expert": ["proxima/expert-fallback"]}
  allowed_fails: 3      # cool a deployment down after repeated failures
  cooldown_time: 30

litellm_settings:
  num_retries: 2        # retry transient failures on the same deployment first
  request_timeout: 600  # absolute per-request ceiling (seconds)

general_settings:
  disable_spend_logs: true   # Prometheus already records spend; skip unbounded DB rows

To activate the fallback, point it at a redundant DeepSeek deployment:

export PROXIMA_DEEPSEEK_FALLBACK_API_BASE="https://<backup-endpoint>/v1"
export PROXIMA_DEEPSEEK_FALLBACK_API_KEY="<backup-key>"
docker compose up -d litellm   # restart to pick up the env

Until those are set the *-fallback deployments are inert (the env vars are defined-but-empty so the gateway still starts cleanly, and the fallback is only reached if a primary fails). After changing the config, confirm a clean start:

docker compose up -d litellm && curl -fsS http://localhost:4000/health

BYOK (Bring Your Own Key)

Planned — not yet available

Per-client LLM keys are not yet implemented. Today every chat request uses the platform-configured key, so creating an ai_provider credential currently has no effect on chat. The section below describes the intended future behavior.

The planned design lets a client supply their own LLM API key so their chat requests use it instead of the platform key (the client pays their provider directly and can choose a different model):

Go to Admin → Credentials for the client
Create a credential with type ai_provider
Set the provider (anthropic, openai, deepseek, moonshot, google)
Enter the API key

Encryption

All chat message content is encrypted at rest using envelope encryption:

Each client gets a unique Data Encryption Key (DEK) — a random 256-bit AES key
The DEK is encrypted by a Vault Transit Key Encryption Key (KEK) before storage
Message content is encrypted with AES-256-GCM using the client's DEK
Metadata (tokens, model, timestamps) is stored in plaintext for billing/analytics

Crypto-shredding: Deleting a client's DEK makes all their chat messages permanently unreadable. This enables GDPR right-to-erasure compliance.

Security

RBAC scoping: All tool calls use the authenticated user's client permissions
Write tool gating: acknowledge_alert requires alerts:write, run_runbook requires runbooks:execute
Destructive runbooks: Require explicit user confirmation with 60-second timeout
Prompt injection defense: System prompt includes safety rules; tested with promptfoo red team suite (10/10 pass)
No MCP round-trips: Tools execute directly against backend stores — no over-privileged service account key

Prerequisites​

Environment Variables​

LiteLLM Gateway​

LLM Provider Keys (Platform Fallback)​

Reliability & Fallback​

Chat Behavior​

Encryption​

LiteLLM Setup​

1. Create the Vault Transit KEK​

2. Add LiteLLM to Docker Compose​

3. Create the LiteLLM Database​

4. Configure Model Routing​

5. Start LiteLLM​

6. (Optional) Configure a fallback endpoint​

BYOK (Bring Your Own Key)​

Encryption​

Security​

Prerequisites

Environment Variables

LiteLLM Gateway

LLM Provider Keys (Platform Fallback)

Reliability & Fallback

Chat Behavior

Encryption

LiteLLM Setup

1. Create the Vault Transit KEK

2. Add LiteLLM to Docker Compose

3. Create the LiteLLM Database

4. Configure Model Routing

5. Start LiteLLM

6. (Optional) Configure a fallback endpoint

BYOK (Bring Your Own Key)

Encryption

Security