AI Chat Observability

Prometheus Metrics

Metric	Type	Labels	Description
proxima_chat_messages_total	counter	role, client_id	Total chat messages (user + assistant)
proxima_chat_tokens_total	counter	direction (input/output), model	Token consumption
proxima_chat_cost_usd_total	counter	model, client_id	Estimated AI cost from LiteLLM
proxima_chat_tool_calls_total	counter	tool, status, client_id	Tool call count
proxima_chat_tool_duration_seconds	histogram	tool	Per-tool execution latency
proxima_chat_response_duration_seconds	histogram	model, tier	Full response latency (user-perceived)
proxima_chat_model_routing_total	counter	tier, reason	Routing decisions (llm/keyword_fallback/user_override)
proxima_chat_conversations_active	gauge	client_id	Active conversations
proxima_chat_errors_total	counter	type	Error breakdown (llm_error, tool_error, rate_limit, encryption)
proxima_chat_credential_source_total	counter	source, client_id	Credential source — currently always platform (BYOK is planned, not yet implemented)

Cost tracking: The proxima_chat_cost_usd_total metric reads cost data from LiteLLM's x-litellm-model-response-cost response header — no manual price table maintenance needed.

OpenTelemetry Traces

chat.request (root span)
  ├── chat.auth_resolve
  ├── chat.context_prefetch
  ├── chat.encryption.dek_resolve     (cache hit/miss attribute)
  ├── chat.model_route.llm            (tier + reason attributes)
  ├── chat.llm_call                   (model, tokens_in, tokens_out, cost)
  ├── ChatHandler.executeTool          (repeated per tool call)
  │     attributes: tool name, duration, success/error
  ├── chat.encryption.encrypt          (message content encryption)
  ├── chat.store_message               (DB insert)
  └── chat.stream_response             (streaming duration)

Structured Logging

All chat operations log with component=api.chat or component=chat.*:

Level	Event	When
INFO	AI chat enabled	Backend startup with LiteLLM configured
INFO	model routing via LLM	Every classification with tier + message preview
INFO	DEK created, encrypted, and cached	First chat for a new client
WARN	LLM classification failed, falling back to keywords	LiteLLM unreachable
WARN	context pre-fetch failed	Store error during pre-fetch (graceful)
WARN	Valkey unavailable, auto-rejecting confirmation	Valkey down during write tool
ERROR	LLM API error	LiteLLM/provider failure with status code
ERROR	failed to encrypt/decrypt DEK	Vault Transit failure

Grafana Dashboard

The backend observability dashboard includes an "AI Chat" row with the following panels:

• Token usage over time (by model, by client)
• Cost breakdown (by client, by model tier)
• Tool call popularity (top 10 tools by call count)
• Response latency P50/P95/P99
• Credential source (currently always platform — BYOK is planned)
• Error rate by type
• Routing tier distribution
• Active conversations gauge

LLM Evaluation

Promptfoo evaluation suite in tests/llm-eval/:

# Model routing accuracy (23 test cases)
DEEPSEEK_API_KEY=sk-... npx promptfoo eval -c tests/llm-eval/promptfooconfig.yaml

# Prompt injection defense (10 adversarial cases)
DEEPSEEK_API_KEY=sk-... npx promptfoo eval -c tests/llm-eval/promptfoo-redteam.yaml

# View HTML dashboard
npx promptfoo view