Service Desk Observability

The service desk feature exposes 5 Prometheus/OpenTelemetry metrics, structured slog logging on every error path, and integrates with the existing request tracing pipeline. Key handlers are instrumented with OpenTelemetry trace spans for end-to-end visibility.

Metrics

All metrics are registered under the proxima_ namespace and can be scraped from the /metrics endpoint.

Metric	Type	Labels	Description
proxima_service_desk_queries_total	Counter	source, endpoint	Total ClickHouse/JSM queries executed
proxima_service_desk_query_duration_seconds	Histogram	source, endpoint	Query latency distribution
proxima_service_desk_writes_total	Counter	action	Successful JSM write operations
proxima_service_desk_write_errors_total	Counter	action	Failed JSM write operations
proxima_service_desk_pending_writes	UpDownCounter	—	Current count of active pending writes

Label Values

source: clickhouse or jsm

endpoint (ClickHouse): tickets_count, tickets, ticket_detail, ticket_history, ticket_sla, ticket_delivery, kanban, metrics_sla, metrics_delivery, summary_counts, summary_sla, export_csv

endpoint (JSM): comments, approvals, transitions, attachments, request_type_fields

action: create, comment, transition, approve, attach

Example PromQL Queries

# Service desk query rate by source
rate(proxima_service_desk_queries_total[5m])

# Average ClickHouse query latency
rate(proxima_service_desk_query_duration_seconds_sum{source="clickhouse"}[5m])
  / rate(proxima_service_desk_query_duration_seconds_count{source="clickhouse"}[5m])

# JSM write error rate
rate(proxima_service_desk_write_errors_total[5m])

# Current pending writes gauge
proxima_service_desk_pending_writes

# P99 query latency by endpoint
histogram_quantile(0.99,
  rate(proxima_service_desk_query_duration_seconds_bucket[5m])
)

Logging

All service desk handlers use slog.ErrorContext(r.Context(), ...) with consistent attributes:

{
  "level": "ERROR",
  "msg": "clickhouse data query failed",
  "component": "api",
  "error": "clickhouse: query: status 500: ...",
  "trace_id": "abc123",
  "span_id": "def456"
}

Write operations include additional context:

{
  "level": "INFO",
  "msg": "service desk ticket created",
  "component": "api",
  "issue_key": "PROJ-100"
}

The JSM client logs errors with context for trace correlation:

{
  "level": "ERROR",
  "msg": "jsm api error",
  "component": "jsmclient",
  "status": 429,
  "error": "jsm: rate limited: rate limited",
  "body": "..."
}

Pending Writes Cleanup

A background goroutine runs every 60 seconds to clean up expired pending writes:

• Deletes rows where expires_at < NOW()
• Decrements the proxima_service_desk_pending_writes gauge by the number of deleted rows
• Logs the cleanup count at DEBUG level
• Respects server shutdown context for graceful termination

OTel Trace Spans

The following handlers create OTel trace spans, visible in Tempo:

Handler	Span Name	Notes
ListTickets	service_desk.list_tickets	Parent span; ClickHouse count and data queries are child spans
GetTicket	service_desk.get_ticket	Parent span; history, SLA, and delivery queries run as parallel child spans
CreateTicket	service_desk.create_ticket	Covers JSM API call and PostgreSQL pending write insert

The parallel sub-spans on GetTicket make it easy to identify which supplementary query (history, SLA, delivery) is slowest for a given request.

Cross-Signal Correlation

Use the standard observability workflow:

# 1. Find service desk errors
./scripts/obs.sh errors 15m | grep "service.desk\|clickhouse\|jsm"

# 2. Trace a specific request
./scripts/obs.sh trace <request_id>

# 3. Check service desk metrics trend
./scripts/obs.sh metrics proxima_service_desk_

Alerting Suggestions

Consider setting alerts for:

Condition	Query	Severity
JSM write error rate > 5%	rate(proxima_service_desk_write_errors_total[5m]) / rate(proxima_service_desk_writes_total[5m]) > 0.05	Warning
ClickHouse P99 latency > 5s	histogram_quantile(0.99, rate(proxima_service_desk_query_duration_seconds_bucket{source="clickhouse"}[5m])) > 5	Warning
Pending writes accumulating	proxima_service_desk_pending_writes > 50	Warning
JSM rate limiting	rate(proxima_service_desk_write_errors_total{action=~".*"}[5m]) > 0 and rate(proxima_service_desk_queries_total{source="jsm"}[5m]) > 40	Info