Troubleshooting Toolkit

The scripts/obs.sh script provides a unified CLI for querying all three observability signals. It's also exposed via Makefile targets for convenience.

Commands

Command	Makefile Target	Description
obs.sh status	make obs-status	Health check of all observability services
obs.sh errors [duration]	make obs-errors	Find errors in logs (default: last 15 minutes)
obs.sh trace <request_id>	make obs-trace REQ_ID=...	Full request correlation: logs + Tempo trace link
obs.sh slow [duration] [threshold_ms]	make obs-slow	Find slow requests (default: >1000ms in last 15m)
obs.sh auth	make obs-auth	Auth failure investigation (metrics + failure logs)
obs.sh workers	make obs-workers	NATS worker errors + JetStream stats
obs.sh metrics <pattern>	make obs-metrics PATTERN=...	Grep Prometheus metrics by pattern
obs.sh logs <query> [duration]	make obs-logs QUERY=...	Raw LogsQL query against VictoriaLogs

Usage Examples

# Quick health check
make obs-status

# Find errors in the last hour
./scripts/obs.sh errors 1h

# Correlate a specific request: logs + trace span waterfall
make obs-trace REQ_ID=abc-123-def

# Find slow requests (last hour, >500ms threshold)
./scripts/obs.sh slow 1h 500

# Auth failure investigation
make obs-auth

# NATS worker errors + JetStream stats
make obs-workers

# Search Prometheus metrics
make obs-metrics PATTERN=proxima_auth_

# Raw LogsQL query
./scripts/obs.sh logs '"request completed"' 30m
./scripts/obs.sh logs 'level:ERROR' 1h

Recommended Workflow

When investigating a production issue:

1. obs.sh errors -- find the error and note the request_id
2. obs.sh trace <request_id> -- see full logs + trace span waterfall
3. obs.sh metrics <pattern> -- check if the error is part of a broader trend

Query Cheat Sheets

Tempo API (:3200)

# Get a specific trace by ID
curl -s 'http://localhost:3200/api/traces/{traceID}' | jq .

# Search traces using TraceQL
curl -s 'http://localhost:3200/api/search' \
  --data-urlencode 'q={resource.service.name="proxima-backend"}' \
  --data-urlencode 'limit=10'

# Search traces by request_id tag
curl -s 'http://localhost:3200/api/search' \
  --data-urlencode 'q={span.request_id="abc-123"}' \
  --data-urlencode 'limit=5'

# Find slow traces (> 1s)
curl -s 'http://localhost:3200/api/search' \
  --data-urlencode 'q={resource.service.name="proxima-backend" && duration > 1s}' \
  --data-urlencode 'limit=10'

MetricsQL (VictoriaMetrics)

# HTTP error rate (5xx) over 5 minutes
curl -s 'http://localhost:8481/select/0/prometheus/api/v1/query' \
  --data-urlencode 'query=rate(proxima_http_requests_total{status=~"5.."}[5m])'

# Auth login failures
curl -s 'http://localhost:8481/select/0/prometheus/api/v1/query' \
  --data-urlencode 'query=rate(proxima_auth_login_total{result="failed"}[5m])'

# HTTP request latency p95
curl -s 'http://localhost:8481/select/0/prometheus/api/v1/query' \
  --data-urlencode 'query=histogram_quantile(0.95, rate(proxima_http_request_duration_seconds_bucket[5m]))'

Prometheus /metrics (:8080)

# All backend metrics
curl -s http://localhost:8080/metrics

# HTTP counters
curl -s http://localhost:8080/metrics | grep proxima_http_

# Auth counters
curl -s http://localhost:8080/metrics | grep proxima_auth_

NATS Monitoring (:8222)

# Server info
curl -s http://localhost:8222/varz

# JetStream info
curl -s http://localhost:8222/jsz

# Connections
curl -s http://localhost:8222/connz

See also

• Degraded Mode & NATS Outages — what happens when NATS (or another non-critical backing service) is down, and the step-by-step operator runbook for diagnosing and recovering a NATS outage.