MCP Server Observability

The MCP server provides three pillars of observability: structured logs, distributed traces, and Prometheus metrics.

Prometheus Metrics

All metrics are exported at GET /metrics (port 8081) in Prometheus exposition format. Metric names follow the proxima_mcp_* prefix convention.

Tool Calls

Metric	Type	Labels	Description
proxima_mcp_tool_calls_total	counter	tool, status	Total tool invocations (status: success, error)
proxima_mcp_tool_duration_seconds	histogram	tool	Tool execution latency in seconds

Auth Cache

Metric	Type	Labels	Description
proxima_mcp_auth_validations_total	counter	result	Auth validation outcomes (result: hit, miss, error)

Backend HTTP Client

Metric	Type	Labels	Description
proxima_mcp_backend_requests_total	counter	method, path, status_code	Total backend API calls
proxima_mcp_backend_request_duration_seconds	histogram	method, path	Backend API call latency in seconds

Example Queries

# Tool call rate by tool name (last 5 minutes)
rate(proxima_mcp_tool_calls_total[5m])

# Average tool latency by tool
rate(proxima_mcp_tool_duration_seconds_sum[5m]) / rate(proxima_mcp_tool_duration_seconds_count[5m])

# Auth cache hit ratio
sum(rate(proxima_mcp_auth_validations_total{result="hit"}[5m])) /
sum(rate(proxima_mcp_auth_validations_total[5m]))

# Backend error rate
sum(rate(proxima_mcp_backend_requests_total{status_code=~"5.."}[5m])) /
sum(rate(proxima_mcp_backend_requests_total[5m]))

# Slowest tools (P95 latency)
histogram_quantile(0.95, rate(proxima_mcp_tool_duration_seconds_bucket[5m]))

Scrape Configuration

Add the MCP server to your vmagent or Prometheus scrape config:

- job_name: proxima-mcp
  static_configs:
    - targets: ['localhost:8081']
  metrics_path: /metrics
  scrape_interval: 15s

Distributed Tracing

When PROXIMA_MCP_OTEL_ENABLED=true, the MCP server exports traces via OTLP gRPC to the configured endpoint (default localhost:4317). The tracer name is proxima-mcp.

Span Hierarchy

Each tool call produces a trace with the following span tree:

tool/{tool_name}          (SpanKind: Server)
  └── auth/validate       (internal)
  └── backend/GET         (SpanKind: Client)
  └── backend/POST        (SpanKind: Client)

Span Attributes

Tool spans (tool/*):

• mcp.tool — tool name
• mcp.tool.status — success or error

Auth spans (auth/validate):

• auth.key_prefix — first 10 characters of the API key
• auth.result — hit, miss, or error
• auth.user_id — authenticated user ID (on success)

Backend spans (backend/*):

• http.method — GET or POST
• http.url — backend API path
• http.status_code — response status code

Viewing Traces

If using the standard Proxima observability stack (Tempo + Grafana):

1. Open Grafana at https://grafana-console.prxm.uz
2. Go to Explore > Tempo
3. Search by service name proxima-mcp
4. Or filter by span name (e.g. tool/query_hosts)

Structured Logging

All logs are JSON-formatted to stdout via Go's slog package. Every log entry includes "component":"mcp".

Log Levels

Level	What's logged
debug	Auth cache hits/misses, backend request details, tool call start
info	Tool call completions with status and duration
warn	Auth validation failures, invalid env var values
error	Fatal startup errors, OTel shutdown errors

Key Log Fields

Field	Description
component	Always "mcp"
tool	Tool name (on tool call logs)
status	"success" or "error"
duration_ms	Operation duration in milliseconds
method	HTTP method for backend requests
path	Backend API path
status_code	Backend response status code
key_prefix	First 10 chars of API key (masked)
user_id	Authenticated user ID

Example Log Lines

{"time":"2026-03-12T10:00:01Z","level":"INFO","msg":"server started","component":"mcp","addr":":8081","transport":"http"}
{"time":"2026-03-12T10:00:05Z","level":"INFO","msg":"tool call completed","component":"mcp","tool":"query_hosts","status":"success","duration_ms":142}
{"time":"2026-03-12T10:00:05Z","level":"DEBUG","msg":"backend request completed","component":"mcp","method":"GET","path":"/api/v1/search","status_code":200,"duration_ms":89}
{"time":"2026-03-12T10:00:06Z","level":"WARN","msg":"auth validation failed","component":"mcp","error":"api error (status 401): unauthorized"}

MCP Protocol Notifications

The MCP server sends real-time feedback to connected clients during tool execution:

• Log messages via notifications/message — tool start/completion status
• Progress updates via notifications/progress — multi-step tools report intermediate progress (e.g. "Resolving host" → "Fetching metrics" → "Done")

These notifications appear in clients that support them (e.g. Claude Desktop shows progress indicators).

Health Check

The /healthz endpoint returns HTTP 200 with {"status":"ok"} when the server is running. This is used by:

• The Kubernetes readiness probe on the console-mcp Deployment
• Docker HEALTHCHECK in the container
• Monitoring tools (e.g. uptime checks)

Troubleshooting

MCP server won't start

MCP runs as a Kubernetes pod in console-system, reconciled by ArgoCD and fronted by an istio HTTPRoute — not as a systemd unit behind nginx.

# Pod status and recent restarts
kubectl -n console-system get pods -l app=console-mcp

# Logs
kubectl -n console-system logs deploy/console-mcp -f

# Effective environment (PROXIMA_API_URL and friends)
kubectl -n console-system exec deploy/console-mcp -- env | grep PROXIMA_

Tools returning "Proxima API is currently unavailable"

The MCP server cannot reach the backend. Check:

# Is the backend running?
curl -sf http://localhost:8080/healthz

# Can the MCP container reach it?
docker exec proxima-mcp wget -q -O- http://127.0.0.1:8080/healthz

High auth cache miss rate

If proxima_mcp_auth_validations_total{result="miss"} is high relative to tool calls, consider increasing PROXIMA_MCP_AUTH_CACHE_TTL. Each cache miss triggers a backend GET /api/v1/auth/validate call.

Slow tool responses

Check proxima_mcp_backend_request_duration_seconds to determine if slowness is in the backend or the MCP server itself. Most tool latency comes from backend API calls.