Terminal Observability

Metrics

Metric	Type	Labels	Description
proxima_terminal_sessions_total	counter	login, role	Total sessions started
proxima_terminal_sessions_active	updowncounter	—	Currently active sessions
proxima_terminal_session_duration_seconds	histogram	end_reason	Session duration distribution
proxima_terminal_auth_denied_total	counter	login, role	RBAC denials at connect time
proxima_terminal_errors_total	counter	—	Session-level errors (I/O, PTY, NATS)
proxima_terminal_template_resolve_total	counter	status (ok|error)	Total login template resolution attempts
proxima_terminal_template_resolve_duration_seconds	histogram	—	Time to resolve login templates per session
proxima_recording_upload_duration_seconds	Histogram	—	R2 upload time
proxima_recording_upload_errors_total	Counter	—	Failed recording uploads
proxima_recording_sessions_total	Counter	status (completed/truncated/failed)	Recording outcomes
proxima_terminal_peers_active	gauge	mode	Currently connected session peers
proxima_terminal_peer_joins_total	counter	mode	Total peer join events
proxima_terminal_file_transfer_total	counter	direction, status	Total file transfers
proxima_terminal_file_transfer_bytes_total	counter	direction	Total bytes transferred
proxima_terminal_file_transfer_duration_seconds	histogram	direction	Transfer duration
proxima_grpc_terminal_sessions_total	counter	—	gRPC terminal sessions started (the pc CLI path)
proxima_grpc_terminal_sessions_active	updowncounter	—	Currently active gRPC terminal sessions

Recording metrics have no terminal_ segment

They are proxima_recording_*, not proxima_terminal_recording_*. A PromQL query written with the longer name matches no series.

Traces

Span	Attributes	Description
TerminalHandler.CheckAccess	host_id	RBAC access check
TerminalHandler.HandleWebSocket	host_id, login, user_id	Full WebSocket session
TerminalHandler.ResolveTemplates	terminal.template_logins_raw, terminal.template_logins_resolved, terminal.traits_count	Login template resolution
Agent.CreateHostUser	terminal.sudoers_count, terminal.default_shell, terminal.uid_min, terminal.uid_max	Host user auto-creation

Trace Attribute Reference

Attribute	Where	Description
terminal.sudoers_count	Agent.CreateHostUser	Number of sudoers rules written for the user
terminal.default_shell	Agent.CreateHostUser	Shell path set for the auto-created user
terminal.uid_min	Agent.CreateHostUser	Lower bound of the UID range from role options
terminal.uid_max	Agent.CreateHostUser	Upper bound of the UID range from role options
terminal.template_logins_raw	TerminalHandler.ResolveTemplates	Comma-separated raw template strings before resolution
terminal.template_logins_resolved	TerminalHandler.ResolveTemplates	Comma-separated resolved login names after template execution
terminal.traits_count	TerminalHandler.ResolveTemplates	Number of OIDC traits available to the template engine

Audit Events

Event	Details	Description
terminal.session_started	session_id, host_id, linux_user	Session opened
terminal.session_ended	session_id, host_id, linux_user	Session closed

Security Logging

The agent logs security-relevant events at WARN level when session token verification fails. Look for these in agent logs:

Log Message	Meaning
session token verified	Token valid, PTY allocated (INFO)
terminal session rejected: no token provided	Agent received a command with empty token (WARN)
terminal session token verification failed	Invalid signature, expired, or wrong host_id (WARN)
terminal session token login mismatch	Token's login doesn't match requested user (WARN)

Query agent logs for failed terminal auth attempts:

./scripts/obs.sh logs '"terminal session token verification failed"' 1h
./scripts/obs.sh logs '"terminal session rejected"' 1h

Database

Query active sessions:

SELECT id, user_id, host_id, linux_user, role_name, started_at
FROM terminal_sessions
WHERE ended_at IS NULL
ORDER BY started_at DESC;

Query session history:

SELECT id, linux_user, role_name, started_at, ended_at, end_reason,
       EXTRACT(EPOCH FROM (ended_at - started_at)) AS duration_s
FROM terminal_sessions
WHERE host_id = '<host-id>'
ORDER BY started_at DESC
LIMIT 20;