Terminal Observability
Metrics
| Metric | Type | Labels | Description |
|---|---|---|---|
proxima_terminal_sessions_total | counter | login, role | Total sessions started |
proxima_terminal_sessions_active | updowncounter | — | Currently active sessions |
proxima_terminal_sessions_lingering | updowncounter | — | Sessions in the disconnect linger window (PTY alive, WebSocket gone) |
proxima_terminal_session_duration_seconds | histogram | end_reason | Session duration distribution |
proxima_terminal_reconnects_total | counter | result (success|expired|denied) | Auto-reconnect attempts |
proxima_terminal_auth_denied_total | counter | login, role | RBAC denials at connect time |
proxima_terminal_errors_total | counter | — | Session-level errors (I/O, PTY, NATS) |
proxima_host_users_created_total | counter | host_id | Host users auto-created by the agent |
proxima_host_users_deleted_total | counter | host_id | Host users cleaned up in drop mode |
proxima_host_user_create_errors_total | counter | host_id, step | User creation failures (by step: useradd, chage, homedir, groups, etc.) |
proxima_host_user_create_duration_seconds | histogram | — | Time to complete the full user creation sequence |
proxima_terminal_template_resolve_total | counter | status (ok|error) | Total login template resolution attempts |
proxima_terminal_template_resolve_duration_seconds | histogram | — | Time to resolve login templates per session |
proxima_terminal_recording_upload_duration_seconds | Histogram | — | R2 upload time |
proxima_terminal_recording_upload_errors_total | Counter | — | Failed recording uploads |
proxima_terminal_recording_sessions_total | Counter | status (completed/truncated/failed) | Recording outcomes |
proxima_terminal_peers_active | gauge | mode | Currently connected session peers |
proxima_terminal_peer_joins_total | counter | mode | Total peer join events |
proxima_terminal_file_transfer_total | counter | direction, status | Total file transfers |
proxima_terminal_file_transfer_bytes_total | counter | direction | Total bytes transferred |
proxima_terminal_file_transfer_duration_seconds | histogram | direction | Transfer duration |
Traces
| Span | Attributes | Description |
|---|---|---|
TerminalHandler.CheckAccess | host_id | RBAC access check |
TerminalHandler.HandleWebSocket | host_id, login, user_id | Full WebSocket session |
TerminalHandler.ResolveTemplates | terminal.template_logins_raw, terminal.template_logins_resolved, terminal.traits_count | Login template resolution |
Agent.CreateHostUser | terminal.sudoers_count, terminal.default_shell, terminal.uid_min, terminal.uid_max | Host user auto-creation |
Trace Attribute Reference
| Attribute | Where | Description |
|---|---|---|
terminal.sudoers_count | Agent.CreateHostUser | Number of sudoers rules written for the user |
terminal.default_shell | Agent.CreateHostUser | Shell path set for the auto-created user |
terminal.uid_min | Agent.CreateHostUser | Lower bound of the UID range from role options |
terminal.uid_max | Agent.CreateHostUser | Upper bound of the UID range from role options |
terminal.template_logins_raw | TerminalHandler.ResolveTemplates | Comma-separated raw template strings before resolution |
terminal.template_logins_resolved | TerminalHandler.ResolveTemplates | Comma-separated resolved login names after template execution |
terminal.traits_count | TerminalHandler.ResolveTemplates | Number of OIDC traits available to the template engine |
Audit Events
| Event | Details | Description |
|---|---|---|
terminal.session_started | session_id, host_id, linux_user | Session opened |
terminal.session_ended | session_id, host_id, linux_user | Session closed |
Security Logging
The agent logs security-relevant events at WARN level when session token verification fails. Look for these in agent logs:
| Log Message | Meaning |
|---|---|
session token verified | Token valid, PTY allocated (INFO) |
terminal session rejected: no token provided | Agent received a command with empty token (WARN) |
terminal session token verification failed | Invalid signature, expired, or wrong host_id (WARN) |
terminal session token login mismatch | Token's login doesn't match requested user (WARN) |
Query agent logs for failed terminal auth attempts:
./scripts/obs.sh logs '"terminal session token verification failed"' 1h
./scripts/obs.sh logs '"terminal session rejected"' 1h
Database
Query active sessions:
SELECT id, user_id, host_id, linux_user, role_name, started_at
FROM terminal_sessions
WHERE ended_at IS NULL
ORDER BY started_at DESC;
Query session history:
SELECT id, linux_user, role_name, started_at, ended_at, end_reason,
EXTRACT(EPOCH FROM (ended_at - started_at)) AS duration_s
FROM terminal_sessions
WHERE host_id = '<host-id>'
ORDER BY started_at DESC
LIMIT 20;