VictoriaMetrics Architecture

Proxima Console uses VictoriaMetrics Cluster for time-series metrics storage. This replaced TimescaleDB in v0.2.0.

Production vs local dev

In production, the Console backend (running in the proxima-production Kubernetes cluster) does not run its own VM cluster. It writes and reads metrics against the shared in-cluster VictoriaMetrics cluster (in the monitoring namespace) through the vmauth-proxy, isolated in its own projectID 42 as the vmuser-console VMUser. The docker-compose VM cluster described in Cluster Components and Docker Compose Setup below is local-dev only. See Production Deployment.

Cluster Components

Component	Port	Role
vmstorage	8482	Stores raw metric data. Handles compaction, retention, and deduplication.
vminsert	8480	Accepts incoming metrics via Prometheus import API. Routes to vmstorage.
vmselect	8481	Handles MetricsQL queries. Reads from vmstorage and returns results.
vmagent	—	Scrapes backend /metrics endpoint and remote-writes to vminsert (account 0).

Configuration

The backend connects to VictoriaMetrics via two environment variables (local-dev values shown):

VM_INSERT_URL=http://localhost:8480    # vminsert endpoint
VM_SELECT_URL=http://localhost:8481    # vmselect endpoint
VM_WRITE_TIMEOUT=10                   # Write timeout (seconds)
VM_QUERY_TIMEOUT=30                   # Query timeout (seconds)

In production these point at vmauth-proxy.monitoring:8427 instead — see Production Deployment and the VictoriaMetrics environment variables.

Production Deployment

In the proxima-production Kubernetes cluster, Console does not run its own VM cluster. Instead, both the metrics write path (MetricsWorker) and read path (Metrics API) go through the shared in-cluster VictoriaMetrics cluster's vmauth-proxy:

# Production (set on the backend Rollout; secrets via External-Secrets ← Vault)
VM_INSERT_URL=http://vmauth-proxy.monitoring.svc.cluster.local:8427
VM_SELECT_URL=http://vmauth-proxy.monitoring.svc.cluster.local:8427
VM_PROJECT_ID=42                       # Console's isolated VM project
VM_AUTH_USERNAME=vmuser-console        # VMUser vmauth authenticates Console as
VM_AUTH_PASSWORD=...                   # injected via ESO from vault.prxm.uz

Aspect	Local dev (docker-compose)	Production (k8s)
VM cluster	Console's own (vmstorage/vminsert/vmselect)	Shared in-cluster VM cluster (monitoring ns)
Endpoint	vminsert:8480 / vmselect:8481 directly	vmauth-proxy.monitoring:8427 (both paths)
Auth	none	VMUser vmuser-console (password from Vault via ESO)
Tenant scope	accountID only, project 0	accountID + projectID 42 → tenant <accountID>:42

The vmauth-proxy authenticates Console as the vmuser-console VMUser and confines it to projectID 42, so Console's per-client tenants are written and read as <accountID>:42 — fully separated from the shared cluster's other workloads, which use the default :0 project. Tenant resolution (the per-client vm_account_id) is unchanged; the projectID is the second, Console-wide isolation dimension layered on top. See Multi-Tenancy.

Docker Compose Setup

Local development only

The docker-compose VM cluster below runs only for local development. Production uses the shared in-cluster VM cluster via vmauth-proxy (see Production Deployment).

The development docker-compose.yml runs all three components:

vmstorage:
  image: victoriametrics/vmstorage:v1.136.0-cluster
  command:
    - "-retentionPeriod=120d"
    - "-storageDataPath=/storage"
  volumes:
    - vmstorage-data:/storage

vminsert:
  image: victoriametrics/vminsert:v1.136.0-cluster
  command:
    - "-storageNode=vmstorage:8400"
  ports:
    - "8480:8480"

vmselect:
  image: victoriametrics/vmselect:v1.136.0-cluster
  command:
    - "-storageNode=vmstorage:8401"
  ports:
    - "8481:8481"

Backend Metrics Scraping (vmagent)

The backend exposes Prometheus metrics at /metrics via the OTel SDK with a Prometheus exporter. vmagent scrapes these metrics and remote-writes them to vminsert under account 0 (reserved for backend self-monitoring, separate from tenant-scoped agent metrics).

Scrape Config

The scrape configuration lives at infra/vmagent/scrape.yml:

scrape_configs:
  - job_name: proxima-backend
    scrape_interval: 15s
    static_configs:
      - targets: ["host.docker.internal:8080"]
    metrics_path: /metrics

Grafana Datasources

Two VictoriaMetrics datasources are provisioned in Grafana:

Datasource	Account	Purpose
VictoriaMetrics	1+	Agent/host metrics (per-client tenant isolation)
VictoriaMetrics-Backend	0	Backend self-monitoring metrics (not tenant-scoped)

A pre-built dashboard "Proxima Backend Observability" (infra/grafana/dashboards/backend-observability.json) visualizes all backend metrics across 11 rows and 37 panels, organized using RED (Rate, Errors, Duration) and USE (Utilization, Saturation, Errors) methodology. See Grafana Dashboards for the full structure and Metrics Collected — Backend Self-Monitoring for the metric list.

Multi-Tenancy

VictoriaMetrics Cluster supports native multi-tenancy via URL path segments. Proxima Console uses this to isolate each client's metrics:

1. Each client has a vm_account_id column (auto-incrementing sequence, defined in the initial schema migration 000001).
2. The MetricsWorker resolves the client's account ID via the tenantcache package.
3. Writes go to /insert/{accountID}/prometheus/api/v1/import.
4. Reads go to /select/{accountID}/prometheus/api/v1/query_range.

No cross-client query is possible through the API — tenant isolation is enforced at the URL routing level.

In production, a second isolation dimension is added: the shared in-cluster VM cluster is multi-tenant across many ProximaOps workloads, so Console is confined to projectID 42 as the vmuser-console VMUser via vmauth-proxy. Each client's data is therefore stored under the composite tenant <accountID>:42, keeping Console's metrics separate from everything else in the cluster (which uses the default project 0). The per-client accountID logic above is identical in both local dev and production — only the surrounding projectID and the vmauth-proxy front door differ. See Production Deployment.

Tenant Cache

The tenantcache package maintains an in-memory cache of host-to-tenant mappings:

• TTL: 5 minutes (configurable)
• Thread-safe: Uses sync.RWMutex for concurrent access
• Batch resolution: Can resolve multiple hosts in a single database query
• Eviction: Entries expire after TTL; no manual invalidation needed

Data Import Format

Metrics are sent to vminsert using the Prometheus import API:

{
  "metric": {
    "__name__": "memory_used_bytes",
    "host_id": "c5f8a2b0-...",
    "environment_id": "e3a1b4c2-...",
    "labels_key": ""
  },
  "values": [4294967296],
  "timestamps": [1708099200000]
}

Query Language

The backend uses MetricsQL (VictoriaMetrics' extended PromQL) for all queries:

# Raw metric value (aggregated across labels)
sum({__name__="cpu_usage_ratio", host_id="..."})

# Per-series metric (specific disk)
{__name__="disk_used_ratio", host_id="...", labels_key="device=sda,mount=/"}

# Rate of change for counter metrics
sum(rate({__name__="disk_read_bytes_total", host_id="..."}[1m]))

# Environment-level average
avg({__name__="cpu_usage_ratio", environment_id="..."})

Why VictoriaMetrics?

The migration from TimescaleDB to VictoriaMetrics was driven by:

Factor	TimescaleDB	VictoriaMetrics
Compression	10-20x	20-70x
Query speed	SQL (slow for high cardinality)	Native vector operations
Multi-tenancy	Schema-level (complex)	URL-path native
Maintenance	PostgreSQL extension	Standalone binary
Retention	Manual policy	Built-in -retentionPeriod
Dependencies	Requires PostgreSQL extension	Independent service

Added in v0.2.0

VictoriaMetrics replaced TimescaleDB for all time-series storage. PostgreSQL is now used only for relational data (clients, environments, hosts, inventory). See the Upgrade Guide for migration details.