HoginHogin
Go back

VictoriaMetrics instead of Prometheus+Thanos: one binary instead of three — when it's honest savings

9 мин чтения

Prometheus is excellent as long as your metrics fit the local disk: two weeks of retention, one StatefulSet, zero external dependencies. The pain starts with “what happened a month ago?” — and Thanos bolts onto Prometheus: sidecar, store gateway, compactor, query. VictoriaMetrics covers the same scenario with a single process, and in July 2026 it shipped the v1.148.0 LTS line with a year of fixes ahead — a good moment to lay out honestly at what metric size this is savings, and where it is self-deception.

Table of contents

Open Table of contents

Where “just Prometheus” ends up

The local TSDB stores blocks on a PV: a year of retention means a terabyte disk and cold-cache restarts. HA means two Prometheus instances and dedup at read time.

The standard answer is Thanos. Honest architecture for terabytes — but count the components: a sidecar next to every Prometheus, a store gateway (index over old blocks), a compactor (compaction and downsampling), and query (stitching fresh and old). Four to five deployments with versions and upgrades of their own, plus a bucket in S3.

Prometheus+Thanos: five components for long-term storage — versus one VictoriaMetrics process

On a small cluster each is nearly empty: a store gateway over gigabytes, a query nobody loads. You operate a topology built for terabytes to serve tens of gigabytes.

What VictoriaMetrics changes

What you actually need from the ecosystem at small scale:

The VictoriaMetrics stack: vmagent writes to vmsingle, vmalert reads, vmbackup snapshots to S3, Grafana on top

The applicability threshold: single vs cluster

Operate by three numbers: active time series (unique label sets), ingest rate (samples/sec), and churn (series born and dying per hour).

The scale ladder: under 100k — single, no questions; up to 1M — comfort zone; beyond — cluster

A caveat about HA: single is one process. vmbackup backups are mandatory, and live ingest failover is an argument for cluster mode — not for “a second vmsingle”. vmagent’s buffer softens the picture: while the storage is unavailable, metrics pile up on the agent’s disk.

Prometheus+Thanos vs VM single

Prometheus + ThanosVM single
Storage components4–5 deployments1 (+ a backup CronJob)
StorageS3 + local TSDBPV + S3 for backups
Stack RAM~5–6 GB~1–1.5 GB
Disk for the same data1×usually 2–4× less
Deep queriesquery + store, with lagone endpoint
Changing retentioncompaction, downsampling-retentionPeriod + restart
Upgrades4 components in syncone tag for the stack
Ingest HAtwo Prometheus + dedupcluster mode
Rules and alertsPrometheus rulesvmalert + Alertmanager, same rules
Query languagePromQLMetricsQL (a superset)

Two migration paths

Remote-write from the existing Prometheus. Minimal risk: add a remote_write to http://vmsingle:8428/api/v1/write, shrink local retention, move rules to vmalert as you go. The rollback is deleting one line. History older than retention is migrated by vmctl prometheus (TSDB snapshot import).

Full replacement. vmagent takes over scraping (the same kubernetes_sd roles and relabel configs), Prometheus goes away. A whole stack and most of the monitoring RAM disappear; the price is moving rules. That’s the scenario deployed below.

What it takes to assemble the stack in one namespace

The whole stack lives in the monitoring namespace, on the v1.148.0 LTS (fixes until summer 2027): vmagent writes to vmsingle, vmalert runs rules into Alertmanager, vmbackup snapshots to S3 every 6 hours, Grafana reads through one datasource.

1. Storage: vmsingle. A Deployment with a PVC and a Service:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vmsingle
  namespace: monitoring
spec:
  replicas: 1
  selector: { matchLabels: { app: vmsingle } }
  template:
    metadata: { labels: { app: vmsingle } }
    spec:
      containers:
        - name: vmsingle
          image: victoriametrics/victoria-metrics:v1.148.0
          args:
            - -storageDataPath=/storage
            - -retentionPeriod=12m      # months
            - -search.maxUniqueTimeseries=500000   # above your series count
          ports: [{ containerPort: 8428 }]
          volumeMounts: [{ name: data, mountPath: /storage }]
      volumes:
        - name: data
          persistentVolumeClaim: { claimName: vmsingle-data }
---
apiVersion: v1
kind: Service
metadata:
  name: vmsingle
  namespace: monitoring
spec:
  selector: { app: vmsingle }
  ports: [{ port: 8428 }]

2. Scraper: vmagent. The scrape config is plain Prometheus:

apiVersion: v1
kind: ConfigMap
metadata:
  name: vmagent-config
  namespace: monitoring
data:
  scrape.yml: |
    global:
      scrape_interval: 30s
    scrape_configs:
      - job_name: kubernetes-pods
        kubernetes_sd_configs:
          - role: pod
        relabel_configs:
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
            action: keep
            regex: "true"
          - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
            action: replace
            regex: ([^:]+)(?::\d+)?;(\d+)
            replacement: $1:$2
            target_label: __address__
      # nodes — role: node via the apiserver proxy, config identical to Prometheus

The agent’s args — cardinality-explosion protection lives here too:

# Deployment vmagent, excerpt
image: victoriametrics/vmagent:v1.148.0
args:
  - -promscrape.config=/config/scrape.yml
  - -remoteWrite.url=http://vmsingle:8428/api/v1/write
  - -remoteWrite.tmpDataPath=/buffer      # buffer while storage is down
  - -remoteWrite.maxHourlySeries=300000   # cap on new series per hour
  - -remoteWrite.maxDailySeries=1000000   # and per day: a bad-release fuse

3. Rules: vmalert. Rules from kube-prometheus-stack move over unchanged; an example:

apiVersion: v1
kind: ConfigMap
metadata:
  name: vmalert-rules
  namespace: monitoring
data:
  rules.yaml: |
    groups:
      - name: k8s
        rules:
          - alert: PodRestartLoop
            expr: increase(kube_pod_container_status_restarts_total[15m]) > 3
            for: 5m
            labels: { severity: warning }
# Deployment vmalert, excerpt
image: victoriametrics/vmalert:v1.148.0
args:
  - -datasource.url=http://vmsingle:8428
  - -remoteWrite.url=http://vmsingle:8428      # recording rules — back into VM
  - -remoteRead.url=http://vmsingle:8428       # alert state after restart
  - -notifier.url=http://alertmanager:9093     # Alertmanager sits alongside
  - -rule=/config/*.yaml

4. Backups: vmbackup on a CronJob. The snapshot is created on the vmsingle side:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: vmbackup
  namespace: monitoring
spec:
  schedule: "0 */6 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: vmbackup
              image: victoriametrics/vmbackup:v1.148.0
              args:
                - -snapshot.createURL=http://vmsingle:8428/snapshot/create
                - -dst=s3://metrics-backup/vmsingle
                - -storageDataPath=/storage
                - -s3.endpoint=s3.eu-central-1.amazonaws.com
              volumeMounts: [{ name: data, mountPath: /storage }]
          volumes:
            - name: data
              persistentVolumeClaim: { claimName: vmsingle-data }

S3 keys come from environment variables out of a Secret (AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY).

5. Grafana. A prometheus datasource — the API is compatible:

apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-datasources
  namespace: monitoring
data:
  vm.yaml: |
    apiVersion: 1
    datasources:
      - name: VictoriaMetrics
        type: prometheus
        url: http://vmsingle:8428
        isDefault: true

Cardinality: the main budget killer

Cardinality is the number of unique label sets. Every new pod and endpoint multiplies it, and one wrong label (user_id, request_id) turns 100k series into 10M. Find the offenders with two queries:

topk(10, count({__name__=~".+"}) by (__name__))
topk(10, count by (pod) ({__name__=~".+"}))

The cure, in escalating order:

Numbers: RAM and disk

A typical small prod setup: three nodes, ~50 pods, kube-prometheus-stack with a couple of exporters — around 150k active series at a 30s interval. The order of magnitude:

So: three times less RAM for the stack and two to four times less disk. The savings aren’t in licenses (everything you need is open source) but in operations: one object to upgrade instead of five.

How to verify it works

kubectl -n monitoring get pods
curl -s http://vmsingle.monitoring:8428/health
curl -s http://vmsingle.monitoring:8428/api/v1/series/count

Bottom line

Up to ~a million active time series, VictoriaMetrics single-binary is the cheapest-to-operate way to keep metrics for long: one process, one PV, one backup CronJob, several times less RAM and disk than Prometheus+Thanos on the same data. Thanos is justified when there are many clusters, more than one tenant, and terabytes of metrics — there its topology is a necessity, not redundancy. The dividing line is measurable: active time series. And so logs and traces live next to your metrics — see the minimal OpenTelemetry Collector covered separately.

Share this post:
Share this post on TelegramShare this post via email
Next Post
Crossplane 2.0 and the coming v1.20 EOL: cloud infrastructure via GitOps instead of Terraform state