Inside VictoriaMetrics: Why It Needs Less RAM Than Prometheus

The first time I watched Prometheus get OOM-killed at 3am, I did what everyone does: I gave it a bigger box. That worked for about four months. Then it happened again, and the box I needed next was expensive enough that somebody was going to ask me to justify it.

That's the point where most teams go looking at VictoriaMetrics, see "up to 7x less RAM" on the homepage, and reasonably assume it's marketing. It isn't, mostly. But the number on its own isn't useful. What's useful is understanding where Prometheus spends memory that VictoriaMetrics doesn't, because that tells you whether the savings will show up in your workload or not.

So let's open it up.

The numbers people actually quote

Those benchmark figures come from VictoriaMetrics themselves, so treat them as a best case rather than a promise. The architecture underneath is real regardless, and it's the part worth knowing.

Only one file stays in memory

This is the whole story, so I'll put it up front.

VictoriaMetrics splits each per-month partition into parts, and a part isn't a file. It's a directory of column files: timestamps.bin, values.bin, index.bin, and metaindex.bin. Timestamps live apart from values, which is what makes the compression work later.

Of those four, metaindex.bin is the only one held permanently in memory. It's a small top-level directory of block locations. Everything else gets read on demand and lands in the OS page cache, where the kernel is free to evict it under pressure.

Compare that to Prometheus, which keeps the head block, its own series index, and recent chunks in the Go heap. Heap memory doesn't get evicted when the machine gets tight. It gets OOM-killed.

That's the actual mechanism. VictoriaMetrics pushes hot data into memory the kernel already knows how to reclaim, and keeps almost nothing in the heap that scales with your series count.

The write path, briefly

Incoming samples buffer in RAM as an in-memory part. Every second or so, that buffer flushes to disk as a compressed part and gets fsynced.

The neat detail: because the payload is compressed before it hits the disk, a flush is roughly 50 KB, where a comparable Prometheus WAL write is closer to 2 MB. Small enough that fsync-on-every-flush stops being scary on an SSD.

From there you get three tiers of parts inside a partition: in-memory, small parts freshly flushed, and big parts produced by background merges. Same columnar layout at every tier. Parts are immutable once written, which is why snapshots are just hard links and take effectively no time on multi-terabyte data.

Why the compression gets so far

Gorilla, the Facebook paper everybody's TSDB borrowed from, gets a 16-byte data point down to 2 to 5 bytes using delta-of-delta on timestamps and XOR on consecutive float values.

VictoriaMetrics layers more on top. It varint-encodes deltas for both timestamps and values, then runs zstd over the resulting block. Blocks hold up to 8,192 rows belonging to the same series, so everything inside a block is highly self-similar by construction, which is exactly what a general-purpose compressor wants.

Sorting by series first and compressing second is why column separation matters. A block of 8,192 timestamps from one scrape target compresses far better than 8,192 interleaved (timestamp, value) pairs from different targets.

IndexDB is where cardinality bites

Samples are the easy part. Finding them is where time-series databases actually struggle, and it's where your memory goes if you have a cardinality problem.

VictoriaMetrics keeps an inverted index in a structure called a mergeset. Internally, every series gets a numeric TSID, and IndexDB maintains mappings in both directions: tag to metric IDs, metric ID to TSID, metric ID to metric name, plus per-day variants so a query bounded to last Tuesday doesn't scan the global index.

The memory knob that matters here is the TSID cache, which by default takes about 37% of allowed memory. On ingestion, a cache hit skips the IndexDB lookup entirely. A cache miss doesn't.

So if you're ingesting a firehose of new series (a label with a request ID in it, say), you miss that cache constantly, and no amount of clever storage layout saves you. High cardinality is still high cardinality. VictoriaMetrics degrades more gracefully than Prometheus here, but it does degrade.

Where it's genuinely worse

I'd rather you hear this from me than find out in production.

Cold queries hit disk. The flip side of relying on page cache is that a query over data nothing has touched in a week reads from disk. Prometheus keeping things in heap means recent queries are consistently fast. VictoriaMetrics is fast when the cache is warm and merely fine when it isn't.

Alerting is a separate binary. There's no built-in rule evaluation. You run vmalert alongside it. Not hard, but it's another component in your diagram.

MetricsQL is a superset, not a clone. It accepts your PromQL, and mostly behaves identically. The differences are subtle enough that a dashboard can look right while being slightly wrong. Check your recording rules after migrating rather than assuming.

Troubleshooting the two things that will happen

Memory is still high. Almost always cardinality, not the engine. Hit /api/v1/status/tsdb to see your top series by label. Then lower -memory.allowedPercent from its default of 60 if you're squeezing caches to leave the OS more page cache, or raise it if you're seeing high disk IO from cache misses. Both directions are wrong sometimes.

Disk fills during merges. Big-part merges need room to write the new part before dropping the old ones. Keep meaningful headroom, and don't size the volume to your steady-state usage.

Reasonable starting point: -retentionPeriod defaults to 31 days and won't go below 24h, and -dedup.minScrapeInterval set to your scrape interval will quietly drop duplicate samples if you're running HA pairs.

Should you switch?

If Prometheus is comfortable on your current box, stay. The migration isn't free and you'd be trading a boring component for a slightly less boring one.

If you're rehearsing the "we need a bigger monitoring box" conversation for the second time, the architecture above is why the numbers move. Just measure your own cardinality before you assume you'll get the benchmark figures.

You can spin up managed VictoriaMetrics on Elestio if you'd rather not own the merge tuning and upgrade path yourself. VMs start at $16/month for the entry NETCUP config (2 vCPU, 4 GB RAM, 60 GB NVMe), though for real metric volume you'll want to size up from there.

Thanks for reading ❤️ See you in the next one 👋