LSM compaction is a background job that will wake you at 3am

Write latency is flat for six weeks. Then p99 on a write path that has never been slow jumps to four seconds, holds there for twenty minutes, and returns to normal on its own. Traffic did not change. No deploy went out. The disk was at 100 percent utilisation the entire time and the application was barely issuing queries.

Something was writing hard to that disk and it was not your workload. It was the engine rewriting data it had already written, which is the deal you accepted when you chose an LSM tree, whether or not anyone told you the terms.

Writes are fast because the work is deferred, not avoided

An LSM engine takes a write, appends it to a commit log for durability, and puts it in an in-memory table. That is the whole write path. No seeking to find the right page, no updating in place, no page splits. It is close to the fastest thing a storage engine can do, and it is why Cassandra, RocksDB, ScyllaDB and every LSM backed store advertise write throughput as their headline number.

When the memtable fills, it gets flushed to disk as an immutable sorted file. Now you have a file. Do that a few hundred times and you have a few hundred files, each individually sorted and collectively overlapping, and a point lookup has to consider all of them.

Compaction is the process that merges those files back into fewer, larger, non-overlapping ones. It is not an optimisation you can skip. Without it, read amplification grows without bound until a single key lookup is checking hundreds of files.

flowchart TB
    W["Write"] --> WAL["Commit log<br/>append only"]
    W --> MEM["Memtable<br/>in memory"]
    MEM -->|"flush when full"| L0["Level 0<br/>overlapping files"]
    L0 -->|"compaction"| L1["Level 1<br/>non-overlapping"]
    L1 -->|"compaction"| L2["Level 2<br/>10x larger"]
    L2 -->|"compaction"| LN["Level N"]

    style L0 stroke:#f59e0b,stroke-width:3px,color:#fff
    style LN stroke:#4ade80,stroke-width:2px,color:#fff

Every one of those arrows rewrites data that was already on disk. A row written once can be physically written ten or more times on its way down the levels. That multiplier is write amplification, and it is the price of the fast write path at the top.

This is the same trade the RUM conjecture describes, just made concrete: LSM trees optimise update cost and space at the expense of read cost, and compaction is the machinery that pays down the read cost in the background.

The stall is a feature, and it feels like an outage

Here is the part that makes it a 3am problem rather than a capacity planning problem.

If writes arrive faster than compaction can drain level zero, files accumulate. Reads get slower with every file added, because a lookup that misses in the memtable has to check each level zero file individually (level zero files overlap, so you cannot binary search across them). Left alone, read latency would degrade without limit.

The engine refuses to let that happen. RocksDB has two thresholds:

level0_slowdown_writes_trigger   default 20   -> writes are throttled
level0_stop_writes_trigger       default 36   -> writes block entirely

Cross the first and the engine deliberately adds delay to every write to let compaction catch up. Cross the second and writes stop until it does.

Your application sees write latency go from 2ms to 4000ms with no error, no exception, and no obvious cause. The database is not down. It is applying backpressure, correctly, at the only layer that can see the problem. That decision is defensible and I still find it unsettling to watch, because from the application side it is indistinguishable from the storage layer having a stroke.

Cassandra shows the same shape through pending compactions. A number that sits at 2 is healthy. A number that climbs steadily and never comes back down means compaction is losing, and the time to act is while it is climbing rather than when it reaches the threshold.

The metric that predicts it

Disk utilisation is the wrong signal, and it is the one on every dashboard. Compaction saturates the disk during normal healthy operation, so 100 percent utilisation tells you compaction is running rather than that anything is wrong.

What predicts a stall is the backlog:

Pending compaction bytes is the one to alert on, and the alert should fire on a sustained upward trend rather than a fixed threshold, because the absolute number is workload dependent and the direction is not.

Compaction needs room to work

Merging files means writing the output before deleting the inputs. For the duration of the merge, both exist.

That has a consequence people meet exactly once: a disk at 80 percent capacity can be unable to compact, because the merge needs space the disk does not have. Compaction fails, files accumulate, level zero grows, writes stall, and the obvious remedy of deleting data does not work because deletions in an LSM tree are tombstones, which are writes, which need compaction to reclaim space.

You need free space to free space. Running an LSM engine at high disk utilisation is not thrift, it is a countdown.

Size-tiered compaction is worse for this than leveled, because it merges several similarly sized files at once and can temporarily need close to double the size of the data being merged. The commonly cited guidance of keeping half the disk free under size-tiered compaction sounds absurdly wasteful until you have watched a cluster wedge itself at 85 percent.

Choosing the strategy is choosing which pain

Leveled compaction keeps each level fully sorted and non-overlapping, merging eagerly downward. Reads touch few files, space overhead is low, and writes pay for it: a row can be rewritten once per level.

Size-tiered compaction waits for several files of similar size and merges them together. Writes are much cheaper. Reads have to check more files, and space can balloon during a merge.

leveled      -> low read amp, low space amp, high write amp
size-tiered  -> low write amp, high read amp, high transient space amp

Neither is correct in general. A time series or event ingestion workload where reads are rare and mostly recent wants size-tiered. A workload doing frequent point lookups across the whole key space wants leveled. Picking the wrong one produces exactly the symptoms you were trying to avoid, which is why the default is worth revisiting rather than inheriting.

Bloom filters sit alongside this and take some of the read amplification off. A lookup that would otherwise open several files can skip most of them on a negative filter result, which is why the memory footprint of the filter and whether it stays in cache matters more than its theoretical false positive rate.

What I would put in place before the next incident

Alert on pending compaction bytes trending up over an hour, not on a threshold. The threshold varies by workload and the trend does not.

Keep enough free disk that the largest possible compaction can complete, and treat the resulting utilisation number as a floor rather than waste. A disk that looks half empty is a disk that can still compact.

Schedule bulk loads and large deletes with compaction in mind. Dropping a few hundred million rows generates tombstones that will not free space until compaction processes them, and doing it during peak traffic means competing with the foreground for the same disk.

Check whether the compaction strategy matches the access pattern rather than the one that shipped as default. This is a one line configuration change and one of the few places where a single setting genuinely moves the shape of the system.

The framing that helped me most was giving up on compaction as maintenance. It is not a background chore that happens to run occasionally. It is the second half of every write your application performed, executing later, on a schedule the engine controls and your traffic pattern decides. The write returned quickly because the work was deferred. Compaction is that deferral coming due, and it comes due at whatever time your write volume dictates, which is rarely a time anyone chose.