Cache invalidation is a distributed systems problem in a convenience costume

A product price is updated in the admin panel. The database is updated, the cache key is deleted, and the next read repopulates from the database. The code is four lines and it is the pattern everybody uses.

A customer keeps seeing the old price. Not for a few seconds. For the full hour of the TTL, on one pod, while every other pod shows the new one.

Nothing failed. No exception, no error, no dropped connection. Two operations that were correct individually happened in an order nobody considered, and the cache is now holding a value that will not be corrected until it expires.

Two stores, no transaction

The joke about cache invalidation being hard is repeated so often that it has stopped carrying information. The specific reason it is hard is worth stating plainly: you are keeping two independent stores consistent with no shared transaction between them.

That is a distributed systems problem. It has the same shape as any dual write, and it does not become easier because one of the stores is called a cache and the operation is called a delete.

Every version of it has a window:

sequenceDiagram
    participant W as Writer
    participant DB as Database
    participant C as Cache
    participant R as Reader

    R->>C: GET price:42
    C-->>R: miss
    R->>DB: SELECT price
    DB-->>R: 100 (old value)
    Note over R: reader now holds 100,<br/>about to write it back
    W->>DB: UPDATE price = 120
    W->>C: DEL price:42
    R->>C: SET price:42 = 100
    Note over C: Cache holds 100.<br/>Database holds 120.<br/>Nothing will fix this until the TTL.

The reader did nothing wrong. It missed, loaded, and populated. The writer did nothing wrong. It updated and invalidated. The interleaving is what produced a cache entry that outlives the value it describes.

This race is narrow, which is worse than it being wide. Narrow races reproduce rarely, survive code review, pass load tests, and surface as a support ticket from one customer that nobody else can replicate.

Ordering: invalidate after, not before

The first thing to get right is which side goes first, and the answer is less obvious than it looks because both orderings have a failure mode.

Invalidate first, then write the database, and the window above is wide open: any reader arriving between the delete and the commit loads the old value and caches it.

Write the database first, then invalidate, and the window shrinks to readers who had already loaded the old value before the commit and have not yet written it back. That is a much smaller target, because it requires the reader to be paused in the middle of a few microseconds of work.

So database first, cache second, always. That much is settled.

The remaining race is closed by invalidating twice, which is a technique that feels like a hack and is actually the standard answer.

def update_price(product_id, price):
    db.execute("UPDATE products SET price = %s WHERE id = %s", price, product_id)
    cache.delete(f"price:{product_id}")

    # Second delete after a short delay, to catch a reader that loaded the
    # old value before the commit and wrote it back after the first delete.
    schedule_after(0.5, lambda: cache.delete(f"price:{product_id}"))

The delay needs to exceed the maximum time between a reader’s database read and its cache write, which is normally a couple of milliseconds and occasionally much longer under GC pause or scheduling delay. Half a second is a common choice and it is a probabilistic fix rather than a guarantee, which is the honest way to describe it.

Delete, do not update

The other decision is whether to write the new value into the cache or remove the key.

Writing the new value looks more efficient. It saves the next reader a database round trip and it feels like less waste.

It also introduces a race that deletion does not have. Two concurrent writers updating the same key produce two cache writes, and there is nothing forcing those writes to land in the same order as the database commits.

Writer A: DB write price=100
Writer B: DB write price=120        <- database ends at 120
Writer B: cache set price=120
Writer A: cache set price=100       <- cache ends at 100, permanently wrong

Two deletes cannot do this. Delete is idempotent and order independent, so whichever order they arrive in, the key ends up absent and the next read repopulates from the database, which holds the winner.

That property is worth more than the saved round trip. Deletion also avoids computing and serialising a value that may never be requested, and it keeps the shape of the cached object owned by the read path rather than duplicated in every writer.

Every replica has its own copy of the problem

Local caches multiply this. A delete against Redis is one operation against one store. A delete against an in-process cache has to reach every instance, and there is no single place to send it.

The usual mechanism is a pub/sub broadcast, and the guarantees are weaker than they appear. Redis pub/sub is fire and forget: a pod that is restarting, GC-pausing, or briefly disconnected simply does not get the message, and nothing retries it. That pod now serves stale data for the rest of its TTL with no indication anything went wrong.

This is where the per-instance TTL stops being a performance knob and becomes the bound on incorrectness. It is the maximum time a missed invalidation can persist, and choosing it is choosing how wrong you are willing to be. I keep local cache TTLs short specifically for this reason, because the coherence cost is what makes a second cache tier more expensive than its hit rate suggests.

Deriving invalidation from the database’s own log

The version of this that actually removes the dual write is to stop having the application do both.

If invalidation is driven by the database’s replication stream, there is one ordered source of truth for what changed and in what order. The application writes to the database and nothing else. A consumer reads the change log and invalidates.

app -> database -> WAL / binlog -> CDC consumer -> cache delete

Ordering comes free, because the log is ordered. Missed invalidations become consumer lag, which is a thing you can measure and alert on rather than an event that vanishes. And a write that never committed never produces an invalidation, which removes the case where the cache is cleared for a transaction that rolled back.

The cost is a pipeline, a consumer to operate, and lag between the write and the invalidation. That lag is real and it is bounded and observable, which is a considerably better property than a race that is unbounded and invisible.

What I would actually do

Order the operations database first, cache second, and delete rather than update. Those two are free and remove most of the failure surface.

Treat the TTL as the correctness bound rather than the efficiency knob, and set it based on how long you can tolerate being wrong when an invalidation is lost. It will be lost eventually, because pub/sub does not retry and processes restart.

Reach for change data capture when the cost of staleness is high enough to justify a pipeline, and be honest that it trades an invisible race for a visible delay.

And accept that none of this makes the two stores atomic. There is no ordering of two operations against two systems that gives you a transaction. Every option here is about making the window smaller, more observable, or less damaging, which is a different goal from making it go away.