Local cache plus distributed cache: the coherence bill nobody budgets for

An API endpoint sits at 8ms p99 and most of that is Redis round trips. Somebody adds a Caffeine cache in front: hot keys are held in process, Redis is only consulted on a local miss.

p99 drops to 1.2ms. The change is 30 lines. It ships.

Two months later a support ticket says a customer’s tier shows as Basic in the app and Premium on the billing page. Both are served by the same service. Refreshing sometimes fixes it and sometimes does not, depending on which pod the load balancer picks.

Twelve pods, twelve independent caches, one of which missed an invalidation message four minutes ago and will keep serving the old tier until its TTL runs out.

Every instance is now its own source of truth

A distributed cache has one copy. Getting it wrong means everyone is wrong together, which is unpleasant and at least consistent.

Add a local tier and you have one copy per instance. Twelve pods means twelve caches that can each diverge independently, and the divergence is not visible from anywhere, because no instance can see any other instance’s memory.

flowchart TB
    W["Write: tier = premium"] --> DB[("Database")]
    W --> R[("Redis: invalidated")]
    W --> PS(("pub/sub broadcast"))
    PS --> P1["Pod 1<br/>evicted"]
    PS --> P2["Pod 2<br/>evicted"]
    PS -.->|"GC pause,<br/>message dropped"| P3["Pod 3<br/>still holds basic"]

    style P3 stroke:#ef4444,stroke-width:3px,color:#fff

Pod 3 is not broken. It is not reporting an error. It is serving a value it obtained legitimately and was never told to discard, and there is no health check that can detect this because the pod is healthy.

The routing layer decides which customers experience the bug, which is why it presents as intermittent and unreproducible. The customer refreshing gets a different pod and a different answer, and support closes the ticket as unable to reproduce.

Pub/sub is not a delivery guarantee

The standard invalidation mechanism is a broadcast, and it is worth being precise about what it promises.

Redis pub/sub is at most once. There is no persistence, no acknowledgement, no retry, and no replay. A subscriber that is not connected at the moment of publish does not receive the message and has no way to know it existed.

The situations where an instance is not connected are entirely ordinary:

A pod is starting up and has not subscribed yet, which happens on every deploy.

A pod is in a long garbage collection pause and the socket buffer fills, which is the same stop the world pause that shows up unexplained in your p99.

A network blip drops the connection and the client reconnects a second later, having missed everything in between.

A pod is being terminated and is draining, but still serving requests.

Redis keyspace notifications, which look like a more principled version of the same thing, are delivered over the same pub/sub mechanism and carry the same guarantee. Using them does not change the analysis.

Making invalidation reliable means a durable log with per-instance consumer groups, so each instance has its own cursor and can resume after a restart. That is Kafka, or something like it, and it is a real amount of machinery to attach to what began as a 30 line optimisation. Sometimes justified. Rarely what people had in mind.

The TTL is the correctness bound

Because the broadcast will fail eventually, the local TTL is the only mechanism guaranteed to repair a diverged entry.

That reframes the setting entirely. It is not a tuning parameter balancing hit rate against memory. It is a statement about the maximum duration of incorrectness the system can produce.

Caffeine.newBuilder()
    .maximumSize(10_000)
    // Not a performance knob. This is the longest a missed invalidation
    // can go uncorrected, because pub/sub will drop messages eventually.
    .expireAfterWrite(Duration.ofSeconds(20))
    .build();

Twenty seconds of staleness for a product name is fine. Twenty seconds for an entitlement that a user just paid for is a support ticket. Thirty minutes for either is a bug you will be debugging without any of the evidence, because by the time anyone looks the entry has expired and the system is behaving correctly.

expireAfterWrite rather than expireAfterAccess matters here too. Access based expiry keeps a frequently read key alive indefinitely, which means the hottest key, the one most likely to be observed as wrong, is also the one least likely to be corrected by the TTL.

Where the tier genuinely earns its keep

I do not want to argue nobody should do this, because for the right data the win is large and the risk is close to zero.

Data that is effectively immutable is the clear case. Feature flag definitions, currency codes, tax rate tables keyed by version, country lists, configuration loaded at startup. If a value only changes on deploy, cache it locally for as long as you like, because there is no coherence problem to have.

Data where staleness is invisible is the second case. Aggregate counts on a dashboard, recommendation lists, anything where being a minute behind is indistinguishable from being current.

Where I would not put a local tier is anything a user can change and then immediately look at. That flow specifically compares what the system told them with what they just did, which makes it the one path where a few seconds of staleness reads as a bug rather than as latency. It is the same failure as reading your own write from a lagging replica, reproduced inside the application process.

The version that avoids most of this

If the concern is Redis round trip latency rather than Redis throughput, there is a middle option people skip: keep one tier and make it faster.

Pipelining batches multiple Redis commands into one round trip, which addresses the same cost that the local cache was added to avoid, without adding a second copy of the data.

Request coalescing within an instance, so concurrent requests for the same key share one Redis lookup, removes duplicate work without retaining anything between requests.

Both keep a single source of truth and neither introduces a coherence problem. They are less dramatic than a near cache and they are also less likely to produce a bug that only manifests on one pod for four minutes at a time.

If you are keeping it

Set the local TTL from how long you can tolerate being wrong, not from hit rate, and prefer write based expiry so hot keys still refresh.

Broadcast invalidations and treat them as an optimisation that reduces average staleness, never as the mechanism that guarantees correctness.

Put the instance id in your responses during debugging, or in a header, because the first genuinely hard part of diagnosing this is establishing that two users are talking to different pods.

And write down which caches exist. A service with a near cache, a Redis tier, an HTTP cache header and a CDN in front of it has four places a value can be stale, and I have watched more than one investigation spend its first hour on the wrong one.