Hot keys defeat consistent hashing

A Redis cluster has six nodes, virtual nodes are configured, and the key distribution is textbook: each node holds within two percent of the same number of keys. Somebody checks this specifically, because one node is at 90 percent CPU while the other five sit at 20.

The distribution is not the problem. A single key holding the homepage product carousel receives roughly 40 percent of all cache reads, and that key hashes to node 3. It will hash to node 3 tomorrow, and after a rebalance, and after adding six more nodes, because a hash function is deterministic and that is the entire point of it.

Adding capacity does not help. Node 3 does not care how many other nodes exist.

Balanced keys is not balanced load

This is worth separating clearly, because the two things get conflated in almost every discussion of consistent hashing.

Consistent hashing solves two problems. It spreads the keyspace evenly across nodes, and it limits how much of that keyspace moves when a node joins or leaves. Virtual nodes exist to make the first one work properly, because a handful of physical nodes hashed onto a ring clump rather than spreading, which is the reason a naive ring implementation puts most of the traffic on one server.

Neither of those is about how often each key is requested. The ring has no idea that one key is read a million times a second and its neighbour is read once an hour.

flowchart TB
    subgraph even["Key distribution: even"]
        N1["Node 1<br/>1.67M keys"]
        N2["Node 2<br/>1.66M keys"]
        N3["Node 3<br/>1.67M keys"]
    end

    subgraph load["Request distribution: not even"]
        L1["Node 1<br/>12k req/s"]
        L2["Node 2<br/>11k req/s"]
        L3["Node 3<br/>140k req/s<br/>one key is 40% of traffic"]
    end

    style L3 stroke:#ef4444,stroke-width:3px,color:#fff

Real access patterns are close to Zipfian: a small number of items take a large share of requests. Popular products, a trending post, the default configuration blob every request loads, the feature flag set fetched on every page. Those are normal and they are exactly the keys that break the assumption the ring is built on.

Where hot keys come from

The genuinely unavoidable ones are content driven. A product goes viral, a celebrity account gets linked from a news site, a flash sale concentrates traffic on one SKU. Nobody designed that and nobody can prevent it.

The self-inflicted ones are more common and easier to fix. A configuration object cached under one key and read on every single request. A feature flag bundle. A currency rate table. A tenant metadata blob loaded at the start of every handler. These are hot because of how the code was written rather than because of user behaviour, and they are usually the first thing to find when a node is running hot.

The pattern to look for is any key read on a code path that runs for every request, since that key’s request rate is by definition your total request rate.

Detecting it

redis-cli --hotkeys

This samples key access frequency and reports the most requested keys. It needs maxmemory-policy set to an LFU variant, because it reads the frequency counters that policy maintains, and it will tell you so if that is not configured.

MONITOR shows every command in real time, which is useful for a few seconds and is itself a meaningful load on a busy server, so it is a diagnostic of last resort rather than something to leave running.

The external signal that needs no Redis-side tooling is uneven resource use across nodes that hold an even share of keys. One node at three times the CPU or network throughput of its peers, with a balanced keyspace, is a hot key until proven otherwise.

Replicating the key across the ring

The most direct fix is to stop having one key. Write the value under several suffixed keys, and have readers choose one at random.

HOT_KEY_COPIES = 8

def read_carousel():
    # Each replica hashes to a different ring position, so reads spread
    # across nodes instead of converging on one.
    suffix = random.randrange(HOT_KEY_COPIES)
    return cache.get(f"carousel:home:{suffix}")

def write_carousel(value, ttl=300):
    pipe = cache.pipeline()
    for suffix in range(HOT_KEY_COPIES):
        # Jittered TTLs so the replicas do not all expire together
        # and recreate the stampede you were avoiding.
        pipe.set(f"carousel:home:{suffix}", value,
                 ex=int(ttl * random.uniform(0.85, 1.15)))
    pipe.execute()

Reads divide by the number of copies. Writes multiply by it, which is the trade, and it is a good trade specifically because hot keys are almost always read heavy. A key at 140,000 reads per second and 2 writes per minute can absorb being written eight times without anybody noticing.

The jittered TTLs matter. Eight replicas written in the same operation with the same TTL expire in the same millisecond, which turns a hot key into a synchronised stampede across eight nodes at once.

Sharding the value rather than replicating it

Replication works when the value is read whole and rarely written. When the hot key is a large mutable collection, the better move is to split the structure.

A leaderboard sorted set with millions of members, or a set used for membership checks across a huge population, is hot both because of request volume and because every operation against it is linear in its size on a single threaded server.

SHARDS = 64

def shard_for(member):
    return f"leaderboard:{crc32(member.encode()) % SHARDS}"

def add_score(member, score):
    cache.zadd(shard_for(member), {member: score})

Each operation touches one shard, the shards spread across the ring naturally, and no single command scans the whole structure. Reads that need a global view have to query all shards and merge, which is real work and the reason this is worth doing only when the collection is genuinely large.

The local cache that actually fits here

I am generally cautious about in-process caches because each instance becomes an independently stale copy, and a hot key is the case where that caution mostly does not apply.

The reasoning is specific. There are a handful of hot keys, not thousands, so the memory cost is trivial. They are read constantly, so even a two second TTL absorbs an enormous number of requests. And a config blob or carousel is usually the kind of data where two seconds of staleness is invisible.

A two second local cache on a key receiving 140,000 requests per second reduces network traffic for that key by roughly a factor of the request rate times the TTL. The hot node stops being hot. The coherence risk is bounded by two seconds.

This is the fix I would try first, because it requires no key restructuring and no change to how anything is written.

What I would check

Look at per-node CPU and network alongside per-node key count. Even keys with uneven resource use is the signature, and it takes one dashboard panel to make visible.

Then find every cache read that happens on every request. Those keys have your total request rate by construction, and they are usually configuration rather than content, which makes them the cheapest to fix.

The framing that stuck with me is that consistent hashing is a solution to a keyspace problem, and hot keys are a traffic problem. They are adjacent enough that the first gets recommended for the second constantly, and it is the wrong tool with a very convincing name.