Consumer lag is a capacity signal, not an alert threshold

A payments consumer has a lag alert set at 10,000 messages. It has never fired.

The consumer falls behind on a Tuesday afternoon. Lag climbs to 9,400 and sits there for three hours, because that is where the consumer’s throughput and the producer’s happen to balance. Messages are being processed three hours after they were produced. Refunds are landing in customers’ accounts a working day late.

The alert did exactly what it was configured to do. The number it was watching does not mean what anyone assumed.

Messages is the wrong unit

Ten thousand messages is meaningless without a rate.

On a topic doing 5,000 messages per second, ten thousand messages is two seconds of delay and completely fine. On a topic doing 10 per second, it is nearly seventeen minutes. The same threshold is simultaneously too sensitive for one topic and useless for another, and the topics change throughput over the day.

The number that means something is time to drain: how long until the consumer catches up, given how fast it is actually clearing the backlog.

def time_to_drain(lag, consume_rate, produce_rate):
    net = consume_rate - produce_rate
    if net <= 0:
        return None                # not draining. it will not catch up.
    return lag / net

That net <= 0 branch is the important one. A consumer clearing 900 messages per second while 1,000 arrive is not slow, it is diverging, and the lag will grow forever regardless of its current value. Whether the current lag is 500 or 500,000 tells you almost nothing compared to the sign of that subtraction.

The derivative is the alert worth having

Lag is a stock. What predicts an incident is the flow.

flowchart TB
    L["Raw lag: 9,400 messages"] --> Q{"Is it growing?"}
    Q -->|"shrinking"| OK["Recovering. No action.<br/>Even if the number is large."]
    Q -->|"flat"| BAL["Balanced at capacity.<br/>Fragile. Any burst diverges."]
    Q -->|"growing"| DIV["Diverging. Compute when it<br/>hits the retention window."]

    style OK stroke:#4ade80,stroke-width:2px,color:#fff
    style DIV stroke:#ef4444,stroke-width:3px,color:#fff

A large lag that is shrinking is a consumer recovering from a deploy, which is normal and should not page anyone. A small lag that is growing steadily is the beginning of an incident, and the current value is well below any static threshold.

The alert I would actually write compares projected drain time against the retention window, because that is the point at which the failure becomes unrecoverable rather than merely late.

def should_page(lag, consume_rate, produce_rate, retention_seconds):
    drain = time_to_drain(lag, consume_rate, produce_rate)
    if drain is None:
        return True, "diverging: consume rate is below produce rate"
    if drain > retention_seconds * 0.5:
        return True, f"projected drain {drain:.0f}s approaches retention"
    return False, None

Hitting retention is where lateness becomes loss

The reason this matters more than ordinary slowness: exceeding the retention period is not a delay, it is data loss.

The broker deletes segments older than the retention setting regardless of whether anyone read them. A consumer whose committed offset falls off the end of the log finds itself pointing at data that no longer exists, and auto.offset.reset decides what happens next.

latest skips everything that was lost, silently. The consumer resumes at the head, lag drops to zero, every dashboard turns green, and the messages are gone with no error anywhere. I find this genuinely the worst default in the ecosystem, because the recovery looks like a success.

earliest reprocesses from the oldest surviving message, which is safer and can mean replaying days of traffic through a consumer that is already behind, which is its own outage. It is at least loud.

Neither is a good outcome, which is why the useful alert fires when drain time approaches retention rather than when data is already gone.

Per partition, not per topic

Summing lag across a topic hides the failure mode that actually happens.

One partition with a hot key can sit at 400,000 while the other 23 sit near zero. Total lag looks moderate. One consumer is stuck, one customer’s events are hours late, and the aggregate number never crossed a threshold.

The same shape appears when one consumer instance is unhealthy: its assigned partitions grow while the rest of the group is fine.

Maximum lag across partitions is the number to watch. Per-partition detail is what you need during the investigation, because it distinguishes a consumer that is uniformly slow from one where a single partition is doing something different, and those have completely different causes.

Why a consumer falls behind

The causes are worth separating because they respond to different fixes.

Partition count caps parallelism. A consumer group cannot have more useful instances than there are partitions, so a topic with 6 partitions cannot be consumed by more than 6 instances no matter how many pods you run. Scaling out past that does nothing, which surprises people mid-incident.

Slow processing is the common one, and it is usually a downstream dependency rather than the consumer itself. A handler making a database call per message inherits that call’s latency as its per-message cost, which is the same round trip arithmetic that makes unbatched work slow anywhere.

Rebalances stop consumption entirely while they run, and a group that rebalances frequently spends a meaningful fraction of its time not consuming.

Poison messages block a partition when the consumer retries indefinitely, which turns one bad message into unbounded lag on that partition and nowhere else.

What I would put on the dashboard

Maximum per-partition lag rather than the topic sum, so a single stuck partition is visible.

Estimated time to drain, computed from the actual consume and produce rates, since that is the number a human can reason about during an incident.

The trend of that estimate over the last fifteen minutes, because the direction is what distinguishes recovery from divergence.

And the retention window on the same panel, because the whole question during a bad hour is whether you catch up before the log starts deleting itself, and having both numbers side by side answers it immediately.

Frequently Asked Questions

What is consumer lag in Kafka?
Consumer lag is the difference between the offset of the newest message in a partition and the offset the consumer group has committed, so it counts how many messages have been produced but not yet processed. It is reported per partition and summed per topic. On its own the number is hard to interpret, because the same lag represents very different amounts of delay depending on how fast messages are being produced and consumed.
Should you alert on consumer lag?
Not on the raw message count, because the threshold that is correct at one throughput is wrong at another and will either page constantly during normal bursts or stay silent while a consumer falls hours behind on a quiet topic. Alert on estimated time to drain, which is lag divided by the net drain rate, and on the trend of that number. A lag that is large but shrinking needs no page, while a small lag that is growing steadily does.
What happens if consumer lag exceeds the retention period?
The broker deletes messages the consumer has not read yet, so the consumer's committed offset points at data that no longer exists. Kafka's auto.offset.reset setting then decides the outcome: latest silently skips everything that was lost, and earliest reprocesses from the oldest surviving message, which for an at-least-once consumer means a potentially enormous replay. Both are bad, and losing data quietly is usually worse than the replay.

Sources

[ RELATED_LOGS ]

TTFB: -- ms LOAD: -- s PAYLOAD: -- kb