A payments consumer has a lag alert set at 10,000 messages. It has never fired.
The consumer falls behind on a Tuesday afternoon. Lag climbs to 9,400 and sits there for three hours, because that is where the consumer’s throughput and the producer’s happen to balance. Messages are being processed three hours after they were produced. Refunds are landing in customers’ accounts a working day late.
The alert did exactly what it was configured to do. The number it was watching does not mean what anyone assumed.
Messages is the wrong unit
Ten thousand messages is meaningless without a rate.
On a topic doing 5,000 messages per second, ten thousand messages is two seconds of delay and completely fine. On a topic doing 10 per second, it is nearly seventeen minutes. The same threshold is simultaneously too sensitive for one topic and useless for another, and the topics change throughput over the day.
The number that means something is time to drain: how long until the consumer catches up, given how fast it is actually clearing the backlog.
def time_to_drain(lag, consume_rate, produce_rate):
net = consume_rate - produce_rate
if net <= 0:
return None # not draining. it will not catch up.
return lag / net
That net <= 0 branch is the important one. A consumer clearing 900 messages per second while 1,000 arrive is not slow, it is diverging, and the lag will grow forever regardless of its current value. Whether the current lag is 500 or 500,000 tells you almost nothing compared to the sign of that subtraction.
The derivative is the alert worth having
Lag is a stock. What predicts an incident is the flow.
flowchart TB
L["Raw lag: 9,400 messages"] --> Q{"Is it growing?"}
Q -->|"shrinking"| OK["Recovering. No action.<br/>Even if the number is large."]
Q -->|"flat"| BAL["Balanced at capacity.<br/>Fragile. Any burst diverges."]
Q -->|"growing"| DIV["Diverging. Compute when it<br/>hits the retention window."]
style OK stroke:#4ade80,stroke-width:2px,color:#fff
style DIV stroke:#ef4444,stroke-width:3px,color:#fff
A large lag that is shrinking is a consumer recovering from a deploy, which is normal and should not page anyone. A small lag that is growing steadily is the beginning of an incident, and the current value is well below any static threshold.
The alert I would actually write compares projected drain time against the retention window, because that is the point at which the failure becomes unrecoverable rather than merely late.
def should_page(lag, consume_rate, produce_rate, retention_seconds):
drain = time_to_drain(lag, consume_rate, produce_rate)
if drain is None:
return True, "diverging: consume rate is below produce rate"
if drain > retention_seconds * 0.5:
return True, f"projected drain {drain:.0f}s approaches retention"
return False, None
Hitting retention is where lateness becomes loss
The reason this matters more than ordinary slowness: exceeding the retention period is not a delay, it is data loss.
The broker deletes segments older than the retention setting regardless of whether anyone read them. A consumer whose committed offset falls off the end of the log finds itself pointing at data that no longer exists, and auto.offset.reset decides what happens next.
latest skips everything that was lost, silently. The consumer resumes at the head, lag drops to zero, every dashboard turns green, and the messages are gone with no error anywhere. I find this genuinely the worst default in the ecosystem, because the recovery looks like a success.
earliest reprocesses from the oldest surviving message, which is safer and can mean replaying days of traffic through a consumer that is already behind, which is its own outage. It is at least loud.
Neither is a good outcome, which is why the useful alert fires when drain time approaches retention rather than when data is already gone.
Per partition, not per topic
Summing lag across a topic hides the failure mode that actually happens.
One partition with a hot key can sit at 400,000 while the other 23 sit near zero. Total lag looks moderate. One consumer is stuck, one customer’s events are hours late, and the aggregate number never crossed a threshold.
The same shape appears when one consumer instance is unhealthy: its assigned partitions grow while the rest of the group is fine.
Maximum lag across partitions is the number to watch. Per-partition detail is what you need during the investigation, because it distinguishes a consumer that is uniformly slow from one where a single partition is doing something different, and those have completely different causes.
Why a consumer falls behind
The causes are worth separating because they respond to different fixes.
Partition count caps parallelism. A consumer group cannot have more useful instances than there are partitions, so a topic with 6 partitions cannot be consumed by more than 6 instances no matter how many pods you run. Scaling out past that does nothing, which surprises people mid-incident.
Slow processing is the common one, and it is usually a downstream dependency rather than the consumer itself. A handler making a database call per message inherits that call’s latency as its per-message cost, which is the same round trip arithmetic that makes unbatched work slow anywhere.
Rebalances stop consumption entirely while they run, and a group that rebalances frequently spends a meaningful fraction of its time not consuming.
Poison messages block a partition when the consumer retries indefinitely, which turns one bad message into unbounded lag on that partition and nowhere else.
What I would put on the dashboard
Maximum per-partition lag rather than the topic sum, so a single stuck partition is visible.
Estimated time to drain, computed from the actual consume and produce rates, since that is the number a human can reason about during an incident.
The trend of that estimate over the last fifteen minutes, because the direction is what distinguishes recovery from divergence.
And the retention window on the same panel, because the whole question during a bad hour is whether you catch up before the log starts deleting itself, and having both numbers side by side answers it immediately.
// SPONSORSHIP
If this research saved you time or improved your architecture, consider sponsoring my work on GitHub. All sponsorships go directly toward infrastructure and further technical research.
[ Become a Sponsor ]