Retry storms: how the resilience layer became the outage

A payments dependency gets slow. Not down, just slow: p99 goes from 80 milliseconds to 3 seconds because of a bad query plan after a statistics refresh.

The calling service has a 1 second timeout and retries twice. The gateway in front of it has a 5 second timeout and retries twice. Both configurations were added at different times by different people and both looked reasonable in review.

Load on the payments service goes to nine times normal. It stops responding entirely. The query plan gets fixed twelve minutes later and the service stays down, because nine times normal traffic is still arriving and it cannot serve any of it fast enough to stop the retries.

Retries multiply, and nobody adds them up

Each layer knows only its own policy. The system experiences the product.

flowchart TB
    U["1 user request"] --> G["Gateway: up to 3 attempts"]
    G --> S["Service: up to 3 attempts each"]
    S --> D[("Database sees<br/>up to 9 calls")]

    style D stroke:#ef4444,stroke-width:3px,color:#fff

Three hops each retrying twice is 27. That number never appears in any configuration file, any design document, or any code review, because it does not exist at any single layer.

The multiplier is bad enough on its own. What makes it an outage rather than a spike is timing: the amplification is maximal precisely when the dependency is least able to serve, since retries only fire on failure. A healthy system has a multiplier of 1 and a degraded one has a multiplier of 9, which is the opposite of what you would design if you were choosing when to apply load.

The part that prevents recovery

The self-sustaining property is what turns this from a bad afternoon into a long one.

The dependency is saturated. Requests time out. Each timeout generates retries. Those retries keep it saturated. When the original cause is fixed, the load is still there, and the service cannot get ahead of it because every request it fails to serve in time generates more.

Recovery requires load to drop below capacity, and load will not drop until requests succeed, and requests cannot succeed while load exceeds capacity. That is a stable state and systems can sit in it for a long time.

The usual escape is manual: shed traffic at the edge, let the dependency drain, then reopen. Doing it automatically is what circuit breakers are for, and it is worth being precise about why they help. A breaker converts the failure mode from “keep hammering” to “fail fast for a while, then probe with a small number of requests”, which gives the dependency an interval of survivable load rather than a wall.

Timeouts have to shrink as you go down

There is a configuration mistake here that is independently common and makes everything worse.

If an inner call has a longer timeout than the outer one, the outer times out first and retries while the inner attempt is still running. The work is duplicated, the original is orphaned and still consuming capacity, and the caller has no way to know.

Gateway timeout:   5s, 3 attempts
Service timeout:   1s, 3 attempts     <- fits inside, correct direction
Database timeout: 300ms, 2 attempts   <- fits inside, correct direction

The rule is that each layer’s total time budget, including its retries, must fit inside its caller’s timeout. That means budgets shrink as you go inward, and it means retries eat into the budget rather than sitting outside it.

A cleaner formulation is a deadline that travels with the request rather than a per-hop timeout. Each hop computes its remaining time from the deadline and refuses to start work it cannot finish, which removes the possibility of an inner call outliving the caller that wanted it. gRPC propagates deadlines natively and it is one of its genuinely useful properties.

Retry budgets bound what limits cannot

Per-request limits do not bound system load, because every request has its own allowance. Ten thousand failing requests with a limit of three attempts is thirty thousand calls, and the limit was respected throughout.

A budget caps retries as a fraction of traffic across a rolling window.

// Retries are allowed only while they stay under a share of total traffic.
// Under normal conditions this never engages. During a widespread failure
// it engages immediately and caps the amplification at 1.1x rather than 3x.
public boolean tryRetry() {
    long total = requestCounter.count();
    long retries = retryCounter.count();
    if (retries > total * 0.10) {
        metrics.increment("retry.budget.exhausted");
        return false;
    }
    retryCounter.increment();
    return true;
}

Ten percent is the figure used in Google’s SRE book and it is a reasonable starting point. The property that makes it work is that it distinguishes the two situations that a per-request limit cannot: one request failing while everything else is fine (retry, definitely) and everything failing at once (retrying is pointless and harmful).

Envoy and the gRPC ecosystem implement this directly, and it is worth using rather than reimplementing.

Retry only what is worth retrying

Not every failure benefits from another attempt, and retrying the wrong ones is pure amplification.

A 400 or a 422 will fail identically every time. A 401 will too, unless the retry includes a token refresh. A 429 has already told you to slow down and should be honoured rather than retried immediately, ideally respecting Retry-After.

The retryable set is roughly connection errors, timeouts, 502, 503, 504, and 429 with backoff. Everything else is a waste of the budget.

Idempotency governs whether retrying is even safe. A read is safe to repeat. A charge is not, unless the operation carries an idempotency key that lets the server recognise the retry. Retrying a non-idempotent write on a timeout is how one payment becomes three.

And every backoff needs jitter, because synchronised retries preserve the spike shape even while reducing the rate, which is the alignment problem that shows up wherever a fixed interval exists.

What I would inventory

Count the layers between the edge and each critical dependency, and multiply their retry counts. That number is what the dependency sees during a bad minute, and it is usually the first time anyone has calculated it.

Decide which single layer owns retries for a given call path. Retrying at every layer is the default because each one seems locally sensible, and the fix is usually to retry at exactly one place, ideally the one closest to the failure with the best information about whether another attempt could help.

Check that timeouts shrink inward, and prefer a propagated deadline if the transport supports it.

Then add a budget, because it is the only mechanism in this list that bounds the total rather than the individual, and the total is what takes the dependency down.