Idempotency keys: the API contract that makes retries safe

A payment request times out after 30 seconds. The client sees a socket timeout and does what any reasonable client does: it retries.

The first request had succeeded. The card was charged, the order was created, and the response was on its way back when the connection dropped. The retry charges the card again.

The client did nothing wrong. From its side, a timeout and a failure are indistinguishable, and the only safe assumption is that the request might not have happened. The server is the only party that knows, and nothing in the protocol lets it say so.

The client cannot know, so the server has to remember

This is worth stating plainly because it explains why client side fixes do not work.

A timeout means the response did not arrive. It says nothing about whether the request was processed. The write may have committed and the response been lost in the network, or the request may never have reached the server at all.

sequenceDiagram
    participant C as Client
    participant S as Server
    participant DB as Database

    C->>S: POST /payments
    S->>DB: INSERT payment, charge card
    DB-->>S: committed
    S--xC: response lost
    Note over C: timeout. Did it happen?<br/>No way to tell from here.
    C->>S: POST /payments (retry)
    S->>DB: INSERT payment, charge card
    Note over DB: charged twice

The suggestion that the client should check before retrying does not survive contact with this. The check is another request that can also time out, and there is usually no way to identify the resource without the id the server generated. Even with a way to check, the check and the retry are two operations, and a resource created between them is missed.

Deduplication has to happen on the server, atomically with the write.

The contract

The client generates a unique key per logical operation and sends it as a header.

POST /payments
Idempotency-Key: 8f3d2c1e-4b7a-49f0-9c2e-1d5a7b3f8e42
Content-Type: application/json

The key identifies the operation, not the request. A retry of the same logical payment reuses the key. A genuinely new payment gets a new one. That distinction is the client’s responsibility and it is the part clients most often get wrong, usually by generating the key inside the retry loop rather than outside it.

The server then guarantees that the operation happens at most once, and that any repeat receives the original outcome.

Reserve the key before doing the work

This is the part that separates an implementation that works from one that works most of the time.

// Broken under concurrency. Two simultaneous copies both find nothing,
// both proceed, both charge.
public PaymentResponse create(String key, PaymentRequest req) {
    Optional<Record> existing = repo.findByKey(key);
    if (existing.isPresent()) {
        return existing.get().response();
    }
    PaymentResponse resp = chargeCard(req);
    repo.save(new Record(key, resp));      // too late
    return resp;
}

The check and the write are not atomic, so two copies of the same retry arriving together both pass the check. That is a narrow window and retries are exactly the situation that produces simultaneous duplicates, because a client timing out at the same moment as a load balancer retrying is not unusual.

The fix is to make the key insert the thing that arbitrates, using a unique constraint.

CREATE TABLE idempotency_keys (
    key            text PRIMARY KEY,
    request_hash   text        NOT NULL,
    status         text        NOT NULL,   -- in_progress | completed
    response_body  jsonb,
    response_code  int,
    created_at     timestamptz NOT NULL DEFAULT now()
);

public PaymentResponse create(String key, PaymentRequest req) {
    try {
        // Claim the key first. A concurrent duplicate loses here.
        repo.insertInProgress(key, hash(req));
    } catch (DuplicateKeyException e) {
        Record existing = repo.findByKey(key).orElseThrow();

        if (!existing.requestHash().equals(hash(req))) {
            // Same key, different body. The client has a bug and we
            // should say so rather than silently returning the old result.
            throw new IdempotencyKeyReusedException();
        }
        if (existing.status() == IN_PROGRESS) {
            throw new RequestInFlightException();   // 409, retry shortly
        }
        return existing.response();                 // replay
    }

    PaymentResponse resp = chargeCard(req);
    repo.markCompleted(key, resp);
    return resp;
}

Three details in there earn their place.

The request hash catches a client reusing a key for a different payload, which is a client bug and should produce an error rather than silently returning an unrelated result. Stripe returns a 422 for this and I think that is the right call.

The in-progress state handles a duplicate arriving while the first is still running. Returning 409 and asking the client to retry shortly is better than blocking, because blocking holds a connection for the duration of the original request.

Replaying the stored response rather than recomputing it matters more than it looks, which is the next section.

Replay, do not recompute

A tempting shortcut is to store only the created resource id and re-serialise the current state on a repeat.

That returns a different answer than the first caller received. If the payment was subsequently refunded, a retry of the original request now returns a refunded payment, which is not what the original operation produced. The client, which is retrying because it never saw the first response, treats this as the outcome of its request.

Storing the response body and status code verbatim keeps the guarantee honest: the same request returns the same answer, regardless of what happened afterwards.

Making it atomic with the work

The version above still has a gap. If the process dies between chargeCard and markCompleted, the key is stuck in progress and the charge happened.

Where the side effect is a database write, put both in one transaction and the gap closes:

@Transactional
public OrderResponse create(String key, OrderRequest req) {
    repo.insertInProgress(key, hash(req));   // unique constraint arbitrates
    Order order = orderRepo.save(toOrder(req));
    OrderResponse resp = toResponse(order);
    repo.markCompleted(key, resp);
    return resp;                              // all commits together, or none
}

Where the side effect is external, such as a card charge, no local transaction covers it, and the answer is to pass the idempotency key through to the provider. Stripe, Adyen and most payment APIs accept one for exactly this reason. Your key becomes their key, and the deduplication happens at the boundary where the money moves.

That chaining is the general pattern for external effects, and it is the same reasoning as building idempotency at the consumer rather than trusting a broker to deliver exactly once. Nobody can give you exactly once end to end. You get at least once plus deduplication, applied wherever the effect actually occurs.

Expiry, and what it costs

Keys cannot be kept forever. Stripe retains them for 24 hours, which is a reasonable default because it comfortably exceeds any sane client retry window.

The expiry is a real weakening of the guarantee, and it is worth being explicit that a retry arriving after expiry will be treated as a new operation. For a client retrying over seconds or minutes that is irrelevant. For a job that retries a failed batch a week later it is not, and the answer there is a natural key in the domain rather than an idempotency key: a unique constraint on (order_id, invoice_period) protects against duplication for as long as the data exists.

Which endpoints need one

Any non-idempotent operation reachable over a network that a client might retry, which in practice means most POST endpoints that create something or move money.

PUT and DELETE are idempotent by their HTTP definition and usually need nothing extra, provided the implementation actually honours that. A PUT implemented as an append rather than a replace is not idempotent regardless of the verb.

GET needs nothing.

The one I would flag as commonly missed is any endpoint that triggers a side effect elsewhere: sending an email, publishing an event, calling a partner API. Those look read-only from the database’s perspective and are absolutely not safe to repeat, and they rarely have a natural key to fall back on.