The replica lag you do not measure is the one serving checkout

A customer updates their shipping address, sees a confirmation, refreshes the page, and the old address is back. They update it again. Same result. They contact support, who cannot reproduce it, because support is hitting a different replica.

Nothing is broken. The write committed. The read was correct for the node that served it. Replication was 400 milliseconds behind and the page refresh took 200.

This class of bug arrives the week someone adds read replicas to scale reads, and it arrives without any code changing, because the code never said which node it wanted.

The metric that reads zero while everything is wrong

Most replication dashboards show byte lag: the difference between the log position the primary has written and the position the replica has applied.

SELECT client_addr,
       pg_wal_lsn_diff(sent_lsn, replay_lsn) AS replay_lag_bytes
FROM pg_stat_replication;

That number is useful under load and dangerously reassuring otherwise. On an idle primary it reads zero, because nothing new was written, and it reads zero whether the replica is perfectly current or has been disconnected since Tuesday. The quietest hours produce the healthiest looking graph.

Time lag does not have this failure mode:

-- Run on the replica. Keeps growing while replication is stuck.
SELECT now() - pg_last_xact_replay_timestamp() AS replication_delay;

This measures how long it has been since the replica applied anything from the primary, which keeps climbing during an outage regardless of write volume. There is one wrinkle: on a genuinely idle primary it also climbs, because no transaction has arrived to replay. Postgres handles that with a keepalive, and the practical answer is to alert on time lag while checking the replica is still connected in pg_stat_replication, so silence from an idle system is distinguishable from silence from a dead one.

Where lag actually comes from

Replay on a Postgres streaming replica is single threaded. The primary can commit transactions across dozens of backends in parallel and the replica applies them in order, one at a time. A write burst that the primary absorbs comfortably can put the replica behind, and the replica will stay behind until the burst ends.

Long-running queries on the replica make it worse in a way that surprises people. If a query on the replica is reading rows that the incoming stream wants to remove, the replay has to either wait or cancel the query. Postgres decides with max_standby_streaming_delay, which defaults to 30 seconds: replay waits up to that long, then cancels the conflicting query.

That default produces two different complaints depending on which way you tune it. Raise it and reporting queries succeed while lag grows. Lower it and lag stays small while analysts see ERROR: canceling statement due to conflict with recovery. Setting hot_standby_feedback = on tells the primary to hold back vacuum for the replica’s oldest query, which removes the conflict and moves the cost to the primary as table bloat.

There is no setting that makes this free. Running analytics on a replica means choosing where the pain lands.

Adding a replica silently changes your consistency model

The part I think is genuinely underappreciated: putting a read replica behind the same connection string is a consistency change disguised as an infrastructure change.

A single primary gives you read-your-writes for free. Nobody wrote code to get it. The moment reads can land on an asynchronous replica, that guarantee is gone from every flow in the application at once, and nothing in the code indicates which flows depended on it.

sequenceDiagram
    participant C as Client
    participant P as Primary
    participant R as Replica

    C->>P: UPDATE address
    P-->>C: 200 OK, committed
    P->>R: stream WAL (in flight)
    C->>R: SELECT address
    R-->>C: old value
    Note over C,R: Both operations are correct.<br/>The guarantee that connected them is gone.

This is exactly the distinction that Cosmos DB exposes as an explicit consistency level rather than an accident of topology. Session consistency, where a client always sees its own writes, is a named guarantee you can choose there. With Postgres read replicas it is a property you had, lost, and have to rebuild in application code.

Rebuilding the guarantee

Route after write is the blunt version. After a write, pin that session’s reads to the primary for a few seconds.

// Crude but effective: a short window where this user's reads go to the primary.
public void updateAddress(Long userId, Address address) {
    primaryTemplate.update(...);
    recentWriters.put(userId, Instant.now());   // TTL of a few seconds
}

public DataSource routeFor(Long userId) {
    Instant wroteAt = recentWriters.get(userId);
    boolean recent = wroteAt != null
            && Duration.between(wroteAt, Instant.now()).getSeconds() < 5;
    return recent ? primary : replica;
}

It works, and the window is a guess. Set it too short and the bug comes back under load, which is the worst case because load is when lag is highest. Set it too long and the primary serves reads you added replicas to offload.

Waiting on a log position is the precise version. Capture the write’s position at commit and require the replica to have replayed at least that far.

// On the primary, immediately after commit.
String lsn = jdbc.queryForObject("SELECT pg_current_wal_insert_lsn()::text", String.class);

// On the replica, before serving a read that must include that write.
Boolean caughtUp = replicaJdbc.queryForObject(
        "SELECT pg_last_wal_replay_lsn() >= ?::pg_lsn", Boolean.class, lsn);

This is correct rather than approximate. The cost is plumbing: the position has to travel with the user’s session, and every read path that cares has to check it and decide what to do when the answer is no. Waiting is one option and falling back to the primary is usually better.

Classifying reads is the approach I would reach for first, because it forces the question into the open. Reads that must reflect a recent write go to the primary by declaration. Everything else goes to replicas. The classification lives in the code where the read happens, so the next engineer can see it.

The category that catches people is reads that feed writes. A stock check before a reservation, a balance read before a debit, an idempotency key lookup before processing. Those look like reads and they are the read half of a read-modify-write, and serving them from a lagging replica means deciding based on stale state. That is the path where lag stops being a UX annoyance and starts overselling inventory, which puts it in the same family as an isolation level that permits a decision based on state that changed underneath you.

What I would set up

Alert on time since last replay measured from the replica, not on byte lag from the primary, and pair it with a check that the replication connection exists at all.

Put lag on the same dashboard as the endpoints that read from replicas, because the interesting question during an incident is not how far behind the replica is but which user-facing flows were reading from it at the time.

Write down which reads are allowed to be stale. Not in a design doc, in the code, at the point where the datasource is chosen. Every team I have seen do this discovers at least one read in the write path that nobody had thought of as a read.