Distributed systems failure modes
Consistency models, clocks, quorums, and the transactions that span machines. The mechanisms, and the specific ways each one fails in production.
Read in order: later instalments assume the earlier ones. 4 more instalments in the publish queue.
- 01 System Design: Principles for Maintainability, Scalability, and Reliability
Data building blocks, fault tolerance, latency vs response time, scaling strategies, and the operability-simplicity-evolvability triad for durable systems.
- 02 Consistency Models in Azure Cosmos DB: From Strong to Eventual
How Azure Cosmos DB's five consistency levels map onto PACELC tradeoffs, what each level guarantees, and how to choose the right consistency for your workload.
- 03 Clock synchronization is a nightmare: Why Spanner uses TrueTime and the rest of us suffer
Why relying on system clocks causes silent data corruption and how TrueTime solves it for Spanner.
- 04 R plus W greater than N is the arithmetic that decides your consistency
Why quorum overlap is the whole guarantee, what W equals one actually costs, and why the inequality holding does not save you from concurrent writes that no read can order.
- 05 Leader election is easy, failover is where systems die
Why a leader that lost its lease does not know it, how a paused process wakes up and keeps writing as though it is still in charge, and why a fencing token is the only thing that reliably stops it.
- 06 Instrumenting AI: Multi-master replication and the split brain problem
Why active-active embedding stores feel attractive, and why multi-master replication can trigger a split brain failure in AI infrastructure.
- 07 Instrumenting distributed messaging: The fallacy of exactly once delivery
Why Kafka exactly-once delivery is a coordination tax and why you should build idempotency at the edge.
- 08 Idempotency keys: the API contract that makes retries safe
Why a timeout tells the client nothing about whether the write happened, why storing the key after the work is a race rather than a fix, and why the stored response has to be replayed rather than recomputed.
- 09 Two-phase commit is not dead, it is just expensive and you should know the price
What the coordinator actually guarantees, why a participant that has voted yes cannot do anything except wait, and why the blocking window rather than the latency is the reason people abandoned it.
- 10 The saga pattern is compensating transactions with better marketing
Why every saga step is visible to the outside world before the saga finishes, why a compensation is a new business action rather than a rollback, and why the hard part is the steps that cannot be undone.