Distributed systems failure modes
Consistency models, clocks, quorums, and the transactions that span machines. The mechanisms, and the specific ways each one fails in production.
Read in order: later instalments assume the earlier ones. 8 more instalments in the publish queue.
- 01 System Design: Principles for Maintainability, Scalability, and Reliability
Data building blocks, fault tolerance, latency vs response time, scaling strategies, and the operability-simplicity-evolvability triad for durable systems.
- 02 Consistency Models in Azure Cosmos DB: From Strong to Eventual
How Azure Cosmos DB's five consistency levels map onto PACELC tradeoffs, what each level guarantees, and how to choose the right consistency for your workload.
- 03 Clock synchronization is a nightmare: Why Spanner uses TrueTime and the rest of us suffer
Why relying on system clocks causes silent data corruption and how TrueTime solves it for Spanner.
- 04 Instrumenting AI: Multi-master replication and the split brain problem
Why active-active embedding stores feel attractive, and why multi-master replication can trigger a split brain failure in AI infrastructure.
- 05 Instrumenting distributed messaging: The fallacy of exactly once delivery
Why Kafka exactly-once delivery is a coordination tax and why you should build idempotency at the edge.
- 06 Idempotency keys: the API contract that makes retries safe
Why a timeout tells the client nothing about whether the write happened, why storing the key after the work is a race rather than a fix, and why the stored response has to be replayed rather than recomputed.