Distributed Systems
18 engineering logs on Distributed Systems. Newest first.
- Idempotency keys: the API contract that makes retries safe
Why a timeout tells the client nothing about whether the write happened, why storing the key after the work is a race rather than a fix, and why the stored response has to be replayed rather than recomputed.
- Backpressure is not rate limiting, and treating them the same fills your queues
Why a rate limit set from a number you chose cannot protect a system whose capacity moves, how an unbounded queue converts an overload into an outage, and why the useful signal comes from the consumer rather than the producer.
- Local cache plus distributed cache: the coherence bill nobody budgets for
Why adding an in-process cache in front of Redis turns one consistency problem into N, why a pub/sub invalidation that nobody retries is a guarantee you do not have, and why the local TTL is the real bound on how wrong you can be.
- Cache invalidation is a distributed systems problem in a convenience costume
Why deleting a key after a write is a two phase commit you did not design, how the update-then-invalidate ordering produces permanent staleness, and why invalidating rather than updating is the one decision that reliably helps.
- Jitter is the cheapest reliability fix you are not using
Why independent clients converge on the same instant without any coordination, how retries, cron schedules, health checks and reconnects all self-synchronise, and why adding randomness is a one line fix for a class of outage.
- Cache stampede: how one expired key takes down the database
Why TTL expiry is a synchronised event that sends every concurrent request to the origin at once, why the pileup amplifies itself while the recompute runs, and why jitter is the cheapest fix nobody applies.
- I Rebuilt a Minimal BitTorrent Client in Go to Understand Peer-to-Peer Choking Algorithms
BitTorrent choking is bandwidth scheduling disguised as game theory. Rebuilding a minimal client in Go makes the TCP, buffer, and fairness costs visible.
- Glue Work is the New System Design: Why Alignment is the Premium Skill in the Age of AI
AI makes code cheap. The expensive part is getting five teams, three regions, and two data stores to agree on what actually ships.
- Clock synchronization is a nightmare: Why Spanner uses TrueTime and the rest of us suffer
Why relying on system clocks causes silent data corruption and how TrueTime solves it for Spanner.
- Scaling a distributed cache: Why consistent hashing is mandatory
Why modulo-based cache sharding fails in production and how consistent hashing with virtual nodes protects your database.
- Instrumenting distributed messaging: The fallacy of exactly once delivery
Why Kafka exactly-once delivery is a coordination tax and why you should build idempotency at the edge.
- Instrumenting AI: Multi-master replication and the split brain problem
Why active-active embedding stores feel attractive, and why multi-master replication can trigger a split brain failure in AI infrastructure.
- Instrumenting AI Agents: Why the Apology Metric Is a First Class Reliability Signal
Track apology phrases as a first class SLO for AI agents: spikes reveal context starvation, timeout dropouts, and payload truncation across data boundaries.
- Bloom Filters vs Counting Bloom Filters: When Deletions Kill Performance
Why counting (deletable) Bloom filters often lose in production: cache misses, random memory access, and better alternatives like hash tables or Cuckoo filters.
- Designing Resilient APIs: Failure-Handling Patterns for Distributed Systems
Practical resilience patterns for distributed APIs: fail-fast, retries with backoff, circuit breakers, bulkheads, fallbacks, rate limiting, failover, and observability.
- Microservices Deep Dive: Architecting for Scalability and Resilience
How to design, operate, and scale microservices: core principles, when to use them, key patterns, and how to manage complexity in distributed systems.
- Consistency Models in Azure Cosmos DB: From Strong to Eventual
How Azure Cosmos DB's five consistency levels map onto PACELC tradeoffs, what each level guarantees, and how to choose the right consistency for your workload.
- Mastering Event-Driven Architecture with Apache Kafka
How to design scalable, resilient systems using event-driven architecture and Apache Kafka for high-throughput, real-time data processing.