Reliability
9 engineering logs on Reliability. Newest first.
- Idempotency keys: the API contract that makes retries safe
Why a timeout tells the client nothing about whether the write happened, why storing the key after the work is a race rather than a fix, and why the stored response has to be replayed rather than recomputed.
- Thread pool starvation from nested async calls
Why a task that waits on another task in the same pool deadlocks with no error, why the common ForkJoinPool makes this a shared fate across libraries, and why the fix is separate pools rather than bigger ones.
- Backpressure is not rate limiting, and treating them the same fills your queues
Why a rate limit set from a number you chose cannot protect a system whose capacity moves, how an unbounded queue converts an overload into an outage, and why the useful signal comes from the consumer rather than the producer.
- Negative caching: the misses cost more than the hits
Why a cache that only stores successes leaves the expensive path completely unprotected, how an attacker turns that into a denial of service with random keys, and why caching absence needs a different TTL than caching presence.
- Jitter is the cheapest reliability fix you are not using
Why independent clients converge on the same instant without any coordination, how retries, cron schedules, health checks and reconnects all self-synchronise, and why adding randomness is a one line fix for a class of outage.
- Cache stampede: how one expired key takes down the database
Why TTL expiry is a synchronised event that sends every concurrent request to the origin at once, why the pileup amplifies itself while the recompute runs, and why jitter is the cheapest fix nobody applies.
- Designing Resilient APIs: Failure-Handling Patterns for Distributed Systems
Practical resilience patterns for distributed APIs: fail-fast, retries with backoff, circuit breakers, bulkheads, fallbacks, rate limiting, failover, and observability.
- System Migration: Minimize Downtime, Maximize Efficiency
A practical blueprint for system migration: isolated env, sync/async flows, bridge layer, traffic leakage, backup sync, and monitoring.
- System Design: Principles for Maintainability, Scalability, and Reliability
Data building blocks, fault tolerance, latency vs response time, scaling strategies, and the operability-simplicity-evolvability triad for durable systems.