Engineering Logs
Chronological. High signal. One production failure mode at a time.
65 logs published · new log twice a week
2026.10 [5]
- Idempotency keys: the API contract that makes retries safe
Why a timeout tells the client nothing about whether the write happened, why storing the key after the work is a race rather than a fix, and why the stored response has to be replayed rather than recomputed.
- Immutability is a concurrency strategy, not a style preference
Why a shared mutable object is a lock you forgot to take, why defensive copying is the tax you pay for not deciding, and why the allocation cost is smaller than the synchronisation it removes.
- Thread pool starvation from nested async calls
Why a task that waits on another task in the same pool deadlocks with no error, why the common ForkJoinPool makes this a shared fate across libraries, and why the fix is separate pools rather than bigger ones.
- Optimistic vs pessimistic locking: choose by contention, not preference
Why a version column outperforms row locks until it does not, why the crossover point is measurable rather than a matter of taste, and why retrying an optimistic failure without thinking is how you lose a customer's edit.
- Backpressure is not rate limiting, and treating them the same fills your queues
Why a rate limit set from a number you chose cannot protect a system whose capacity moves, how an unbounded queue converts an overload into an outage, and why the useful signal comes from the consumer rather than the producer.
2026.09 [15]
- Heap looks fine, pod still OOM killed
Why a JVM with a 2GB heap needs considerably more than 2GB of container memory, which native regions grow without appearing on any heap graph, and why the kernel kills you without a Java exception.
- Your p99 is a garbage collector setting
Why a tail latency that no profiler explains is usually a stop the world pause, why allocation rate matters more than heap size, and why a bigger heap can make the tail worse.
- The GIL is not why your Python is slow
Why the interpreter lock is released during almost everything a web service does, what the profile actually says when a service is slow, and the narrow case where the GIL is genuinely the bottleneck.
- Async does not mean parallel, and confusing them costs you a thread pool
Why an event loop with one thread can serve ten thousand connections and still be destroyed by one CPU bound handler, and why marking a method async does nothing for work that never waits.
- Deadlocks are a lock ordering bug, and lock ordering is a design decision
Why two transactions doing the same thing in a different order will eventually stop forever, why the database picking a victim is the good outcome, and why sorting your identifiers before you touch them removes an entire class of incident.
- Hot keys defeat consistent hashing
Why a perfectly balanced ring still sends one celebrity key to one node, why adding cache nodes makes no difference to that node, and the three ways to split a key that a hash function cannot split for you.
- Local cache plus distributed cache: the coherence bill nobody budgets for
Why adding an in-process cache in front of Redis turns one consistency problem into N, why a pub/sub invalidation that nobody retries is a guarantee you do not have, and why the local TTL is the real bound on how wrong you can be.
- Why your Redis is slow: it is single threaded and you sent it KEYS
Why one O(N) command blocks every other client on the server, which everyday commands are secretly linear, and why a p99 spike with flat CPU is almost always somebody scanning the keyspace.
- Cache invalidation is a distributed systems problem in a convenience costume
Why deleting a key after a write is a two phase commit you did not design, how the update-then-invalidate ordering produces permanent staleness, and why invalidating rather than updating is the one decision that reliably helps.
- Write-through, write-behind, write-around: picking wrong costs you consistency or throughput
What each write strategy actually guarantees when the process dies mid-operation, why write-behind is the only one that can lose acknowledged data, and why write-around is the right default more often than people expect.
- Negative caching: the misses cost more than the hits
Why a cache that only stores successes leaves the expensive path completely unprotected, how an attacker turns that into a denial of service with random keys, and why caching absence needs a different TTL than caching presence.
- Jitter is the cheapest reliability fix you are not using
Why independent clients converge on the same instant without any coordination, how retries, cron schedules, health checks and reconnects all self-synchronise, and why adding randomness is a one line fix for a class of outage.
- Cache stampede: how one expired key takes down the database
Why TTL expiry is a synchronised event that sends every concurrent request to the origin at once, why the pileup amplifies itself while the recompute runs, and why jitter is the cheapest fix nobody applies.
- One round trip beats a thousand, and your batch API probably is not batching
Why a 100,000 row insert takes forty minutes instead of twelve seconds, why addBatch does nothing on the wire without a driver flag, and why autocommit turns every row into a durability barrier.
- Your JSONB column became the schemaless disaster you migrated away from
Why a metadata column with no schema accumulates forty shapes in two years, what TOAST does to read cost when you fetch one key from a large document, and how to promote the keys that turned out to be load bearing.
2026.08 [16]
- Foreign keys are not overhead, and the thing you measured was a missing index
Why dropping foreign keys for performance usually removes a cheap index lookup and leaves the expensive sequential scan in place, and what actually replaces the guarantee once it is gone.
- The replica lag you do not measure is the one serving checkout
Why byte lag reads zero on a broken replica, how read-your-writes breaks the moment you add a read replica, and why routing all reads to replicas is a correctness decision rather than a scaling one.
- ADD COLUMN is not always free, and the lock queue is what takes you down
Why a metadata-only migration still took the site offline for forty minutes, how a waiting ALTER blocks every query behind it, and why lock_timeout is the setting that turns a migration into a retry.
- LSM compaction is a background job that will wake you at 3am
Why an LSM engine can saturate a disk with no user traffic, how write stalls turn a background merge into a foreground outage, and why the disk needs headroom you will never see in a capacity graph.
- Connection pool sizing is a queueing theory problem, not a tuning knob
Why Little's law gives you the pool size in one line, why raising it past the knee adds latency without adding throughput, and why twenty pods with twenty connections each is a number nobody decided on.
- Soft deletes are a schema decision that breaks every query you write afterwards
Why deleted_at IS NULL leaks into every index, breaks unique constraints in a way that only shows up when a user re-registers, and moves referential integrity out of the database and into whichever service remembers to filter.
- Covering indexes: the cheap 10x that most schemas leave on the table
Why a matching index still costs you one random read per row, how INCLUDE columns turn that into an index-only scan, and why Postgres will quietly keep hitting the heap anyway if the visibility map is stale.
- Why your index is not being used, and why the planner is usually right
The index exists and EXPLAIN still says sequential scan. A field guide to sargability, stale statistics, the leftmost prefix rule, and the cost settings that make a planner reject an index it should have chosen.
- fsync is the only thing between you and data loss, and it is slower than you think
Why a successful write() means nothing, what fsync actually costs in the cloud, and why synchronous_commit off and fsync off are not the same knob despite being discussed as if they were.
- Your ORM issued 400 queries and the p99 looked fine until it didn't
Why N+1 queries are invisible to every database side metric you own, how a getter call becomes a network round trip, and why the fix that looks obvious produces a cartesian product.
- Decoding isolation levels: I built a toy DB to force dirty reads and phantom reads
Why the ANSI isolation table does not describe your database, what a phantom read actually is at the index level, and why snapshot isolation still lets two correct transactions corrupt each other.
- Implementing LFU Cache in O(1) Time: A Hands-on Breakdown
LFU evicts the least popular key, not the oldest. The O(1) version needs two hash maps and linked lists per frequency bucket.
- Boredom is a Signal to Find a Harder Problem: When Your CRUD API Stops Teaching You
Every backend engineer builds the same todo API. Boredom hits when CRUD stops surprising you. That is the signal to move up the stack.
- I Rebuilt a Minimal BitTorrent Client in Go to Understand Peer-to-Peer Choking Algorithms
BitTorrent choking is bandwidth scheduling disguised as game theory. Rebuilding a minimal client in Go makes the TCP, buffer, and fairness costs visible.
- Promotions are Proactive: The 3P Framework for Pitching Your Next Level
Promotion is not a reward for busy sprints. It is recognition that you already operate at the next level's blast radius. The 3P framework turns that into a pitch.
- Glue Work is the New System Design: Why Alignment is the Premium Skill in the Age of AI
AI makes code cheap. The expensive part is getting five teams, three regions, and two data stores to agree on what actually ships.
2026.05 [6]
- The AI code review bottleneck: When writing code is 5x faster, but reviewing is 2x slower
Why AI speeds up code production while reviewers pay the latency tax of architecture and network validation.
- Stop Writing RFCs Like Mystery Novels: Why the Best Design Docs Start with the Conclusion
Why architecture docs should lead with the conclusion and why the rest of the content belongs at the bottom.
- Clock synchronization is a nightmare: Why Spanner uses TrueTime and the rest of us suffer
Why relying on system clocks causes silent data corruption and how TrueTime solves it for Spanner.
- Scaling a distributed cache: Why consistent hashing is mandatory
Why modulo-based cache sharding fails in production and how consistent hashing with virtual nodes protects your database.
- Instrumenting distributed messaging: The fallacy of exactly once delivery
Why Kafka exactly-once delivery is a coordination tax and why you should build idempotency at the edge.
- Instrumenting AI: Multi-master replication and the split brain problem
Why active-active embedding stores feel attractive, and why multi-master replication can trigger a split brain failure in AI infrastructure.
2026.03 [23]
- Instrumenting AI Agents: Why the Apology Metric Is a First Class Reliability Signal
Track apology phrases as a first class SLO for AI agents: spikes reveal context starvation, timeout dropouts, and payload truncation across data boundaries.
- The Expensive Cosplay of Local Models: True 3 AM Operational Cost of Hosting Llama-3
Self-hosted Llama-70B looks cheap until VRAM, KV cache, and HBM bandwidth cap throughput. TCO is idle GPUs, batching latency, and ML infra on call.
- AI agents break connection pooling by holding the slot while they think
Agents keep pooled DB connections open for LLM inference, exhausting pools and evicting buffer cache. Decouple reasoning from data, route agents to replicas, and never hold a connection across an inference call.
- Thread-per-Core Architecture: Why Extra Threads Eventually Destroy Throughput
Oversized thread pools stall: timeslicing, context switches, cache thrashing. Thread-per-core, CPU pinning, and async I/O match physical cores.
- Branch Prediction: Why an if Inside a Hot Loop Costs Milliseconds
How CPU pipelining and branch predictors work, why mispredictions flush the pipeline, and how sorting, branchless code, and loop unrolling help.
- CPU Caches and Spatial Locality: Why an Array is 3x Faster Than a Linked List for the Exact Same Big-O Complexity
Why arrays are faster than linked lists on real CPUs: cache lines, spatial locality, hardware prefetchers, and pointer chasing.
- RSS vs VSZ in Virtual Memory: What the OOM Killer Actually Counts
malloc() grows VSZ, page faults commit RSS, and the Linux OOM killer only counts RSS. How virtual memory and lazy allocation decide which process dies.
- Cuckoo Filters: Cache-Friendly Membership Checks With Deletions
How Cuckoo filters work: fingerprints, two-bucket lookups, kick-out insertions, why they stay cache-friendly, and the real tradeoff of insertion failure.
- Bloom Filters vs Counting Bloom Filters: When Deletions Kill Performance
Why counting (deletable) Bloom filters often lose in production: cache misses, random memory access, and better alternatives like hash tables or Cuckoo filters.
- Pagination at Scale: Why OFFSET and SKIP Will Eventually Break Your API
Why OFFSET/SKIP pagination degrades linearly with depth, how cursor-based pagination keeps latency flat, and when to switch before production bites back.
- The RUM Conjecture: You Cannot Optimize Reads, Updates, and Memory at Once
How the RUM Conjecture explains real-world database trade-offs between read latency, write throughput, and memory overhead across B-Trees, LSM-Trees, and hash indexes.
- Why UUID Primary Keys Quietly Destroy Database Performance
How random UUID primary keys break clustered indexes, cause page splits and buffer pool churn, and what to use instead for mechanically sympathetic database design.
- Designing Resilient APIs: Failure-Handling Patterns for Distributed Systems
Practical resilience patterns for distributed APIs: fail-fast, retries with backoff, circuit breakers, bulkheads, fallbacks, rate limiting, failover, and observability.
- Microservices Deep Dive: Architecting for Scalability and Resilience
How to design, operate, and scale microservices: core principles, when to use them, key patterns, and how to manage complexity in distributed systems.
- Consistency Models in Azure Cosmos DB: From Strong to Eventual
How Azure Cosmos DB's five consistency levels map onto PACELC tradeoffs, what each level guarantees, and how to choose the right consistency for your workload.
- Zero Trust Architecture: From Perimeter Walls to "Never Trust, Always Verify"
How Zero Trust Architecture replaces perimeter-based security: core principles, differences from traditional models and ZTNA, enabling technologies, and real-world implementations.
- Mastering Event-Driven Architecture with Apache Kafka
How to design scalable, resilient systems using event-driven architecture and Apache Kafka for high-throughput, real-time data processing.
- Transitioning from REST to gRPC: System Design and Tradeoffs
How gRPC changes API design versus REST: protocol model, protobuf schemas, service interfaces, streaming patterns, and when gRPC or REST is the right architectural choice.
- HTTP/2 System Design: How It Fixes HTTP/1.1
Deep dive into HTTP/2: why HTTP/1.1 hit scaling limits, how multiplexing, server push, binary framing, and prioritization work, and why it matters for web performance.
- Shazam finds songs by voting on time offsets, not by comparing audio
How Shazam's fingerprinting works according to the published Wang 2003 paper: constellation maps, combinatorial peak pairing into 32-bit hashes, and the offset histogram that turns song matching into counting.
- System Migration: Minimize Downtime, Maximize Efficiency
A practical blueprint for system migration: isolated env, sync/async flows, bridge layer, traffic leakage, backup sync, and monitoring.
- System Design: Principles for Maintainability, Scalability, and Reliability
Data building blocks, fault tolerance, latency vs response time, scaling strategies, and the operability-simplicity-evolvability triad for durable systems.
- Your week is a queueing system, and you are running it at 100% utilization
Time management for software engineers, treated as the scheduling problem it is: utilization and latency, context switches as cache eviction, Little's law for work in progress, and priority inversion in your calendar.