Engineering Logs
Chronological. High signal. One production failure mode at a time.
46 logs published · new log every two days
2026.09 [1]
2026.08 [16]
- Foreign keys are not overhead, and the thing you measured was a missing index
Why dropping foreign keys for performance usually removes a cheap index lookup and leaves the expensive sequential scan in place, and what actually replaces the guarantee once it is gone.
- The replica lag you do not measure is the one serving checkout
Why byte lag reads zero on a broken replica, how read-your-writes breaks the moment you add a read replica, and why routing all reads to replicas is a correctness decision rather than a scaling one.
- ADD COLUMN is not always free, and the lock queue is what takes you down
Why a metadata-only migration still took the site offline for forty minutes, how a waiting ALTER blocks every query behind it, and why lock_timeout is the setting that turns a migration into a retry.
- LSM compaction is a background job that will wake you at 3am
Why an LSM engine can saturate a disk with no user traffic, how write stalls turn a background merge into a foreground outage, and why the disk needs headroom you will never see in a capacity graph.
- Connection pool sizing is a queueing theory problem, not a tuning knob
Why Little's law gives you the pool size in one line, why raising it past the knee adds latency without adding throughput, and why twenty pods with twenty connections each is a number nobody decided on.
- Soft deletes are a schema decision that breaks every query you write afterwards
Why deleted_at IS NULL leaks into every index, breaks unique constraints in a way that only shows up when a user re-registers, and moves referential integrity out of the database and into whichever service remembers to filter.
- Covering indexes: the cheap 10x that most schemas leave on the table
Why a matching index still costs you one random read per row, how INCLUDE columns turn that into an index-only scan, and why Postgres will quietly keep hitting the heap anyway if the visibility map is stale.
- Why your index is not being used, and why the planner is usually right
The index exists and EXPLAIN still says sequential scan. A field guide to sargability, stale statistics, the leftmost prefix rule, and the cost settings that make a planner reject an index it should have chosen.
- fsync is the only thing between you and data loss, and it is slower than you think
Why a successful write() means nothing, what fsync actually costs in the cloud, and why synchronous_commit off and fsync off are not the same knob despite being discussed as if they were.
- Your ORM issued 400 queries and the p99 looked fine until it didn't
Why N+1 queries are invisible to every database side metric you own, how a getter call becomes a network round trip, and why the fix that looks obvious produces a cartesian product.
- Decoding isolation levels: I built a toy DB to force dirty reads and phantom reads
Why the ANSI isolation table does not describe your database, what a phantom read actually is at the index level, and why snapshot isolation still lets two correct transactions corrupt each other.
- Implementing LFU Cache in O(1) Time: A Hands-on Breakdown
LFU evicts the least popular key, not the oldest. The O(1) version needs two hash maps and linked lists per frequency bucket.
- Boredom is a Signal to Find a Harder Problem: When Your CRUD API Stops Teaching You
Every backend engineer builds the same todo API. Boredom hits when CRUD stops surprising you. That is the signal to move up the stack.
- I Rebuilt a Minimal BitTorrent Client in Go to Understand Peer-to-Peer Choking Algorithms
BitTorrent choking is bandwidth scheduling disguised as game theory. Rebuilding a minimal client in Go makes the TCP, buffer, and fairness costs visible.
- Promotions are Proactive: The 3P Framework for Pitching Your Next Level
Promotion is not a reward for busy sprints. It is recognition that you already operate at the next level's blast radius. The 3P framework turns that into a pitch.
- Glue Work is the New System Design: Why Alignment is the Premium Skill in the Age of AI
AI makes code cheap. The expensive part is getting five teams, three regions, and two data stores to agree on what actually ships.
2026.05 [6]
- The AI code review bottleneck: When writing code is 5x faster, but reviewing is 2x slower
Why AI speeds up code production while reviewers pay the latency tax of architecture and network validation.
- Stop Writing RFCs Like Mystery Novels: Why the Best Design Docs Start with the Conclusion
Why architecture docs should lead with the conclusion and why the rest of the content belongs at the bottom.
- Clock synchronization is a nightmare: Why Spanner uses TrueTime and the rest of us suffer
Why relying on system clocks causes silent data corruption and how TrueTime solves it for Spanner.
- Scaling a distributed cache: Why consistent hashing is mandatory
Why modulo-based cache sharding fails in production and how consistent hashing with virtual nodes protects your database.
- Instrumenting distributed messaging: The fallacy of exactly once delivery
Why Kafka exactly-once delivery is a coordination tax and why you should build idempotency at the edge.
- Instrumenting AI: Multi-master replication and the split brain problem
Why active-active embedding stores feel attractive, and why multi-master replication can trigger a split brain failure in AI infrastructure.
2026.03 [23]
- Instrumenting AI Agents: Why the Apology Metric Is a First Class Reliability Signal
Track apology phrases as a first class SLO for AI agents: spikes reveal context starvation, timeout dropouts, and payload truncation across data boundaries.
- The Expensive Cosplay of Local Models: True 3 AM Operational Cost of Hosting Llama-3
Self-hosted Llama-70B looks cheap until VRAM, KV cache, and HBM bandwidth cap throughput. TCO is idle GPUs, batching latency, and ML infra on call.
- AI agents break connection pooling by holding the slot while they think
Agents keep pooled DB connections open for LLM inference, exhausting pools and evicting buffer cache. Decouple reasoning from data, route agents to replicas, and never hold a connection across an inference call.
- Thread-per-Core Architecture: Why Extra Threads Eventually Destroy Throughput
Oversized thread pools stall: timeslicing, context switches, cache thrashing. Thread-per-core, CPU pinning, and async I/O match physical cores.
- Branch Prediction: Why an if Inside a Hot Loop Costs Milliseconds
How CPU pipelining and branch predictors work, why mispredictions flush the pipeline, and how sorting, branchless code, and loop unrolling help.
- CPU Caches and Spatial Locality: Why an Array is 3x Faster Than a Linked List for the Exact Same Big-O Complexity
Why arrays are faster than linked lists on real CPUs: cache lines, spatial locality, hardware prefetchers, and pointer chasing.
- RSS vs VSZ in Virtual Memory: What the OOM Killer Actually Counts
malloc() grows VSZ, page faults commit RSS, and the Linux OOM killer only counts RSS. How virtual memory and lazy allocation decide which process dies.
- Cuckoo Filters: Cache-Friendly Membership Checks With Deletions
How Cuckoo filters work: fingerprints, two-bucket lookups, kick-out insertions, why they stay cache-friendly, and the real tradeoff of insertion failure.
- Bloom Filters vs Counting Bloom Filters: When Deletions Kill Performance
Why counting (deletable) Bloom filters often lose in production: cache misses, random memory access, and better alternatives like hash tables or Cuckoo filters.
- Pagination at Scale: Why OFFSET and SKIP Will Eventually Break Your API
Why OFFSET/SKIP pagination degrades linearly with depth, how cursor-based pagination keeps latency flat, and when to switch before production bites back.
- The RUM Conjecture: You Cannot Optimize Reads, Updates, and Memory at Once
How the RUM Conjecture explains real-world database trade-offs between read latency, write throughput, and memory overhead across B-Trees, LSM-Trees, and hash indexes.
- Why UUID Primary Keys Quietly Destroy Database Performance
How random UUID primary keys break clustered indexes, cause page splits and buffer pool churn, and what to use instead for mechanically sympathetic database design.
- Designing Resilient APIs: Failure-Handling Patterns for Distributed Systems
Practical resilience patterns for distributed APIs: fail-fast, retries with backoff, circuit breakers, bulkheads, fallbacks, rate limiting, failover, and observability.
- Microservices Deep Dive: Architecting for Scalability and Resilience
How to design, operate, and scale microservices: core principles, when to use them, key patterns, and how to manage complexity in distributed systems.
- Consistency Models in Azure Cosmos DB: From Strong to Eventual
How Azure Cosmos DB's five consistency levels map onto PACELC tradeoffs, what each level guarantees, and how to choose the right consistency for your workload.
- Zero Trust Architecture: From Perimeter Walls to "Never Trust, Always Verify"
How Zero Trust Architecture replaces perimeter-based security: core principles, differences from traditional models and ZTNA, enabling technologies, and real-world implementations.
- Mastering Event-Driven Architecture with Apache Kafka
How to design scalable, resilient systems using event-driven architecture and Apache Kafka for high-throughput, real-time data processing.
- Transitioning from REST to gRPC: System Design and Tradeoffs
How gRPC changes API design versus REST: protocol model, protobuf schemas, service interfaces, streaming patterns, and when gRPC or REST is the right architectural choice.
- HTTP/2 System Design: How It Fixes HTTP/1.1
Deep dive into HTTP/2: why HTTP/1.1 hit scaling limits, how multiplexing, server push, binary framing, and prioritization work, and why it matters for web performance.
- Shazam finds songs by voting on time offsets, not by comparing audio
How Shazam's fingerprinting works according to the published Wang 2003 paper: constellation maps, combinatorial peak pairing into 32-bit hashes, and the offset histogram that turns song matching into counting.
- System Migration: Minimize Downtime, Maximize Efficiency
A practical blueprint for system migration: isolated env, sync/async flows, bridge layer, traffic leakage, backup sync, and monitoring.
- System Design: Principles for Maintainability, Scalability, and Reliability
Data building blocks, fault tolerance, latency vs response time, scaling strategies, and the operability-simplicity-evolvability triad for durable systems.
- Your week is a queueing system, and you are running it at 100% utilization
Time management for software engineers, treated as the scheduling problem it is: utilization and latency, context switches as cache eviction, Little's law for work in progress, and priority inversion in your calendar.