AI infrastructure without the hype
What LLM systems cost and where they break: connection pools, retrieval, caching, evals, guardrails, and the bill nobody attributes.
Read in order: later instalments assume the earlier ones. 12 more instalments in the publish queue.
- 01 AI agents break connection pooling by holding the slot while they think
Agents keep pooled DB connections open for LLM inference, exhausting pools and evicting buffer cache. Decouple reasoning from data, route agents to replicas, and never hold a connection across an inference call.
- 02 The Expensive Cosplay of Local Models: True 3 AM Operational Cost of Hosting Llama-3
Self-hosted Llama-70B looks cheap until VRAM, KV cache, and HBM bandwidth cap throughput. TCO is idle GPUs, batching latency, and ML infra on call.
- 03 The AI code review bottleneck: When writing code is 5x faster, but reviewing is 2x slower
Why AI speeds up code production while reviewers pay the latency tax of architecture and network validation.