Service mesh: what it buys and the latency it charges

A team adopts a service mesh for mutual TLS, which security asked for. It takes a sprint, it works, and every service now has encrypted transport with no code changes. Genuinely a good outcome.

Three months later, p99 on the checkout path has grown by 9 milliseconds and nobody can attribute it. The chain is five services deep. Each hop passes through a sidecar on the way out and another on the way in, and each of those adds a fraction of a millisecond that is invisible individually.

Nothing regressed. The cost was always there and it was distributed thinly enough that no single dashboard showed it.

What the extra hops actually cost

Without a mesh, a call is one network hop: caller to callee.

With a sidecar mesh it is three: caller to its own sidecar over loopback, sidecar to the remote sidecar over the network, remote sidecar to the application over loopback.

flowchart LR
    A["Service A"] -->|"loopback"| SA["Sidecar A"]
    SA -->|"network + mTLS"| SB["Sidecar B"]
    SB -->|"loopback"| B["Service B"]

    style SA stroke:#f59e0b,stroke-width:2px,color:#fff
    style SB stroke:#f59e0b,stroke-width:2px,color:#fff

The loopback hops are cheap but not free: a syscall, a copy, and a scheduling decision each way. The TLS handshake is amortised across a pooled connection, so it is the per-request framing and policy evaluation that dominates.

Half a millisecond to two milliseconds per call is the range I would plan around, with the tail considerably worse than the median because the proxy has its own connection pools filling up, its own buffers, and its own memory to reclaim.

The per-pod resource cost is the part that surprises finance rather than engineering. A sidecar typically wants somewhere in the region of 50 to 150MB of memory and a slice of CPU. Multiply by pod count. At 400 pods that is tens of gigabytes of cluster memory doing proxying, and it scales with pod count rather than with traffic, so a service scaled wide for availability pays disproportionately.

The comparison that makes the decision honest

The framing I see most often is mesh versus nothing, and that is not the choice anybody actually faces.

Mutual TLS, retries with backoff, circuit breaking, timeouts, load balancing, traffic shifting and consistent request telemetry all exist whether or not you have a mesh. Without one they live in libraries, and libraries have costs that are easy to forget because they are spread across teams rather than shown on a dashboard.

A library has to exist per language. Two languages means two implementations that must behave identically, and they will not. Upgrading it means a coordinated rollout across every service that depends on it, which for a security fix is a scheduling problem measured in weeks. A team that pins an old version silently opts out of the guarantee everyone assumes is universal.

That last row is the one I would weigh most heavily, and it gets discussed least.

The debugging cost lands on the wrong people

When a mesh misbehaves, the symptom appears in an application that did nothing wrong.

A 503 with UO in the Envoy response flags means upstream overflow: the proxy’s connection pool was full. The application sees a 503 from a service that is perfectly healthy, and nothing in its own logs explains it. Diagnosing it requires knowing that Envoy exists, that response flags exist, and where to find them.

Retry policy defined in the mesh interacts with retry logic in the application, and neither knows about the other. That is retries multiplying across layers, with one of the layers being infrastructure that application developers did not configure and may not know is retrying.

mTLS certificate rotation failing produces connection errors that look like network problems. Sidecar startup ordering means an application container can start before its proxy is ready and fail its first requests, which shows up as flaky deploys.

None of these are unfixable and all of them require mesh expertise to diagnose. If that expertise lives in one platform person, every mesh-related incident queues behind them.

Where it clearly pays

Many services across several languages, where library sprawl is the dominant cost. This is the case the technology was built for and it holds up.

A hard requirement for encryption in transit between every service, where doing it in application code across a large estate is genuinely worse.

Progressive delivery that needs traffic splitting by percentage or by header. Doing canary routing at the mesh layer is clean and doing it in application code is not.

Uniform telemetry across services you do not control, including third party components that will never be instrumented by you.

Where I would not

A handful of services in one language. A shared client library gives you the same capabilities, costs no latency, and is debuggable with a stack trace.

Latency-critical paths where a couple of milliseconds per hop matters and the call chain is deep, since the cost compounds with depth.

An organisation with no platform team, because a mesh is a distributed system that has to be operated, and adopting one without someone who owns it means every incident becomes an archaeology exercise.

If you already have one

Measure the added latency directly rather than inferring it. Comparing p50 and p99 with the sidecar injected against a bypassed path on the same service gives a real number, and it is usually different from the number in the vendor’s benchmark.

Decide where retries live and configure them in exactly one place. Mesh-level retries plus application-level retries is a multiplier nobody chose.

Look at ambient or sidecarless modes if memory is the pressure point. Moving the per-pod cost to a per-node one is a substantial saving on a cluster with many small pods, at the cost of losing some per-pod isolation.

And put the proxy’s own metrics next to the application’s on the same dashboard, because the failure you will spend the longest on is the one where the application is healthy and the thing in front of it is not.