Async does not mean parallel, and confusing them costs you a thread pool

A Python service handles 4,000 concurrent connections comfortably on one process. It is async throughout, latency is flat, and the CPU sits around 25 percent.

Someone adds an endpoint that generates a PDF summary. It is a reasonable feature, it takes about 300 milliseconds of pure computation, and it is called maybe twice a minute.

p99 across every other endpoint jumps by 300 milliseconds. Not the PDF endpoint. All of them. A health check that does nothing but return a constant now occasionally takes a third of a second.

The PDF handler was marked async def, which everybody read as making it safe to run on the event loop, and which actually means nothing of the sort.

Two different words for two different things

Concurrency is a structuring property: multiple tasks are in progress and each can make progress independently. Parallelism is a hardware property: multiple things execute in the same instant on different cores.

A single threaded event loop is extremely concurrent and not parallel at all. It handles thousands of connections by interleaving them, and exactly one line of code is executing at any moment.

That works because a typical request spends most of its life waiting. Waiting on a database, waiting on an HTTP call, waiting on a socket write to drain. During every one of those waits the thread has nothing to do, and an event loop uses that time to advance some other request.

flowchart TB
    subgraph io["I/O bound work on an event loop"]
        A["Task A: await db"] -.->|"yields"| B["Task B: await http"]
        B -.->|"yields"| C["Task C: await db"]
        C -.->|"yields"| A2["Task A resumes"]
    end

    subgraph cpu["CPU bound work on an event loop"]
        D["Task D: compute 300ms"] --> D2["...still computing..."]
        D2 --> D3["...nothing else runs..."]
        D3 --> E["every other task resumes late"]
    end

    style E stroke:#ef4444,stroke-width:3px,color:#fff

The loop only regains control at an await. A function with no await point runs to completion no matter how long it takes, because there is no preemption. Nothing interrupts it.

async def is a promise about waiting, not about speed

The keyword marks a function as capable of suspending. It does not confer any ability to run elsewhere.

# Concurrent and useful. The await yields control while the network works.
async def fetch_user(session, user_id):
    async with session.get(f"/users/{user_id}") as resp:
        return await resp.json()

# Async in name only. There is no await, so the loop is held for the
# full duration and every other pending task waits.
async def render_pdf(document):
    return heavy_pdf_library.render(document)     # 300ms of pure CPU

The second function is worse than the synchronous version would be, because its author now believes it is safe to call from a request handler.

The tell is mechanical: an async def with no await in the body is either doing nothing that needs to be async, or it is holding the loop. Both are worth a second look.

Getting CPU work off the loop

The correct move is to run it somewhere that is genuinely parallel, which in Python means another process, because the interpreter lock prevents two threads executing bytecode simultaneously within one process.

from concurrent.futures import ProcessPoolExecutor
import asyncio

pool = ProcessPoolExecutor(max_workers=4)

async def render_pdf(document):
    loop = asyncio.get_running_loop()
    # Real parallelism: another process, another core, loop stays free.
    return await loop.run_in_executor(pool, heavy_pdf_library.render, document)

Now the handler genuinely awaits, the loop serves other requests during those 300 milliseconds, and the work happens on a different core.

The cost is that arguments and results are pickled and copied between processes, which for a large document is not trivial. If the payload is big enough that copying dominates, the answer is usually a separate service with its own scaling rather than a process pool inside the API.

A thread pool executor is the right choice when the blocking call releases the interpreter lock, which covers most C extension work and any legacy synchronous library doing socket I/O. It is the wrong choice for pure Python computation, where the threads will contend on the lock and give you concurrency without parallelism, which is what you already had.

The blocking call that hides in a library

The failure that is hardest to spot is not an obviously heavy function. It is an innocuous synchronous library call inside an async handler.

async def get_profile(user_id):
    # requests is synchronous. This blocks the loop for the whole
    # network round trip while pretending to be a normal call.
    resp = requests.get(f"https://api.internal/users/{user_id}")
    return resp.json()

That is a 40 millisecond network wait during which the entire server does nothing. It looks identical to the correct version at a glance, and it is the single most common async bug I have seen.

The same applies to a synchronous database driver, time.sleep instead of asyncio.sleep, file reads with the built-in open, and any SDK that has not been written for async. In Node the equivalents are fs.readFileSync, synchronous crypto operations, and JSON.parse on a very large payload.

Python can tell you when this happens:

loop = asyncio.get_running_loop()
loop.set_debug(True)
loop.slow_callback_duration = 0.05    # warn on anything holding the loop 50ms+

That warning in a staging environment finds these quickly, and it is worth enabling by default in non-production. The equivalent instinct in Node is to watch event loop lag as a first class metric, because it is the number that goes up when this is happening and the one that stays flat when the problem is elsewhere.

Where this shows up in JVM services

The same confusion appears with a different vocabulary. A CompletableFuture chained without specifying an executor runs on the common ForkJoinPool, which is sized to the core count and shared with parallel streams. A blocking call inside one of those stages occupies a pool thread that other work depends on, and enough of them will starve the pool entirely.

Reactive frameworks make it explicit and easier to get wrong, because the scheduler is a parameter you have to think about. A blocking JDBC call inside a Reactor chain on the event loop scheduler blocks an event loop thread, and there are only as many of those as you have cores.

The question worth asking

Before making anything async, ask what it waits for.

If the answer is a network call, a disk read, or a queue, async is the right tool and the win is large: one thread serving thousands of concurrent operations because they are all idle most of the time.

If the answer is nothing, if the function just computes, then async gives you scheduling overhead and a way to block everything else. The work needs a different core, and that means a process, a pool, or another service.

The distinction I keep coming back to is that async removes waiting, and it cannot remove work. A request that needs 300 milliseconds of computation will always need 300 milliseconds of some core’s time. The only decision available is whose latency pays for it, and putting it on the event loop means everyone’s does.