Your trace has a gap exactly where the bug is

A checkout request takes 4.2 seconds. The trace shows one span, the HTTP handler, with a duration of 4.2 seconds and three children totalling 180 milliseconds.

Four seconds of the request happened somewhere the trace cannot see. There is no error, no exception, and nothing in the span attributes hinting at what occupied that time.

The handler submits work to an executor. The executor threads do the actual work, including two database calls and an HTTP request to a pricing service, and none of it appears in the trace because the trace context did not travel with the task.

Context is a variable, and it lives on a thread

Tracing works by passing a small object down the call path. It holds the trace id, the current span id, and sampling flags. Every instrumented library reads the current context to decide what its span’s parent is.

That current context is stored somewhere ambient, which in a JVM means a thread local. So the moment execution moves to a different thread, the new thread has no context and any span it starts becomes the root of a new trace.

flowchart TB
    subgraph req["Request thread"]
        H["HTTP span<br/>context present"] --> SUB["executor.submit(task)"]
    end
    subgraph pool["Pool thread"]
        T["task runs<br/>context ABSENT"] --> DB["DB span, orphaned"]
        T --> API["HTTP span, orphaned"]
    end
    SUB -.->|"context not carried"| T

    style T stroke:#ef4444,stroke-width:3px,color:#fff

The parent span stays open until the future completes, so it records the full duration. The children exist as separate traces with no parent, or are dropped entirely if the sampler decides independently. Either way the trace you open shows time with nothing in it.

The instrumentation is not broken. It is doing exactly what it was told, on a thread that was never told anything.

The four boundaries that break it

Thread pools are the common one, and it is fixable at the point of submission.

// Loses context. The task runs with whatever the pool thread had, which is nothing.
executor.submit(() -> chargeCard(order));

// Carries it. Context.taskWrapping returns an executor that captures the
// submitting thread's context and reinstates it inside the task.
ExecutorService traced = Context.taskWrapping(executor);
traced.submit(() -> chargeCard(order));

Wrapping the executor once at construction is better than remembering at every call site, and it is the change I would make first because it fixes every submission through that pool at once.

Message queues break it because the consumer runs in a different process entirely. Context has to be serialised into the message and read back out.

// Producer: inject the current context into the record headers.
GlobalOpenTelemetry.getPropagators().getTextMapPropagator()
        .inject(Context.current(), record.headers(), KafkaHeaderSetter.INSTANCE);

// Consumer: extract it before starting the processing span.
Context extracted = GlobalOpenTelemetry.getPropagators().getTextMapPropagator()
        .extract(Context.current(), record.headers(), KafkaHeaderGetter.INSTANCE);

Whether the consumer span becomes a child or a link is a modelling decision worth getting right. A child implies the consumer ran within the producer’s request, which is false for anything asynchronous and makes the producer’s trace duration meaningless. A link records the causal relationship without the containment, and it is the correct choice for queues.

Callbacks and reactive chains lose context whenever the continuation runs on a scheduler that did not capture it. Reactor propagates context through its own mechanism rather than thread locals, which means the automatic instrumentation only works if the context is bridged, and the failure is silent.

Retries inside client libraries are the sneakiest. A resilience layer that retries three times may produce one span covering all three attempts, so a trace showing a 3 second call is actually three 1 second calls and two failures, and nothing indicates that. Instrumenting each attempt separately is worth the extra spans, because the retry count is usually the thing you needed to know.

A gap is not always missing instrumentation

Before assuming propagation broke, there is a second explanation with the same signature.

A stop the world garbage collection pause halts the thread without executing anything, so the span duration includes it and no child span can exist. That produces an identical looking gap, and it correlates with pauses across unrelated endpoints at the same timestamp, which is how you tell the two apart.

Waiting for a connection from a pool is a third. The wait happens before any query span starts, so it lands inside the parent with nothing to show for it. Instrumenting acquisition separately turns an invisible wait into a visible one, and it is worth doing because pool saturation looks exactly like a slow database until you can see the split.

Span span = tracer.spanBuilder("db.connection.acquire").startSpan();
try (Scope scope = span.makeCurrent(); Connection conn = dataSource.getConnection()) {
    span.end();                      // the wait is now its own span
    return runQuery(conn);
}

Sampling decides what you can debug

Head sampling makes the keep-or-drop decision at the start of a trace, before anything has happened. That is cheap and it means the decision is made without knowing whether the request turned out to be slow or failed. At one percent sampling, 99 out of 100 incidents have no trace.

Tail sampling buffers the whole trace and decides at the end, so you can keep everything that errored or exceeded a latency threshold and drop the ordinary successes. It costs memory in the collector and requires all spans of a trace to reach the same collector instance, which is a real deployment constraint.

For debugging, tail sampling is what you want, because the traces worth keeping are exactly the ones head sampling discards at random.

One property of head sampling worth knowing: the decision propagates. A downstream service cannot decide to keep a trace that an upstream service already sampled out, which means a service that is being debugged is at the mercy of whoever called it.

The check that finds this quickly

Take a slow trace and add up the child span durations. If the total is well below the parent’s duration, you have unaccounted time, and the question becomes which of the four boundaries it fell through.

def unaccounted_time(span):
    # Rough, and rough is enough to tell you where to look.
    children = sum(c.duration for c in span.children)
    return span.duration - children

Then look at what the code does in that window. An executor submit, a queue publish, a callback registration or a client with retries configured accounts for nearly all of it.

The thing I would push back on is treating a complete-looking trace as proof that nothing is missing. A trace shows what was instrumented and linked. Work that fell off the context is not marked as absent, it simply is not there, and the shape of the trace looks entirely reasonable without it.