Instrumenting distributed messaging: The fallacy of exactly once delivery

I watch engineering teams tick a configuration box for “exactly once” delivery in Kafka and assume their data pipeline is bulletproof. I genuinely do not know how to feel about this. The abstraction is beautifully simple on paper. In production, it ignores the physical reality of how computers talk to each other.

When you architect a high throughput retail backend or a data pipeline, relying on your message broker to magically solve state duplication will burn your infrastructure budget and destroy your latency.

The Physics of a Message

Let us look at what happens when your producer sends an event to a broker.

Your application makes a system call. The CPU copies data from user space to kernel space. The network interface card reads that data over the PCIe bus and converts it into electrical signals. Those signals travel over physical switches and cables. The receiving NIC catches those pulses, triggers a hardware interrupt, and the receiving CPU pulls the data into memory before flushing it to a spinning disk or SSD.

Every single point in that chain can fail.

Cables get loose. Switches drop packets when buffers fill up. A heavy garbage collection pause on the broker can delay the acknowledgement packet.

If the sender does not receive an acknowledgment in time, it has a binary choice. It can drop the message, giving you “at most once” delivery. Or it can send the message again, giving you “at least once” delivery.

There is no physical mechanism for “exactly once” over an unreliable network. You either risk data loss or you risk duplication.

sequenceDiagram
  participant P as Producer
  participant K as Kafka Broker
  participant C as Consumer
  participant R as Redis
  participant D as Primary DB

  P->>K: send event
  K-->>P: ack receipt
  K->>C: deliver message
  C->>R: check deterministic hash
  alt hash exists
    R-->>C: duplicate detected
    C-->>K: ack offset without processing
  else hash missing
    R-->>C: no duplicate
    C->>D: upsert/update state
    C->>R: set hash key
    C-->>K: ack offset
  end

The Hidden Coordination Tax

Kafka supports exactly once semantics. They achieved this by building a heavy transactional wrapper around producers and consumers.

To make this work, the system forces distributed consensus. The producer requests a transactional ID. It writes to a transaction state topic. It writes the actual messages. Then it sends a commit request.

You are paying a massive latency tax. You are trading raw throughput for multiple network round trips and forced disk fsyncs on every batch of messages. If you are building a real time inventory sync or feeding a machine learning pipeline, you are bottlenecking your entire system on disk I/O and coordination overhead.

Idempotency at the Edge

You have to accept the messiness of the network. Drop the illusion of exactly once delivery at the message broker level.

Configure your brokers for “at least once” delivery. Accept that duplicates will happen. The network wants to drop packets and retry. Let it. You solve the duplication problem by moving the reconciliation logic to the edge of your compute boundary.

Make your consumers fiercely idempotent.

• Generate a deterministic hash of your message payload before sending it.
• When your consumer pulls a message, check that hash against a fast in memory store like Redis.
• If the key exists, drop the message immediately.
• If it does not exist, set the key, run your database update, and acknowledge the message back to the broker.
• Use upserts and unique constraints in your primary database to catch any race conditions.

This is mechanical sympathy applied to software architecture. You stop fighting the unreliable nature of the network and you stop burning compute cycles on forced broker consensus. Build your software to handle the echo, and your pipelines will scale gracefully.