Exactly-Once Is the Wrong Default Goal

At-least-once delivery means: once the broker has durably stored a message, it will be delivered to consumers one or more times. Prefer “do not lose this work” over “never deliver it twice.”

For most business events, the contract that works is:

At-least-once on the path + handling that is safe if the message arrives twice (often called idempotent handling).

You will understand why “exactly-once” is a risky default goal. Useful if you send or process queue/event work. Assumes service + database basics. Not a broker setup or code tutorial.

Why this bites in production

A payment service publishes PaymentCaptured. A ledger service inserts one journal row, then dies before it acknowledges the message. The broker redelivers. The ledger inserts again. Two lines for one payment. Someone says the bus is “reliable.” Half right: nothing was lost; it was delivered more than once.

The opposite also happens: crash before the insert, redelivery, then one success. Retries exist for that. The painful case is success, then success again.

Duplicates are not only a consumer story. A producer can send, the broker stores, the ack is lost, the producer retries, and two copies exist before any consumer runs. Idempotent producers reduce that risk; they do not make every business side effect safe.

Building blocks

• Producer — creates and sends the message
• Broker — stores and routes it (Kafka, RabbitMQ, Service Bus, SQS, …)
• Consumer — applies real work (DB write, payment call, email)
• Ack — consumer signals “done” so the broker can stop redelivering
• Retry / redelivery — same message sent again after failure or crash

“Accepted” can mean different moments (producer thought it sent, broker wrote to disk, consumer applied work). At-least-once usually means: after the broker confirms it has the message, keep trying until a consumer completes or the message is parked (for example a DLQ).

No broker can atomically commit both your database transaction and an arbitrary external API call. That is why “exactly-once for the whole business workflow” is hard.

Mental model

Three parties fail on their own clocks: producer, broker, consumer. Between “I applied the effect” and “I confirmed completion,” there is a crack. Redelivery prefers duplicates over silence.

(If you use Kafka, this often shows up around consumer offsets: progress markers. Die before the offset is committed, and the same records can be read again. Other brokers use different names for the same idea.)

Guarantee	May lose messages?	May deliver twice?	Typical use
At-most-once	Yes	Rarely	Best-effort metrics, some telemetry
At-least-once	Try not to, after broker accept	Yes	Orders, payments, inventory, audit
Exactly-once	Only inside carefully defined boundaries	Still not the whole business for free	Broker EOS, FIFO dedupe, or effectively once via safe handling

Kafka exactly-once semantics, Service Bus duplicate detection, and SQS FIFO deduplication are real capabilities. The mistake is scope: they strengthen parts of the path. They do not automatically make every database insert or external API call safe under redelivery.

Takeaway formula:

At-least-once delivery + safe-to-repeat handling = effectively once business effect.

That is a contract you can design. “The bus promised exactly-once for every side effect in my company” is not.

Delivery guarantees and ordering guarantees are separate. A system may deliver every message and still allow messages to arrive out of order. Do not assume “at-least-once” implies “in order.”

What people get wrong

They only harden the producer. Events leave the service carefully. One common approach is the Outbox Pattern: store the business change and the outgoing event in the same database transaction, then publish later, which prevents the classic “database committed but event never sent” problem. See The Outbox Pattern: A Design Deep-Dive. If the consumer then always inserts a new row with no duplicate check, you reduced loss on the way out and invited double application on the way in.

They stretch broker guarantees into business guarantees. “Exactly-once” on a broker segment is not the same as “charged once, emailed once, reserved once” across three systems.

They treat retries as cheap to add and free of consequence. Retries are easy to configure. They are expensive if handlers are not safe to repeat. Backoff without safe handling turns a blip into two shipments.

They confuse message id with business id. Skipping a duplicate broker id is not the same as “this paymentId was already posted to the ledger.” Finance reconciles business keys.

When at-least-once is the right default

Prefer it when losing an event hurts more than handling a duplicate:

• Money and ledger entries
• Inventory and reservations
• Access grants and entitlements
• Audit trails you cannot rebuild easily

Also prefer it when you already use retries, redelivery, or outbox-style relays. You have already chosen at-least-once in practice. Write the consumer rule down so the team stops arguing with physics.

When not to force it

Do not overbuild this contract when:

• The data is loss-tolerant and huge (some product analytics, sampled telemetry). At-most-once or sampling can be the honest cheaper choice.
• There is no durable messaging boundary (pure in-process work).
• Nobody will invest in safe-to-repeat consumers (natural keys, inbox / processed-id store, idempotent APIs). At-least-once into unsafe handlers is how you buy silent corruption.

Design-review test (every consumer): What happens if this message arrives twice? If nobody can answer, you are not ready for “guaranteed delivery” language.

What “good” looks like in production

The common enterprise shape (conceptually):

Producer
   → Outbox (optional but common for DB + event)
   → Broker
   → Consumer
   → Inbox / idempotency store
   → Business logic

Outbox helps the producer side leave the database without dual-write loss.
Inbox (or an equivalent processed-id store) helps the consumer side ignore or safely absorb redelivery.
Neither replaces thinking about business keys.

Practical checklist:

• Prefer natural keys (paymentId, orderId + event type).
• Record already processed ids when the effect is not naturally safe to repeat. That consumer-side pattern is commonly called the Inbox Pattern.
• Prefer external APIs that honor your idempotency key. Payment APIs such as Stripe expose idempotency keys so repeated requests produce one payment rather than multiple charges.
• Cap retries. Messages that fail repeatedly should move to a Dead Letter Queue (DLQ) so you can investigate without blocking the rest of the workload forever.
• Watch duplicates and DLQ depth, not only “messages per second” or lag.

You do not need a full code sample for that. You need every consumer PR to answer: What happens if this message arrives twice?

Conclusion

Exactly-once is a comforting default goal. Distributed systems do not sell comfort; they sell trade-offs and boundaries.

For most business events, the contract that survives production is simple: deliver at least once; process as if more than once will happen. Producers reduce publication failures. Brokers reduce transport failures. Consumers preserve business correctness. When all three share that contract, “reliable messaging” becomes an architecture, not a slogan.

Where next

1. The Outbox Pattern: A Design Deep-Dive — getting the event out of the service without dual-write loss.
2. Coming next in this series: idempotent consumers and the inbox pattern (how receivers keep redelivery from double-applying work).
3. In-process .NET notifications are not a message bus: reliability note in Implementing CQRS with MediatR in .NET Core.