The Outbox Pattern: A Design Deep-Dive

At some point in your career, you have written these two lines. Save a record to the database. Then send a confirmation email. Or save the order and notify the warehouse. Or update the status and fire off a webhook. Two operations, one after the other. Straightforward.

And at some point, you have had this thought: what if the second one fails? The record is already saved. The email was never sent. The warehouse never heard about it. The customer is waiting. Do you retry? Do you roll back? Do you send the email again and risk a duplicate? There is no clean answer, and most developers either add a try-catch and hope for the best, or quietly accept that sometimes things get missed.

That feeling, that quiet discomfort, is the Dual Write Problem. You have already experienced it. You just did not know it had a name.

Now scale that problem up. Instead of a single server sending an email, you have a microservice committing to PostgreSQL and then publishing an event to Kafka. Instead of a simple try-catch, you have Kubernetes pods restarting mid-operation, network partitions lasting 300 milliseconds, and broker timeouts that happen silently under load. The same two-line problem, at a scale where the consequences are financial inconsistency, missed orders, and support tickets that nobody can explain.

The Dual Write Problem

Every time a service needs to write to its own database and notify another system, it faces the Dual Write Problem. These are two completely separate systems, two separate network calls, two separate operations that can each succeed or fail independently.

The danger is not that it always fails. The danger is that it almost always succeeds. In happy-path conditions, the window between a database commit and a broker publish is a few milliseconds. Nothing goes wrong. Your tests pass, your staging environment looks clean, your monitoring shows green. And then, in production, under real load, on a Tuesday afternoon when a rolling deployment restarts your pod at exactly the wrong moment, the database commits and the event disappears. Silently. No error. No alert. Just an inconsistency that someone will discover the next morning.

• Both succeed. Everything is fine.
• The database write fails before the event is published. Nothing downstream hears about it, but the database is consistent. You can retry the whole operation safely.
• The database write succeeds, then the service crashes before publishing the event. The database has the new state. The broker has nothing. The downstream world is now out of sync.
• The database write succeeds, the publish call is made, the broker accepts it, but the database commit is then rolled back due to an unrelated constraint. An event describing something that did not happen is already in flight.

Outcomes three and four are the ones that haunt production systems. They are rare enough that they survive staging. They are silent enough that your monitoring does not catch them immediately. And they are consequential enough in financial, healthcare, or order processing contexts that they become incidents when they do occur.

The Dual Write Problem: two separate network calls with no shared transaction guarantee

The Forces at Play

Relational databases provide ACID transactions: a suite of guarantees that ensure committed writes survive crashes, concurrent operations stay isolated, and multiple writes within a single transaction all land or none of them do. These are powerful guarantees, and they are the foundation of every reliable data layer.

But here is the catch. The moment you step outside the database boundary, all of those guarantees stop. When you make a network call to a message broker, you are in a completely different system with completely different guarantees. The broker does not know about your database transaction. The database does not know about your broker publish. There is no shared transaction coordinator between them.

The difference between these two worlds is the difference between “succeeded individually” and “succeeded atomically.” Two operations that succeed individually can still leave your system in an inconsistent state if the first succeeds and the second does not, or if they succeed in the wrong order, or if one succeeds and then needs to be rolled back. Atomic success means all-or-nothing as a single unit. You can have atomic success within one database. You cannot have it, natively, across two separate systems.

Why the Obvious Fixes Fail

When developers first encounter the Dual Write Problem, three solutions tend to come up quickly. All three are reasonable instincts. All three fall short in important ways.

Retry Loops

The first instinct is to add retry logic around the broker publish. If publishing fails, try again. Add exponential backoff. Retry several times before giving up. This is a genuine improvement. It handles transient network failures and momentary broker unavailability. But it does not solve the fundamental problem.

Retry logic assumes the process keeps running. If the service crashes between the database commit and the first publish attempt, there is no process left to run the retries. The state was committed. The event was never sent. No amount of retry logic in the dead process helps you recover from that. Retry loops improve resilience for transient failures. They do not provide durability guarantees for persistent ones.

Distributed Transactions (Two-Phase Commit)

If the problem is that two separate systems cannot share a transaction, the theoretical answer is a distributed transaction coordinator. Two-Phase Commit, or 2PC, is the classic approach: a coordinator asks both participants to prepare, waits for both to confirm readiness, and then tells both to commit. If either participant cannot commit, both are told to roll back.

In theory, this gives you atomicity across two systems. In practice, the industry largely moved away from 2PC for several reasons. It requires all participating systems to support the protocol. Many modern message brokers do not. It introduces a coordinator that becomes a single point of failure. The two-phase protocol blocks resources while waiting for confirmation, which creates latency and contention under load. And in the presence of network partitions, 2PC can leave participants in a blocked state waiting for a coordinator that never responds. The failure modes are complex and the operational cost is high. For systems that need to scale and remain available, 2PC is generally the wrong tool.

Saga Pattern

The Saga Pattern sometimes comes up as an alternative here. It is worth understanding why it solves a different problem entirely. The Saga Pattern coordinates a multi-step workflow that spans multiple services. Each step has a corresponding compensating action. If step three fails, you trigger compensations for steps two and one. It is a choreography or orchestration approach for distributed business transactions.

The Dual Write Problem is a single-service atomicity problem. A single service needs to write to its own database and publish an event, and those two things need to happen as a reliable unit. Sagas do not address that. A Saga can tell you how to coordinate an order placement, a payment charge, and an inventory reservation across three services. It cannot tell you how to guarantee that the event announcing the order placement ever leaves the first service in the first place. That is a different problem requiring a different solution.

How It Works: The Design

The Outbox Pattern: both writes happen in one transaction, a background processor handles reliable delivery

The Outbox Pattern has three components, and understanding each one separately is the key to understanding how they work together.

The Outbox Table

The Outbox table lives in the same database as your service’s primary data — this is the entire point. Because it shares the same database, it can participate in the same transaction. It is not a message broker or a queue service. It is a regular database table that serves as a staging area: holding a record for each event with its type, payload, creation time, processed status, and optionally the number of delivery attempts.

The Command Handler Change

The command handler, whatever code processes an incoming request and mutates state, does not publish events directly to the broker anymore. Instead, it writes the event record to the Outbox table as part of the same database transaction that writes the state change. The two writes happen together. Both commit or neither commits. There is no window between them where one can succeed and the other fail.

This is the single transaction guarantee that the Outbox Pattern is built on. By keeping both writes inside one database transaction, you get ACID atomicity over both of them. The state change and the intent to notify are inseparable.

The Background Processor

The background processor, sometimes called the Outbox Processor or the Relay, runs separately from the main request path. On a schedule, it queries the Outbox table for unprocessed events, publishes each one to the message broker, and marks it as processed once the broker confirms receipt. If publishing fails, the event stays in the table as unprocessed. The next polling cycle will pick it up and try again.

Trade-offs and Constraints

The Outbox Pattern is not free. It solves a genuine problem, but it introduces real operational complexity and behavioral changes that need to be understood before adoption.

Polling Latency

Events are no longer published in the same request cycle. They are published when the background processor next polls the Outbox table. If your polling interval is five seconds, downstream consumers may not see the event for up to five seconds after the originating transaction commits. For most use cases this is entirely acceptable. For systems where near-real-time propagation is critical, you need to think carefully about your polling interval and whether the Outbox Pattern’s latency characteristics fit your requirements.

Operational Complexity

You now have a background processor to deploy, monitor, and keep alive. If the processor stops running, events accumulate in the Outbox table but are never delivered. Your primary service continues functioning normally. Your downstream consumers stop receiving events. This failure mode is silent unless you have monitoring specifically watching the Outbox table for events that are old and unprocessed.

You also have a table that grows over time. Processed events need to be cleaned up. A cleanup job needs to be written and scheduled. The Outbox table is not a permanent event store. It is a delivery staging area, and it needs to be maintained as such. These are operational responsibilities that did not exist before adopting the pattern.

At-Least-Once Means Duplicates Are Possible

This is not a risk to be minimized. It is a design contract to be honored. Every consumer of events from an Outbox-backed publisher must be designed with the understanding that it will occasionally receive the same event more than once. This is not a failure mode. It is a stated property of the system. Systems that are not designed around this will exhibit bugs that are intermittent, hard to reproduce, and potentially costly in domains like finance or inventory.

Idempotent Consumers Are Required

Idempotency means that processing the same message more than once produces the same outcome as processing it once. A consumer that creates an invoice on receiving an order event is idempotent if it checks whether one already exists before creating a new one. A consumer that creates without checking will produce two invoices on duplicate delivery. Design every consumer around one question: can this handler receive the same message twice without causing a problem? Track processed message identifiers if the answer is not immediately obvious.

Event Ordering Is Not Guaranteed

The Outbox Pattern does not guarantee global event ordering. If an event fails and is retried on the next polling cycle, newer events that were successfully published will have preceded it. In a horizontally scaled deployment, different processor instances may pick up events in different sequences. For most event types this does not matter. For sequences where order is meaningful, such as an AccountCreated event that must arrive before an AccountActivated event, you need an explicit ordering strategy.

The common strategies are partitioning events by aggregate identifier, or using per-aggregate sequence numbers to allow consumers to detect and reorder out-of-order delivery. Either way, this is a design decision to make before implementation, not one to discover in production.

Multi-Instance Concurrency

In a horizontally scaled deployment, you may be running three or four instances of your service. All of them run the Outbox Processor. All of them poll the same Outbox table. Without coordination, all of them will attempt to process the same unprocessed events at the same time, resulting in the same events being published multiple times in rapid succession.

The standard approach is pessimistic locking at the database level: when an instance queries for unprocessed events, it acquires a row-level lock before returning them, preventing other instances from selecting the same rows. Each event is then processed by exactly one instance at a time. The specific mechanism is an implementation detail covered in the companion article. The design requirement is clear: your Outbox Processor needs a concurrency strategy from the start.

Dead Letters and Persistent Failures

What happens when an event cannot be delivered, not because of a transient network blip, but because the payload is malformed, or the broker topic no longer exists, or there is a schema mismatch that prevents deserialization? The processor will retry indefinitely, accumulating failures for that event while newer events pile up behind it.

You need a dead letter strategy. After a configurable number of failed attempts, an event should be moved to a dead letter table or flagged for manual review. It should not block the processing of subsequent events. The dead letter table becomes an operational artifact: it needs to be monitored, triaged, and resolved. This is a normal part of distributed messaging infrastructure, but it is one more responsibility that comes with the pattern.

When to Use It and When Not To

The Outbox Pattern is powerful, but it carries real operational weight. Applying it indiscriminately is as much a mistake as ignoring it where it matters.

Use It When

• A state change in your service must reliably trigger downstream work. Financial transactions, order placements, account creations, payment confirmations. If missing the event would cause a real-world problem, use the Outbox Pattern.
• You are already using a message broker and you cannot afford event loss. If you have invested in an event-driven architecture, the Outbox Pattern is what makes the guarantees real.
• Your downstream consumers take business-critical actions based on events. Inventory reservations, compliance audit records, settlement workflows. The higher the stakes of the downstream action, the more important it is that the event is guaranteed.
• You are running in a horizontally scaled environment where services restart, redeploy, and occasionally crash. The Outbox Pattern is designed for exactly these conditions.

Skip It When

• The side effect is genuinely optional and occasional loss is acceptable. Analytics events, non-critical activity tracking, view counts. If losing one in a thousand of these events causes no real harm, the operational overhead of the Outbox Pattern may not be justified.
• Your system is very low throughput and the operational complexity outweighs the benefit. A small internal tool processing a few dozen requests per day may not need the infrastructure that the Outbox Pattern requires.
• Your entire architecture is synchronous and request-response with no downstream event consumers. The Outbox Pattern solves a messaging delivery problem. If there is no messaging, there is nothing for it to do.

Where This Fits in the Bigger Picture

The Outbox Pattern has a precise scope. It solves within-service atomicity: a single service guaranteeing that its state change and its published event are inseparable. It says nothing about what other services do with that event, or how a multi-step business workflow spanning multiple services is coordinated.

That is where the Saga Pattern lives. A Saga coordinates a distributed workflow: an order service places an order, a payment service charges a card, an inventory service reserves stock. If any step fails, compensating actions undo the previous steps. The Saga Pattern manages the workflow. The Outbox Pattern ensures that each service within that workflow reliably publishes its events. They complement each other at different levels of the architecture.

Change Data Capture as an Alternative

There is another approach worth knowing: Change Data Capture, or CDC. Instead of writing events to an Outbox table and having a processor relay them, CDC reads directly from the database transaction log. Every committed change is captured at the infrastructure level and forwarded to the broker. Tools like Debezium do this for PostgreSQL, MySQL, SQL Server, and others.

CDC has a compelling property: it does not require your application code to know about events at all. The database itself is the source of truth, and CDC extracts changes from it automatically. This eliminates the Outbox table entirely and removes the background processor from your application. The trade-off is that CDC is infrastructure that must be deployed, monitored, and maintained separately, and the events it produces are low-level data change records, not domain events. Transforming them into meaningful business events is additional work. CDC is worth understanding as an alternative, particularly for systems where you cannot modify the application code, or where you want event sourcing at an infrastructure level rather than an application level.

Libraries That Do This For You

In the .NET ecosystem, MassTransit includes a first-class Outbox implementation. It handles the processor, the locking strategy, the retry policy, and the cleanup. NServiceBus has similar capabilities. If you are building in .NET and using either of these libraries, read their documentation on the Outbox before writing your own. The pattern they implement is the same one described here, but the implementation details, particularly around multi-instance concurrency and dead lettering, have been hardened over years of production use. There is real value in that.

Conclusion

The Outbox Pattern is not a clever trick. It is an honest acknowledgment of a fundamental constraint: two separate systems cannot share a transaction, and any design that pretends otherwise will eventually produce data loss.

What it does is build reliability around that constraint rather than against it. Keep both writes inside the one system where you do have atomicity. Let a dedicated processor handle delivery from there. Accept that delivery may be delayed. Accept that it may happen more than once. Design for both of those properties explicitly, and give yourself the tools to monitor them.

It does not promise what it cannot deliver. It does not hide failure modes behind optimistic assumptions. It behaves predictably under failure, which is the only kind of behavior that matters in production.