Exactly-once processing is often treated as a broker setting. Configure delivery semantics, commit the consumer offset, and the event will be processed once.
This describes the normal path. It does not describe a failure between the business write and the acknowledgement. A consumer can create a row, charge a card, or send a message, then stop before it records completion. On restart, the broker correctly delivers the event again. If the second attempt creates a second business effect, the system has no exactly-once guarantee.
Exactly once is a recovery property. Retrying an interrupted operation must converge on the same business effect as the first successful attempt.
What does “once” describe?
Delivery can occur more than once. Networks retry, consumers restart, and a broker redelivers work when it cannot establish that a consumer finished it. Preventing every duplicate delivery is neither realistic nor necessary.
The relevant guarantee concerns the effect. One order event may be delivered three times while producing one order. One source update may be replayed while producing one current value. A request to an external provider may be retried while producing one payment or notification.
The system therefore needs a stable identity for the intended effect. Without one, it cannot distinguish a duplicate attempt from a second legitimate event.
The initial delivery
Consider an inventory service consuming stock-reserved events. It receives event source=checkout, id=8142, creates a reservation, records progress, and acknowledges the event.
On a healthy run, the consumer advances and the dashboard reports normal throughput. This proves that the service can process the event. It does not prove that processing event 8142 again is safe.
The retry after a partial failure
Assume the process stops after creating the reservation but before recording progress or acknowledging the event. The reservation exists, while the broker still considers 8142 unfinished. After restart, it delivers 8142 again.
If the handler treats the retry as new work, it creates another reservation and corrupts inventory. If it identifies 8142 as an already-applied event, it returns the existing reservation. The second behavior is the required guarantee.
A progress marker records what a worker believes it completed. An idempotent effect remains correct when that record was not written.
Why are offsets insufficient?
Offsets support bounded recovery and lag measurement. They do not atomically describe business work performed in another durability domain.
Suppose a consumer writes to PostgreSQL and commits its position to a broker. Committing the position first can lose the event if the process stops before the database write. Writing to PostgreSQL first can cause a retry if the process stops before the position commit. The second order is safer because it preserves the event, but the database write must accept the retry.
The same failure can exist inside one database. A processed_events record that commits separately from the protected order creates a gap. Recording the event first can lose the order. Recording the order first can repeat it. The effect and its deduplication evidence need the same commit boundary.
How is the effect deduplicated?
For an effect contained in one database, use a stable source key and enforce it with a unique constraint or equivalent compare-and-set rule. Commit the effect and the consumption record in one transaction.
The reservation can carry the source system and source event ID, with a uniqueness rule over that pair. The first transaction creates the reservation. A retry with the same key either finds that committed result or creates it when the first transaction never committed. In both cases, the final state is the same.
Derived state requires the same discipline. A running total cannot absorb a repeated message unless it can determine whether the corresponding contribution was already included. Recomputing an aggregate from uniquely stored input facts is often simpler and safer than retaining an unbounded history of increments.
What changes during retry?
After a partial failure, the consumer repeats the same transaction with the same source key. The transaction creates the reservation if no prior commit exists, or returns the prior reservation if it does. It can then acknowledge the event.
Another failure before acknowledgement causes another delivery. That delivery has the same outcome. Recovery is safe because repeated execution converges, not because the runtime avoids retries.
Can a transaction cover an external system?
A transaction protects work within its own commit boundary. It cannot make an arbitrary external provider participate in that boundary.
If a handler writes a database row and then calls a payment provider, a failure between those actions leaves the outcome unresolved. Calling the provider first leaves the same unresolved state in the opposite order.
An outbox makes the application side durable. The initial transaction commits the business change and an intent to contact the provider. A delivery worker sends the intent and records its progress. If the worker stops after sending, it retries the same intent.
The provider must also recognize that intent. It needs a stable idempotency key, a durable query for the prior outcome, or a compensating operation. Without one of those capabilities, the honest contract is at-least-once delivery with reconciliation.
What should operators measure?
The initial-delivery path needs input rate, consumer lag, handler latency, and the age of unacknowledged work. These signals establish whether the service is keeping up.
The recovery path needs redeliveries, duplicate suppressions, idempotency-key conflicts, replay range, and the age of unresolved external intents. These signals establish whether retries converge or accumulate risk.
A high duplicate-suppression count can indicate unstable consumers, slow acknowledgements, or a short lease. A growing outbox can show that the database is healthy while an external provider is unavailable. A stuck intent records a business obligation whose result still needs to be determined.
The scope of the guarantee
The initial delivery establishes that a new event can create a durable effect. Recovery establishes that uncertainty about the first attempt does not create a second effect. Together, they define the guarantee.
Its scope includes the event identity, the durable record of the effect, the transaction or idempotency boundary, the external systems that honor the same identity, and the reconciliation path for systems that do not.
These details are operational choices. Teams need to decide where duplicates are absorbed, how long identities are retained, and which side effects require a provider-level key or human reconciliation. An exactly-once claim without those choices does not survive the first restart.
The operational model
Do not ask only whether a queue supports exactly-once delivery. Ask whether the business effect converges after a crash, timeout, or replay.
This directs design toward stable event identities, atomic boundaries, conflict rules, external idempotency, and recovery evidence. The trade-offs are visible: a wider deduplication window uses more storage, provider-level idempotency depends on API support, and reconciliation requires operator time. Those costs buy an effect that remains correct under uncertainty.
The one-line version
Exactly once is real only when a retried event converges on the same business effect as its first attempt.
Previous: From Backfills to Live Streams