An order must reserve inventory, charge payment and create a shipment across three services. How do you keep it consistent?

Two-phase commit gives real atomicity but requires every participant to hold locks until the coordinator decides, and if the coordinator dies mid-decision the participants block indefinitely - unacceptable across services and impossible with an external payment provider. The practical answer is a saga: a sequence of local transactions, each with a compensating action, coordinated either by choreography through events or by an orchestrator. You give up isolation and accept that intermediate states are visible, in exchange for availability and no distributed locks. | Two-phase commit | Saga. Atomicity | True atomicity | Eventual - via compensation. Isolation | Yes | No - intermediate states are visible. Locks held | Until the coordinator decides | Only within each local transaction. Coordinator failure | Participants block | Retry or compensate; no blocking. Works with external APIs | No | Yes. Complexity | Protocol complexity | Business complexity - you must design every compensation. Use when | Within one trusted database cluster | Across services and third parties The saga, concretely Order saga with compensation: Orchestrator - state machine - Owns the saga's state and drives each step, persisting progress after every transition so it can resume after its own crash. Centralised logic makes the flow readable and debuggable.. 1 · Reserve inventory - ↩ release - Local transaction reserves stock. Compensation releases it. Note this is a *reservation*, not a decrement - reservations are far easier to compensate than committed changes.. 2 · Charge payment - ↩ refund - Calls an external provider with your idempotency key. Compensation is a refund - note that it is not a true inverse: the customer saw a charge and a refund, which is visible and sometimes chargeable.. 3 · C

Is exactly-once delivery possible? How do you build an exactly-once pipeline in practice?

Exactly-once delivery is impossible over an unreliable network - the sender can never distinguish a lost message from a lost acknowledgement, so it must either risk losing it or risk sending it twice. What is achievable is exactly-once processing: at-least-once delivery combined with idempotent consumers that deduplicate by message ID, plus atomic commit of the side effect and the dedup record. The transactional outbox pattern extends this to publishing, writing the message into the same database transaction as the state change so you cannot commit one without the other. Semantic | Mechanism | Failure mode | When acceptable. At-most-once | Fire and forget | Messages lost | Metrics, telemetry, non-critical logs. At-least-once | Retry until acknowledged | Duplicates | The default - pair with idempotent consumers. Exactly-once delivery | Impossible | - | Never; it is a marketing term. Exactly-once processing | At-least-once + dedup + atomic commit | None, within one system | The correct goal Why delivery cannot be exactly-once: The sender transmits and waits. No acknowledgement arrives. Was the message lost, or was the acknowledgement lost? The sender cannot tell, and no protocol can make it tell - this is the Two Generals problem. It must choose: do not resend (risk losing it) or resend (risk a duplicate). Every 'exactly-once' product feature is at-least-once plus deduplication somewhere. The transactional outbox You need to update the database and publish an event. Doing both non-atomically means one can succeed while the other fails: publish first and the database write fails, and you have announced something that never happened; write first and the publish fails, and downstream systems never learn about it. The outbox pattern removes the dual write entirely. Dedupl

Distributed Transactions, Sagas and Exactly-Once Processing

Expertsaga2pctransactionsAsked at Amazon, Uber

30-second answer

Two-phase commit gives real atomicity but requires every participant to hold locks until the coordinator decides, and if the coordinator dies mid-decision the participants block indefinitely - unacceptable across services and impossible with an external payment provider. The practical answer is a saga: a sequence of local transactions, each with a compensating action, coordinated either by choreography through events or by an orchestrator. You give up isolation and accept that intermediate states are visible, in exchange for availability and no distributed locks.

	Two-phase commit	Saga
Atomicity	True atomicity	Eventual - via compensation
Isolation	Yes	No - intermediate states are visible
Locks held	Until the coordinator decides	Only within each local transaction
Coordinator failure	Participants block	Retry or compensate; no blocking
Works with external APIs	No	Yes
Complexity	Protocol complexity	Business complexity - you must design every compensation
Use when	Within one trusted database cluster	Across services and third parties

The saga, concretely

Order saga with compensation

Each forward step has an inverse. When step 3 fails, the orchestrator runs the inverses of steps 2 and 1 in reverse order.

ServiceExternalTap a box for detail

Flow

Orchestrator1 · Reserve inventory
1 · Reserve inventory2 · Charge payment
2 · Charge payment3 · Create shipment
3 · Create shipmentOrder confirmed
3 · Create shipment2 · Charge payment(compensate)
2 · Charge payment1 · Reserve inventory(compensate)

	Choreography	Orchestration
Coordination	Services react to each other's events	A central orchestrator drives each step
Coupling	Loose	Services coupled to the orchestrator
Visibility	Hard - the flow exists nowhere explicitly	Easy - one state machine to inspect
Good for	2–3 steps, simple flows	4+ steps, branching, anything you must debug
Risk	Cyclic event storms; nobody understands the whole flow	Orchestrator becomes a bottleneck or a monolith

Say these points

2PC gives atomicity but blocks on coordinator failure and cannot span external APIs.
Sagas trade isolation for availability: local transactions plus compensations.
Orchestration for complex flows, choreography for two or three steps.
Order irreversible steps last; prefer reserve-then-confirm.
Sagas lack isolation - use semantic locks and re-validation.

Avoid these mistakes

Proposing 2PC across microservices or with a third-party API.
Designing forward steps without designing every compensation.
Assuming compensation is a true inverse.
Ignoring that intermediate saga states are visible to other readers.
Non-idempotent compensations that break when retried.

Expect these follow-ups

The compensation itself fails permanently. What now?
How do you show a half-completed order in the UI honestly?
When is 2PC still the right call?

Expertexactly-onceidempotencymessagingAsked at Confluent, Stripe

30-second answer

Exactly-once delivery is impossible over an unreliable network - the sender can never distinguish a lost message from a lost acknowledgement, so it must either risk losing it or risk sending it twice. What is achievable is exactly-once processing: at-least-once delivery combined with idempotent consumers that deduplicate by message ID, plus atomic commit of the side effect and the dedup record. The transactional outbox pattern extends this to publishing, writing the message into the same database transaction as the state change so you cannot commit one without the other.

Semantic	Mechanism	Failure mode	When acceptable
At-most-once	Fire and forget	Messages lost	Metrics, telemetry, non-critical logs
At-least-once	Retry until acknowledged	Duplicates	The default - pair with idempotent consumers
Exactly-once delivery	Impossible	-	Never; it is a marketing term
Exactly-once processing	At-least-once + dedup + atomic commit	None, within one system	The correct goal

The transactional outbox

You need to update the database and publish an event. Doing both non-atomically means one can succeed while the other fails: publish first and the database write fails, and you have announced something that never happened; write first and the publish fails, and downstream systems never learn about it. The outbox pattern removes the dual write entirely.

sqlOne transaction, both effects

BEGIN;
  UPDATE orders SET status = 'paid' WHERE id = 4711;

  -- The event is a row in the SAME database, in the SAME transaction.
  INSERT INTO outbox (id, aggregate_id, type, payload, created_at)
  VALUES (gen_random_uuid(), 4711, 'OrderPaid', '{"order_id":4711}', now());
COMMIT;
-- Either both are durable or neither is. No dual write, no window.

-- A separate relay polls the outbox (or tails the WAL via CDC) and publishes.
-- It may publish the same row twice after a crash - which is fine, because
-- consumers are idempotent. At-least-once by construction.

pythonIdempotent consumer: dedup and effect in one transaction

def handle(message):
    with db.transaction():
        # The uniqueness constraint arbitrates, not a prior SELECT.
        try:
            db.execute(
                "INSERT INTO processed_messages (message_id, processed_at) VALUES (?, now())",
                message.id,
            )
        except UniqueViolation:
            return                        # already processed; safe to ack and move on

        apply_side_effect(message)        # same transaction as the dedup record

    ack(message)

# Both must commit together. Applying the effect and then recording the id in a
# separate transaction reintroduces exactly the window you were trying to close.

When the side effect is external

If the effect is a call to a third-party API you cannot include it in your transaction. Two options: pass your idempotency key to them so their side deduplicates - the reason every serious payment API supports this - or record intent first, call, then record the outcome, and reconcile anything left in the intent state with a background job that queries the provider for the real status.

Scenario	Pattern
DB write + publish event	Transactional outbox, or CDC from the WAL
Consume event + DB write	Dedup table in the same transaction
Consume event + external API call	Pass an idempotency key; reconcile stragglers
Kafka read-process-write	Kafka transactions with `read_committed` isolation
Non-idempotent effect (send email)	Dedup key plus accept rare duplicates; make the content tolerant

Say these points

Exactly-once delivery is impossible; exactly-once processing is at-least-once plus dedup.
The transactional outbox removes the dual-write window entirely.
Commit the dedup record and the side effect in one transaction.
Pass idempotency keys to external providers so they deduplicate too.
Dedup windows are bounded - state the retention and the residual risk.

Avoid these mistakes

Claiming a broker gives exactly-once delivery end to end.
Publishing an event before the transaction commits.
Recording the dedup key in a separate transaction from the effect.
Unbounded dedup tables.
Assuming a consumer is idempotent without checking what it actually does.

Expect these follow-ups

How does Kafka's exactly-once processing work, and what are its boundaries?
How do you handle a consumer that must send an email - inherently non-idempotent?
What is the trade-off between polling the outbox and using CDC?

Distributed Transactions, Sagas and Exactly-Once

An order must reserve inventory, charge payment and create a shipment across three services. How do you keep it consistent?

The saga, concretely

Order saga with compensation

Is exactly-once delivery possible? How do you build an exactly-once pipeline in practice?

The transactional outbox

When the side effect is external

Check your understanding

Hands-on challenge

Implement an order saga with compensation

Curriculum