Distributed Transactions: The Hard Parts

The most surprising truth about distributed transactions is that "atomicity" is often a luxury you can’t afford, and what you really want is "eventual consistency" with a well-defined failure mode.

Let’s see how this plays out in a common e-commerce scenario: a customer places an order. This involves several independent services: an OrderService to create the order, a PaymentService to process payment, and an InventoryService to decrement stock.

Imagine the OrderService successfully creates the order and then calls the PaymentService. If the PaymentService succeeds but the InventoryService fails (maybe it’s down for maintenance), the order is in a broken state. It’s accepted but there’s no stock. This is where distributed transaction patterns come in.

Two-Phase Commit (2PC)

2PC is the traditional database approach to distributed transactions. It involves a transaction coordinator that manages commits across multiple participants.

1. Phase 1 (Prepare): The coordinator asks all participants if they can commit. Participants lock their resources and respond "yes" or "no".
2. Phase 2 (Commit/Abort): If all participants say "yes," the coordinator tells them to commit. If any participant says "no" or times out, the coordinator tells everyone to abort.

Why it’s a problem: If the coordinator fails after participants have prepared but before they commit, those participants are left holding locks indefinitely. This blocks all other operations on those resources, leading to system-wide outages. It’s a single point of failure that is catastrophic.

Sagas

Sagas break down a large transaction into a sequence of smaller, local transactions. Each local transaction updates its own database and publishes an event. If a local transaction fails, compensating transactions are executed to undo the preceding successful local transactions.

Consider our e-commerce example with Sagas:

1. OrderService: Creates order, publishes OrderCreated event.
2. PaymentService: Listens for OrderCreated, processes payment, publishes PaymentProcessed event.
3. InventoryService: Listens for PaymentProcessed, decrements stock, publishes StockDecremented event.

If InventoryService fails:

• It publishes StockDecrementFailed.
• PaymentService listens for StockDecrementFailed and executes its compensating transaction: refunding the payment. It publishes PaymentRefunded.
• OrderService listens for PaymentRefunded and executes its compensating transaction: marking the order as cancelled.

Configuration Example (Conceptual):

Imagine a message queue (like Kafka or RabbitMQ) orchestrating these events.

• OrderService publishes to order-events topic.
• PaymentService subscribes to order-events, publishes to payment-events.
• InventoryService subscribes to payment-events, publishes to inventory-events.
• Compensation subscriptions are set up similarly (e.g., PaymentService subscribes to stock-decrement-failed).

The mental model: Sagas are about eventual consistency. The system isn’t immediately consistent, but it will become consistent over time as events are processed. The key is that each step is idempotent (can be executed multiple times without changing the result beyond the initial execution) and has a clear compensating action.

Outbox Pattern

The Outbox pattern solves a specific problem within Sagas (or any event-driven system): ensuring that a database transaction and the publishing of an event are atomic. You don’t want to commit to the database and then fail to publish the event, or vice-versa.

How it works:

Instead of directly publishing an event after saving to the database, the application writes the event to an "outbox" table within the same database transaction. A separate process (a "relational event dispatcher") then polls this outbox table and publishes the events to the message broker.

Example Outbox Table Schema:

CREATE TABLE outbox_events (
    id UUID PRIMARY KEY,
    aggregate_type VARCHAR(255) NOT NULL,
    aggregate_id VARCHAR(255) NOT NULL,
    event_type VARCHAR(255) NOT NULL,
    payload JSONB NOT NULL,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
    processed_at TIMESTAMP WITH TIME ZONE NULL
);

Transaction Logic in OrderService:

BEGIN TRANSACTION;

-- Insert order into orders table
INSERT INTO orders (id, customer_id, status) VALUES ('order-123', 'cust-abc', 'PENDING');

-- Insert event into outbox table
INSERT INTO outbox_events (id, aggregate_type, aggregate_id, event_type, payload)
VALUES (gen_random_uuid(), 'Order', 'order-123', 'OrderCreated', '{"customerId": "cust-abc", "status": "PENDING"}');

COMMIT;

Relational Event Dispatcher (Conceptual):

# In a separate process/service
while True:
    events_to_publish = db.query("SELECT * FROM outbox_events WHERE processed_at IS NULL ORDER BY created_at LIMIT 10 FOR UPDATE SKIP LOCKED;")

    for event in events_to_publish:
        try:
            message_broker.publish(topic='order-events', message=event.payload)
            db.execute("UPDATE outbox_events SET processed_at = NOW() WHERE id = :id", {'id': event.id})
            db.commit()
        except Exception as e:
            # Log error, potentially retry later or move to a dead-letter queue
            db.rollback()
            break # Stop processing this batch if one fails
    time.sleep(5) # Poll every 5 seconds

Why it works: The database transaction guarantees that the order record and the outbox event are written atomically. If either fails, neither is written. The separate dispatcher ensures that events are eventually published reliably. This pattern is crucial for making Sagas robust.

The one thing most people don’t realize is that the "relational event dispatcher" process itself needs to be idempotent and fault-tolerant. If it crashes mid-batch, it should be able to restart and pick up where it left off without duplicating messages or losing progress. Using FOR UPDATE SKIP LOCKED in the SQL query is a common way to handle concurrency and prevent multiple dispatchers from picking up the same rows.

The next challenge you’ll face is handling complex compensation logic and ensuring idempotency across all your services.