Articles

Debug Event-Driven Systems by Replaying Historical Events (2026)

April 16, 2026 8 min read

Replaying historical events in an event-driven system isn’t just about debugging; it’s the closest you’ll get to time travel for your application logic.

Let’s see this in action. Imagine a simple e-commerce order processing system. Events like OrderCreated, PaymentReceived, and InventoryUpdated flow through various microservices.

# Example Event Payload
---
event_type: OrderCreated
timestamp: 2023-10-27T10:00:00Z
data:
  order_id: "ORD12345"
  customer_id: "CUST6789"
  items:
    - product_id: "PROD001"
      quantity: 2
    - product_id: "PROD005"
      quantity: 1

A consumer service, say OrderFulfillmentService, subscribes to OrderCreated and PaymentReceived. If OrderFulfillmentService fails to process an OrderCreated event, and then later a PaymentReceived event arrives, the system might enter an inconsistent state. The payment is received, but the order isn’t yet being fulfilled.

To debug this, we can replay the OrderCreated event for ORD12345 before replaying the PaymentReceived event. This allows us to isolate the OrderFulfillmentService’s behavior under the exact conditions it originally faced, without the cascading effects of later events.

The Core Problem: State Drift and Idempotency Failures

Event-driven systems rely on services reacting to events to update their internal state. The "eventual consistency" model means that at any given moment, different services might have slightly different views of the world. This is normal. The trouble starts when a service fails to process an event correctly, leading to a permanent divergence in state (state drift). This often happens because the service’s handling of the event isn’t truly idempotent – meaning processing the same event multiple times should yield the same result as processing it once, but it doesn’t.

How Replay Solves It

Replaying allows you to rewind the tape and play a specific sequence of events again. You can:

  1. Isolate the failing service: Stop the service from consuming new events.
  2. Extract relevant historical events: Query your event store (e.g., Kafka, Kinesis, EventStoreDB) for the events that led up to the problem.
  3. Reset the service’s state: If necessary, clear the service’s problematic state.
  4. Replay events: Feed the extracted events back into the service, one by one or in batches, in their original order.
  5. Observe and debug: Monitor the service’s behavior and state changes during the replay.

Common Replay Scenarios and Techniques

  • Scenario 1: A Service Crashed Mid-Processing

    • Diagnosis: You’ll often see logs indicating a crash or unhandled exception. The event that was being processed when it crashed is the prime suspect.
    • Replay Command (Conceptual Kafka Example):
      # Find the offset of the problematic event
kafka-console-consumer.sh --topic orders --from-beginning --max-messages 1000 | grep "ORD12345"
# Let's say the problematic event is at offset 550
      Then, to replay just that event to a specific replay topic or a dedicated debug consumer:
      kafka-console-consumer.sh --topic orders --partition 0 --offset 550 --max-messages 1 | kafka-console-producer.sh --topic orders-replay
    • Fix: Modify the service to handle the specific error condition. For instance, if a null pointer exception occurred due to missing data, add a check:
      // Original flawed code
// String shippingAddress = event.getData().getShippingAddress(); // Might be null

// Fixed code
if (event.getData() != null && event.getData().getShippingAddress() != null) {
    String shippingAddress = event.getData().getShippingAddress();
    // ... process address
} else {
    // Log error, dead-letter event, or handle missing data gracefully
    log.error("Shipping address missing for order: {}", event.getData().getOrderId());
}
    • Why it works: The corrected code now gracefully handles the edge case that caused the original crash, preventing future failures on that specific event.

  • Scenario 2: Idempotency Bug

    • Diagnosis: The service processed an event, but due to a bug, it processed it again later (e.g., due to a consumer group rebalance or a retry mechanism that didn’t check for previous processing). This can lead to duplicate actions, like charging a customer twice.
    • Replay Command: Replay the same event multiple times to the service.
      # Assume we have a tool that can produce an event N times
replay-tool --event-id "ORD12345-EVENT_TYPE_ORDER_CREATED" --count 5 --output-topic orders-debug
    • Fix: Implement robust idempotency checks. This usually involves storing a unique identifier for each processed event (e.g., event_id or a combination of event_type and order_id) in a database or cache and checking if it’s already been processed before executing the core business logic.
      -- Example SQL for idempotency check
INSERT INTO processed_events (event_id, processed_at)
SELECT 'ORD12345-ORDERCREATED', NOW()
WHERE NOT EXISTS (
    SELECT 1 FROM processed_events WHERE event_id = 'ORD12345-ORDERCREATED'
);
-- If the INSERT succeeds, proceed with processing. If it fails (because the row already exists), skip.
    • Why it works: The service now explicitly checks if an event has already been handled, preventing duplicate side effects even if the event is delivered multiple times.

  • Scenario 3: Incorrect Business Logic for a Specific State

    • Diagnosis: The service processed events, but the final state is wrong. This implies the logic for a particular sequence of events was flawed.
    • Replay Command: Replay a sequence of events that led to the incorrect state.
      # Replay events for order ORD12345 from creation to the problematic point
replay-tool --order-id "ORD12345" --from-event "OrderCreated" --to-event "ShipmentInitiated" --output-topic orders-debug
    • Fix: Analyze the service’s code that handles the specific sequence and correct the conditional logic or state transitions. For example, if an order status incorrectly transitioned from "Processing" to "Shipped" without a "PaymentReceived" event, fix the state machine logic.
      // Example state transition logic
function transition(currentState, event) {
    switch (currentState) {
        case 'AWAITING_PAYMENT':
            if (event.type === 'PaymentReceived') return 'PROCESSING';
            break;
        case 'PROCESSING':
            if (event.type === 'InventoryAllocated') return 'READY_TO_SHIP';
            break;
        // ... other states
    }
    return currentState; // No valid transition
}
    • Why it works: The corrected state transition logic ensures that the system moves between states only via valid, defined paths, preventing illogical state changes.

  • Scenario 4: External Dependency Failure During Event Processing

    • Diagnosis: The service logs errors related to a downstream or external service (e.g., "Failed to call Inventory API," "Payment Gateway timeout").
    • Replay Command: Replay the event that triggered the external call.
      # Replay a specific event that failed due to external dependency
replay-tool --event-id "EVT98765-PAYMENT_RECEIVED" --output-topic orders-debug
    • Fix: Implement retry mechanisms with backoff for external calls, or ensure the external dependency is available and functioning correctly. If the external dependency was down, fix that first. If it was a transient issue, the retry logic will handle it on replay.
      # Example with retries
import requests
from requests.exceptions import Timeout, ConnectionError
from tenacity import retry, stop_after_attempt, wait_fixed

@retry(stop=stop_after_attempt(3), wait=wait_fixed(5))
def call_inventory_api(item_id, quantity):
    try:
        response = requests.post("https://inventory.example.com/allocate", json={"item_id": item_id, "quantity": quantity})
        response.raise_for_status() # Raise HTTPError for bad responses (4xx or 5xx)
        return response.json()
    except (Timeout, ConnectionError, requests.exceptions.HTTPError) as e:
        print(f"API call failed: {e}. Retrying...")
        raise # Re-raise to trigger tenacity retry

# In your event handler:
try:
    inventory_result = call_inventory_api(event.data.item_id, event.data.quantity)
    # ... process successful inventory update
except Exception as e:
    # Handle ultimate failure after retries
    log.error(f"Failed to update inventory for {event.data.item_id} after multiple retries: {e}")
    # Potentially dead-letter the event
    • Why it works: The retry logic allows the service to attempt the external call multiple times, increasing the chance of success if the external dependency was temporarily unavailable.

  • Scenario 5: Configuration Drift

    • Diagnosis: The system worked fine for months, then suddenly started failing after a configuration change.
    • Replay Command: Replay events that are now failing due to the new configuration.
      # Replay events that use a specific feature affected by config
replay-tool --feature-flag "new_discount_logic" --output-topic orders-debug
    • Fix: Revert the faulty configuration change or correct it. For example, if a new feature flag introduced a bug, disable the flag or fix the code path it enables.
      # Example application.properties
# Original (working)
# feature.new_discount_logic.enabled=false

# New (problematic)
feature.new_discount_logic.enabled=true
      Revert to false or fix the logic associated with true.
    • Why it works: By reverting or correcting the configuration, the service operates with the settings it previously handled correctly, resolving the issue caused by the bad configuration.

  • Scenario 6: Data Corruption in the Event Store

    • Diagnosis: Events themselves are malformed or contain invalid data that wasn’t caught by producers.
    • Replay Command: Replay events and observe validation errors.
      # Replay events and pipe through a validator
kafka-console-consumer.sh --topic orders --from-beginning | event-validator --schema-version 1.2 | kafka-console-producer.sh --topic orders-validated-debug
    • Fix: Fix the producer that generated the corrupt event or implement stricter validation in event consumers. If the event store itself is the source of corruption (rare), you might need to rebuild it from a backup.
      # Example validation in consumer
from jsonschema import validate, ValidationError

order_schema = { ... } # Define your JSON schema

def process_order_created(event):
    try:
        validate(instance=event.data, schema=order_schema)
        # ... proceed with processing
    except ValidationError as e:
        log.error(f"Event data validation failed for order {event.data.order_id}: {e}")
        # Dead-letter the event
    • Why it works: Explicit validation at the consumer level catches malformed events before they cause downstream issues, allowing them to be handled or rejected gracefully.

The power of replay lies in its ability to recreate the exact conditions under which an error occurred. It’s not a substitute for good design (like robust error handling and idempotency), but it’s an indispensable tool for understanding and resolving complex failures in distributed, event-driven systems.

The next thing you’ll likely encounter is managing the scale of replays and ensuring your replay environment doesn’t interfere with production traffic.

Want structured learning?

Take the full Event-driven course →