The only way to guarantee event ordering across all consumers for a given event type is to ensure that all events of that type are routed to a single partition, and then have that partition processed by a single consumer at any given moment.
Let’s see this in action. Imagine we have a Kafka topic user_events with 10 partitions. We want to ensure that all user_login events are processed in the exact order they occurred, globally.
Here’s a simplified producer sending user_login events:
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
KafkaProducer<String, String> producer = new KafkaProducer<>(props);
for (int i = 0; i < 100; i++) {
String userId = "user_" + (i % 10); // 10 distinct users
String event = String.format("{\"type\": \"user_login\", \"timestamp\": %d, \"userId\": \"%s\"}", System.currentTimeMillis(), userId);
// The key is crucial for partitioning
producer.send(new ProducerRecord<>("user_events", userId, event));
}
producer.close();
And here’s a consumer group trying to process them:
Properties consumerProps = new Properties();
consumerProps.put("bootstrap.servers", "localhost:9092");
consumerProps.put("group.id", "user_event_processor");
consumerProps.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
consumerProps.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
consumerProps.put("auto.offset.reset", "earliest");
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(consumerProps);
consumer.subscribe(Pattern.compile("user_events"));
while (true) {
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
for (ConsumerRecord<String, String> record : records) {
System.out.printf("Processing: topic = %s, partition = %d, offset = %d, key = %s, value = %s%n",
record.topic(), record.partition(), record.offset(), record.key(), record.value());
// Actual processing logic here...
}
}
If we run this with multiple consumers in the user_event_processor group and the user_id as the key, Kafka’s default partitioner will distribute events for user_0, user_1, etc., across different partitions. This means user_login events for user_0 might end up in partition 3, while user_login events for user_1 might end up in partition 7. If we have multiple consumers, one might get partition 3 and another partition 7. Within a single partition, ordering is guaranteed. However, if consumer A processes partition 3 and consumer B processes partition 7, there’s no guarantee that a user_login event processed by A happened before or after a user_login event processed by B.
The problem this solves is maintaining a strict chronological order for a specific type of event across all instances of that event, regardless of which user or entity generated it. This is critical for scenarios like financial transactions, inventory updates, or any state-dependent process where the sequence of operations is paramount.
Internally, Kafka guarantees ordering within a partition. When a producer sends records with the same key, they are routed to the same partition. If you have a single consumer instance processing that partition, you get guaranteed ordering for all records sent to that partition. The challenge arises when you scale out your consumers; Kafka automatically rebalances partitions among the consumer group. If a partition is assigned to consumer A, and later reassigned to consumer B, consumer B starts processing from the last committed offset. This is fine for idempotency, but not for strict ordering across rebalances.
To achieve global ordering for a specific event type, like user_login, you must ensure all user_login events are directed to the same partition. The simplest way to do this is to use a static key for all these events. For example, you could use a fixed string like "user_login_events_key" as the key for all user_login events:
// In the producer:
producer.send(new ProducerRecord<>("user_events", "user_login_events_key", event));
This forces all user_login events into a single partition (determined by Kafka’s partitioner based on "user_login_events_key"). If you then configure your consumer group to have only one active consumer instance, that single consumer will process all events for that partition, and thus all user_login events, in the order they were produced.
The most surprising thing is that to achieve global ordering for a specific event type, you actively limit your parallelism for that event type to one consumer. The partitioning mechanism, designed for distributing load, becomes the bottleneck when strict global ordering is the absolute requirement. You are essentially sacrificing scalability for the guarantee.
If you have multiple consumers in the group, and only one consumer is assigned to the partition containing all user_login events, Kafka will ensure that consumer processes them in order. However, if you scale up to two consumers, Kafka will rebalance. At that point, the ordering guarantee is broken unless you have a mechanism to handle it. For instance, if the consumer processing the user_login_events_key partition needs to do heavy work, you might offload the actual processing to a separate, single-threaded worker after receiving the event.
The next problem you’ll encounter is how to handle ordering for other event types while still maintaining it for user_login events, especially if you want to scale processing for those other event types.