Distribute One Event to Multiple Consumers with Fan-Out Pattern (2026)

The "fan-out" pattern isn’t just about sending one message to many listeners; it’s fundamentally about decoupling the event producer from the event consumers, allowing them to scale and evolve independently.

Let’s watch this in action. Imagine we have a UserSignup event. Our system needs to notify several downstream services: an email service for welcome messages, a CRM for lead tracking, and a data warehouse for analytics.

Here’s a simplified Kafka setup. We’ll use a topic named user_signups.

// Producer sending the event
{
  "event_type": "UserSignup",
  "timestamp": "2023-10-27T10:00:00Z",
  "payload": {
    "user_id": "user-123",
    "email": "test@example.com",
    "signup_date": "2023-10-27"
  }
}

Now, let’s set up our consumers.

Consumer 1: Email Service

This consumer listens to the user_signups topic. Its job is to send a welcome email.

// Simplified Go consumer for email service
package main

import (
	"fmt"
	"log"

	"github.com/confluentinc/confluent-kafka-go/v2/kafka"
)

func main() {
	consumer, err := kafka.NewConsumer(&kafka.ConfigMap{
		"bootstrap.servers": "kafka-broker1:9092,kafka-broker2:9092",
		"group.id":          "email_service_group",
		"auto.offset.reset": "earliest",
	})
	if err != nil {
		log.Fatalf("Failed to create consumer: %s", err)
	}
	defer consumer.Close()

	err = consumer.SubscribeTopics([]string{"user_signups"}, nil)
	if err != nil {
		log.Fatalf("Failed to subscribe to topics: %s", err)
	}

	for {
		msg, err := consumer.ReadMessage(-1)
		if err == nil {
			fmt.Printf("Email Service received: %s\n", string(msg.Value))
			// Logic to send welcome email based on msg.Value
		} else {
			log.Printf("Consumer error: %v (%v)\n", err, msg)
		}
	}
}

Consumer 2: CRM Service

This consumer also listens to user_signups but adds the user to the CRM.

# Simplified Python consumer for CRM service
from kafka import KafkaConsumer
import json

consumer = KafkaConsumer(
    'user_signups',
    bootstrap_servers=['kafka-broker1:9092', 'kafka-broker2:9092'],
    auto_offset_reset='earliest',
    enable_auto_commit=True,
    group_id='crm_service_group',
    value_deserializer=lambda x: json.loads(x.decode('utf-8'))
)

for message in consumer:
    print(f"CRM Service received: {message.value}")
    # Logic to add user to CRM based on message.value

Consumer 3: Analytics Service

This consumer also listens to user_signups and logs the event for data warehousing.

// Simplified Java consumer for Analytics service
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.serialization.StringDeserializer;

import java.util.Collections;
import java.util.Properties;

public class AnalyticsConsumer {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-broker1:9092,kafka-broker2:9092");
        props.put(ConsumerConfig.GROUP_ID_CONFIG, "analytics_service_group");
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");

        KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
        consumer.subscribe(Collections.singletonList("user_signups"));

        while (true) {
            for (ConsumerRecord<String, String> record : consumer.poll(java.time.Duration.ofMillis(100)).records("user_signups")) {
                System.out.printf("Analytics Service received: %s\n", record.value());
                // Logic to log event for data warehouse
            }
        }
    }
}

The core problem this solves is avoiding a monolithic "event handler" that tries to do everything. By having each consumer focus on a single responsibility, we gain flexibility. If the email service needs to be updated or replaced, it doesn’t affect the CRM or analytics. If the analytics pipeline needs to scale to handle more data, it can do so independently without impacting the others.

The key to fan-out in systems like Kafka is the consumer group. Each consumer group acts as a distinct "listener" for a topic. When a message is published to user_signups, Kafka delivers a copy of that message to every consumer group subscribed to user_signups. Within a single consumer group, however, Kafka ensures that each message is delivered to only one consumer instance (if you have multiple instances of the same consumer type running in parallel for scalability).

This means the email_service_group, crm_service_group, and analytics_service_group all get a full copy of every UserSignup event. The producer just sends one message to the user_signups topic.

The mental model here is that a topic is a log of events, and a consumer group is a process that reads from that log. Multiple independent processes (consumer groups) can read the same log concurrently.

What most people miss is that the group.id is not just an identifier; it’s the key to how Kafka manages delivery within a group. If you have multiple instances of your email service consumer, and they all share the email_service_group ID, Kafka will distribute the messages from user_signups among those instances. This is crucial for load balancing and fault tolerance for a single type of consumer. If you want every consumer type to get every message, each type needs its own, distinct group.id.

The next step you’ll often encounter is needing to ensure exactly-once processing semantics, which requires careful consideration of idempotency and transactional producers/consumers.