EventBridge rules are great for routing events, but by default, they send events to Kinesis streams using a static partition key. This can lead to hot partitions and uneven distribution of your data.
Here’s how a real-world scenario looks:
Imagine you have a fleet of IoT devices sending temperature readings to EventBridge. You want to route these readings to a Kinesis Data Stream for real-time processing.
{
"version": "0.0",
"id": "a1b2c3d4-e5f6-7890-1234-567890abcdef",
"detail-type": "DeviceTemperatureReading",
"source": "my.iot.fleet",
"account": "123456789012",
"time": "2023-10-27T10:00:00Z",
"region": "us-east-1",
"resources": [],
"detail": {
"deviceId": "device-001",
"timestamp": "2023-10-27T10:00:00Z",
"temperature": 25.5,
"unit": "Celsius"
}
}
A standard EventBridge rule targeting a Kinesis stream would look something like this in the AWS Console or via CloudFormation/Terraform:
Resources:
MyKinesisStream:
Type: AWS::Kinesis::Stream
Properties:
ShardCount: 2
MyEventBridgeRule:
Type: AWS::Events::Rule
Properties:
Description: "Route IoT temperature readings to Kinesis"
EventBusName: "default"
EventPattern:
source:
- "my.iot.fleet"
detail-type:
- "DeviceTemperatureReading"
State: "ENABLED"
Targets:
- Arn: !GetAtt MyKinesisStream.Arn
Id: "KinesisTarget"
KinesisParameters:
PartitionKeyPath: "$.detail.deviceId" # This is the key!
The KinesisParameters.PartitionKeyPath is where the magic happens. By setting this to $.detail.deviceId, EventBridge will extract the deviceId from the incoming event’s detail field and use that as the partition key when writing to Kinesis. This ensures that all events from a single deviceId go to the same shard, providing ordering guarantees for that device’s data.
The problem this solves is that without a carefully chosen partition key, Kinesis might receive a skewed distribution of data. If all your events happened to share the same deviceId (or whatever field you’re using), they’d all land on one shard. This single shard would become a bottleneck, processing all the writes and reads, while other shards sit idle. This leads to throttling errors in Kinesis and poor performance.
When you select a field like deviceId, userId, or transactionId that is unique or has high cardinality within your event data, you’re telling EventBridge to dynamically generate the partition key for each event. This distributes the incoming data across your Kinesis shards more evenly, maximizing throughput and minimizing the chance of hitting shard limits.
If you don’t specify a PartitionKeyPath, EventBridge uses a default, which is not guaranteed to provide good distribution. The default partition key is derived from the event ID, which for a large number of distinct events, might provide reasonable distribution, but it’s far better to be explicit.
The PartitionKeyPath uses JSONPath syntax. So, if your device ID was nested deeper, like $.eventData.iotInfo.serialNumber, you’d use that. EventBridge parses this path and uses the value found at that location as the partition key.
Consider what happens if the PartitionKeyPath you specify doesn’t exist in an event. EventBridge will default to using the event ID as the partition key for that specific event. This is a safety net to prevent errors, but it means that some events might not be distributed as you intended.
The KinesisParameters block in the target configuration is where you define how EventBridge interacts with Kinesis. Beyond PartitionKeyPath, you can also specify MessageGroupId if you’re targeting Kinesis Data Streams in FIFO mode, which is crucial for ordered processing within a specific group.
You can also use EventBridge Pipes to achieve similar results, often with more complex transformations or routing logic before events reach Kinesis. Pipes offer a more managed way to connect AWS services and can sometimes simplify the setup compared to a direct EventBridge rule, especially if you need to enrich or filter events before sending them to Kinesis.
The next thing you’ll likely encounter is managing Kinesis stream scaling and monitoring, especially as your event volume grows.