Shovel is a plugin for RabbitMQ that allows you to move messages between brokers. It’s not just a simple copy-paste; it’s a robust, configurable solution for inter-broker communication.
Let’s see it in action. Imagine you have two RabbitMQ instances: broker-a (source) and broker-b (destination). We want to move messages from a queue named source-queue on broker-a to an exchange named dest-exchange on broker-b.
First, ensure the shovel plugin is enabled on both brokers. On broker-a, you’d typically enable it with:
rabbitmq-plugins enable rabbitmq_shovel
And on broker-b:
rabbitmq-plugins enable rabbitmq_shovel
Now, we define the shovel. This is usually done via the RabbitMQ management UI or programmatically. Let’s define a shovel named move-messages.
For the source, we’ll configure:
- Source URI:
amqp://user:password@broker-a:5672/vhost - Source Queue:
source-queue
For the destination, we’ll configure:
- Destination URI:
amqp://user:password@broker-b:5672/vhost - Destination Exchange:
dest-exchange - Destination Routing Key:
my.routing.key(or whatever keydest-exchangeexpects)
Once configured and started, Shovel will begin consuming messages from source-queue on broker-a and publishing them to dest-exchange on broker-b with the specified routing key. The messages are moved, meaning they are acknowledged on the source broker only after they are successfully published to the destination.
The core problem Shovel solves is bridging independent RabbitMQ clusters or even different messaging systems (though primarily designed for AMQP-to-AMQP). This is invaluable for scenarios like:
- Disaster Recovery/High Availability: Replicating critical queues to a secondary broker.
- Data Migration: Gradually moving queues from an old broker to a new one without downtime.
- Federation: Connecting geographically dispersed clusters or creating hybrid cloud deployments.
- Message Aggregation: Consolidating messages from multiple smaller brokers into a central one.
Internally, Shovel operates as a client that establishes connections to both the source and destination brokers. It acts as a consumer on the source queue and a publisher to the destination exchange. The crucial part is its acknowledgment mechanism: a message is only acknowledged on the source after it has been successfully published and acknowledged by the destination broker. This ensures at-least-once delivery semantics between the brokers.
The configuration allows fine-grained control. You can specify:
- Connect timeouts: How long Shovel waits for a broker connection.
- Publish timeouts: How long Shovel waits for a publish confirmation.
- Reconnect intervals: How often Shovel attempts to reconnect if a connection drops.
- TLS/SSL settings: For secure communication.
- Prefetch count: How many messages Shovel prefetches from the source queue.
- Ack mode: Whether to ack messages individually or in batches.
A common, often overlooked, aspect of Shovel configuration is the destination-publish-properties. This allows you to copy or transform message properties from the source message to the destination message. For instance, you might want to preserve the correlation_id or message_id. If you’re not careful, you might end up with messages on the destination that have lost crucial metadata if you don’t explicitly tell Shovel to carry it over.
The next step after mastering basic Shovel is often exploring federation, which offers a more dynamic and decentralized way to manage relationships between brokers.