Cloud Pub/Sub is surprisingly bad at reliably delivering messages to all subscribers simultaneously, and that’s precisely why it’s so good at building event-driven systems.
Let’s see it in action. Imagine we have a simple web application that needs to notify a few downstream services whenever a new user signs up. We can use Pub/Sub to decouple these services.
Here’s a producer sending a message:
from google.cloud import pubsub_v1
publisher = pubsub_v1.PublisherClient()
topic_path = publisher.topic_path("your-gcp-project-id", "user-signup-topic")
def publish_user_signup(user_id):
data = f"New user signed up: {user_id}".encode("utf-8")
future = publisher.publish(topic_path, data)
print(f"Published message ID: {future.result()}")
publish_user_signup("user123")
And here’s a subscriber receiving it:
from google.cloud import pubsub_v1
import time
subscriber = pubsub_v1.SubscriberClient()
subscription_path = subscriber.subscription_path("your-gcp-project-id", "user-signup-subscription")
def callback(message):
print(f"Received message: {message.data.decode('utf-8')}")
message.ack() # Acknowledge the message
streaming_pull_future = subscriber.subscribe(subscription_path, callback=callback)
print("Listening for messages...")
try:
# Keep the main thread alive for at least 60 seconds
streaming_pull_future.result(timeout=60)
except TimeoutError:
streaming_pull_future.cancel()
streaming_pull_future.result() # Block until cancel is complete
The core problem Pub/Sub solves is reliably getting a message from a publisher to potentially many subscribers, even if those subscribers are built with different technologies, run in different environments, or are temporarily unavailable. It acts as a central nervous system for your applications. When a message is published, Pub/Sub stores it and makes it available to all active subscriptions. Subscribers then pull or receive messages from their respective subscriptions. The beauty is that the publisher doesn’t need to know anything about the subscribers, and subscribers don’t need to know about each other.
The internal mechanism involves topics and subscriptions. A topic is a named resource to which messages are sent. A subscription is an entity that represents a message consumer’s interest in messages from a particular topic. When you create a subscription, Pub/Sub starts delivering messages published to the associated topic to that subscription. Pub/Sub handles the heavy lifting of message buffering, ordering (within certain limits), and guaranteed delivery (via acknowledgments).
The key levers you control are:
- Topic Configuration: You define the topic name and any associated schema.
- Subscription Configuration: This is where the magic happens. You set
ack_deadline(how long a subscriber has to acknowledge a message before Pub/Sub redelivers it),message_retention_duration(how long Pub/Sub keeps unacknowledged messages), and delivery types (pull vs. push). You can also configure dead-lettering, which is crucial for handling messages that repeatedly fail to be processed. - Publisher Behavior: How you format your message data, whether you use attributes for metadata, and how you handle publish errors.
- Subscriber Behavior: How quickly your subscriber acknowledges messages, how it handles errors, and its retry logic for processing failed messages.
What most people miss is how Pub/Sub’s internal retry and redelivery mechanisms, driven by the ack_deadline, actually enable fault tolerance. If a subscriber receives a message but crashes before acknowledging it, Pub/Sub will, after the ack_deadline expires, redeliver that same message to another instance of the subscriber (or the same one if it recovers). This is why your subscribers must be idempotent: they should be able to process the same message multiple times without causing unintended side effects. For example, if your subscriber is a function that charges a credit card, it must have logic to check if that specific transaction has already been processed before attempting to charge again.
The next logical step after building reliable event producers and consumers is managing message ordering guarantees.