Cloud Functions are designed to be retried automatically by the underlying infrastructure when they fail, but this can lead to duplicate operations if your function isn’t built to handle it.
Let’s see a simple, non-idempotent function and then make it safe.
import functions_framework
import google.cloud.firestore
db = google.cloud.firestore.Client()
@functions_framework.http
def process_order(request):
request_json = request.get_json()
order_id = request_json.get('order_id')
item_id = request_json.get('item_id')
quantity = request_json.get('quantity')
# This is the problematic part:
# We're just decrementing stock without checking if it's already been done.
item_ref = db.collection('items').document(item_id)
item_data = item_ref.get().to_dict()
new_stock = item_data['stock'] - quantity
item_ref.update({'stock': new_stock})
return 'Order processed', 200
If this process_order function is triggered twice for the same order_id (e.g., due to a transient network error causing a retry), the item_ref.update({'stock': new_stock}) line will run twice. If quantity is 2, and the initial stock is 10, the first execution makes it 8. The second execution, seeing stock at 8, will make it 6. We’ve incorrectly deducted 4 from stock when only 2 were ordered.
To make this idempotent, we need a way to ensure that the core operation (decrementing stock) only happens once per unique request. A common pattern is to use a unique identifier for the operation and store its state.
Here’s the idempotent version:
import functions_framework
import google.cloud.firestore
from google.cloud.firestore_v1.base_client import BaseClient
from google.cloud.firestore_v1.base_document import DocumentReference
db: BaseClient = google.cloud.firestore.Client()
@functions_framework.http
def process_order_idempotent(request):
request_json = request.get_json()
order_id = request_json.get('order_id')
item_id = request_json.get('item_id')
quantity = request_json.get('quantity')
# Use the order_id as a unique identifier for this operation.
# We'll store the status of processing this order.
order_status_ref: DocumentReference = db.collection('order_processing_status').document(order_id)
transaction = db.transaction()
@firestore.transactional
def update_in_transaction(transaction, order_status_ref: DocumentReference, item_id: str, quantity: int):
order_status_doc = order_status_ref.get(transaction=transaction)
# If we've already processed this order, do nothing.
if order_status_doc.exists and order_status_doc.get('processed'):
print(f"Order {order_id} already processed. Skipping.")
return
# Mark the order as being processed or already processed.
# This prevents duplicate processing even if the function is retried
# before the stock update completes.
order_status_ref.set({'processed': True, 'timestamp': firestore.SERVER_TIMESTAMP}, transaction=transaction)
# Now, perform the actual stock update.
item_ref: DocumentReference = db.collection('items').document(item_id)
item_doc = item_ref.get(transaction=transaction)
if not item_doc.exists:
raise ValueError(f"Item {item_id} not found.")
current_stock = item_doc.get('stock')
if current_stock is None or current_stock < quantity:
raise ValueError(f"Insufficient stock for item {item_id}. Available: {current_stock}, Requested: {quantity}")
new_stock = current_stock - quantity
item_ref.update({'stock': new_stock}, transaction=transaction)
print(f"Stock updated for item {item_id}: {current_stock} -> {new_stock}")
try:
update_in_transaction(transaction, order_status_ref, item_id, quantity)
return 'Order processed idempotently', 200
except ValueError as e:
print(f"Error processing order {order_id}: {e}")
# Depending on the error, you might want to roll back the status update
# or handle it differently. For simplicity here, we'll just return an error.
# If the order_status_ref.set was the first operation in the transaction,
# the transaction itself would handle rollback. If it's not part of the
# transaction, explicit rollback logic might be needed.
return f'Error: {e}', 400
except Exception as e:
print(f"An unexpected error occurred: {e}")
return 'Internal Server Error', 500
The core idea is to introduce a "state" record for each operation that needs to be idempotent. In this case, we use a Firestore document in a order_processing_status collection, keyed by the order_id.
When the function is invoked:
- It first checks if a status document for this
order_idalready exists and is marked asprocessed. - If it is, the function exits early, effectively doing nothing.
- If not, it atomically marks the order as
processed(or in the process of being processed) and performs the stock update within a single Firestore transaction.
The firestore.transactional decorator is crucial here. It ensures that the update_in_transaction function runs as an atomic unit. If any part of it fails, the entire transaction is rolled back. By including both the status update and the stock update in the same transaction, we guarantee that either both happen, or neither happens.
The order_status_ref.set({'processed': True, ...}) is the idempotent guard. If the function retries after this line has successfully committed but before the stock update finishes (or if the stock update fails and the function retries), the next invocation will see order_status_doc.exists and order_status_doc.get('processed') as true, and will exit. This prevents the stock from being decremented twice.
The real magic is that this mechanism works even if the function execution is interrupted after the order_status_ref.set but before the item_ref.update. When the function retries, it will read the order_processing_status document, see that it’s already marked as processed, and gracefully exit without performing the stock update again.
The next error you’ll hit is if your order_id is missing from the request payload.