Creating custom APM transactions and spans in your code lets you instrument the parts of your application that standard APM agents might miss, giving you granular visibility into specific business logic or complex workflows.
Let’s say you’re building an e-commerce platform. When a user clicks "Add to Cart," the default APM might just show a POST /cart transaction. But what if that "Add to Cart" involves several distinct steps: checking inventory, applying a discount, updating a user’s session, and then finally persisting to the database? You can use custom instrumentation to break down that single POST /cart into a transaction with multiple spans, each representing one of those steps.
Here’s how it looks in practice, using the popular OpenTelemetry Python SDK:
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
# Configure the tracer provider
provider = TracerProvider()
tracer = provider.get_tracer(__name__)
# Configure the span exporter (e.g., to an OpenTelemetry Collector)
# Replace with your collector's address
span_processor = BatchSpanProcessor(OTLPSpanExporter(endpoint="localhost:4317"))
provider.add_span_processor(span_processor)
# Set the tracer provider as the global tracer provider
trace.set_tracer_provider(provider)
def add_to_cart(user_id, product_id, quantity):
# Start a new transaction (root span) for the "Add to Cart" operation
with tracer.start_as_current_span("Add to Cart") as current_span:
current_span.set_attribute("user.id", user_id)
current_span.set_attribute("product.id", product_id)
current_span.set_attribute("quantity", quantity)
# --- Custom Span 1: Check Inventory ---
with tracer.start_as_current_span("Check Inventory") as inventory_span:
inventory_span.set_attribute("product.id", product_id)
# Simulate inventory check logic
inventory_available = check_stock(product_id, quantity)
inventory_span.add_event("Inventory check complete", {"available": inventory_available})
if not inventory_available:
current_span.set_attribute("status", "failed")
current_span.record_exception(Exception("Insufficient stock"))
return False
# --- Custom Span 2: Apply Discount ---
with tracer.start_as_current_span("Apply Discount") as discount_span:
discount_span.set_attribute("user.id", user_id)
# Simulate discount logic
discount_amount = calculate_discount(user_id, product_id)
discount_span.set_attribute("discount.amount", discount_amount)
# --- Custom Span 3: Update Session ---
with tracer.start_as_current_span("Update User Session") as session_span:
session_span.set_attribute("user.id", user_id)
# Simulate session update logic
update_user_cart_session(user_id, product_id, quantity)
# --- Custom Span 4: Persist to Database ---
with tracer.start_as_current_span("Persist Cart Item") as db_span:
db_span.set_attribute("user.id", user_id)
db_span.set_attribute("product.id", product_id)
# Simulate database persistence
save_cart_item_to_db(user_id, product_id, quantity)
db_span.add_event("Cart item saved to DB")
current_span.set_attribute("status", "success")
return True
# Dummy functions for demonstration
def check_stock(product_id, quantity):
print(f"Checking stock for {product_id} (quantity: {quantity})...")
return True # Assume stock is available
def calculate_discount(user_id, product_id):
print(f"Calculating discount for user {user_id}, product {product_id}...")
return 5.00 # Assume a $5 discount
def update_user_cart_session(user_id, product_id, quantity):
print(f"Updating session for user {user_id}, adding {quantity} of {product_id}...")
def save_cart_item_to_db(user_id, product_id, quantity):
print(f"Saving {quantity} of {product_id} to DB for user {user_id}...")
# Example usage:
if __name__ == "__main__":
add_to_cart("user123", "prod456", 2)
In this example, add_to_cart becomes the main transaction. Each with tracer.start_as_current_span(...) block creates a new span, nested within the parent span. These spans are automatically linked by the tracer. You can add attributes (key-value pairs) to spans to provide context and events to mark significant points within a span’s execution.
The power here is that you can now visualize the "Add to Cart" operation not as a black box, but as a sequence of well-defined, timed steps. You can see exactly how long "Check Inventory" took, how long "Apply Discount" took, and so on. If "Check Inventory" suddenly starts taking 5 seconds instead of 50 milliseconds, you’ve immediately pinpointed the bottleneck.
The core concept is that the tracer object allows you to manually define the boundaries of your work. A "transaction" in APM terms is simply the root span of a trace that represents a meaningful unit of work, often initiated by an incoming request. Custom spans allow you to subdivide that work. The start_as_current_span context manager is key because it automatically manages the start and end times of the span and ensures it’s correctly linked to its parent, or established as a root span if no parent is active.
A common pitfall is forgetting to set attributes on your spans. While the span name tells you what happened, attributes tell you about what happened. For instance, a span named "Process Order" is less useful than a span named "Process Order" with attributes like order.id: "ORD-7890", customer.country: "DE", and order.total: 150.75. This rich context is crucial for effective debugging and performance analysis.
Once you’ve mastered custom transactions and spans, the next logical step is often distributed tracing, where you need to ensure these custom spans correctly propagate across service boundaries.