Adding custom spans to your Python code with ddtrace is how you get visibility into specific, business-critical operations that aren’t automatically captured by libraries.
Here’s how a typical request flows through a Python app, and where custom spans fit in.
from ddtrace import tracer
def process_order(order_id):
with tracer.trace("order_processing", service="my_app", resource=f"order_{order_id}"):
# Simulate fetching order details
order_details = fetch_order_from_db(order_id)
with tracer.trace("db_fetch", service="my_app", resource="fetch_order"):
# Actual database call would go here
pass # Placeholder for DB call
# Simulate calling an external service
payment_result = charge_customer(order_details)
with tracer.trace("payment_service", service="my_app", resource="charge_customer"):
# Actual external API call would go here
pass # Placeholder for API call
# Update order status
update_order_status(order_id, "processed")
with tracer.trace("db_update", service="my_app", resource="update_order_status"):
# Actual database call would go here
pass # Placeholder for DB call
return payment_result
# Example usage:
# process_order("ORD12345")
This process_order function is a good candidate for manual instrumentation. Without custom spans, you’d see a single span for the entire request. By adding tracer.trace blocks, we break down the operation into meaningful sub-segments: db_fetch, payment_service, and db_update. Each with tracer.trace(...) block creates a new span. The arguments service and resource are crucial for organizing and filtering traces in Datadog. service identifies the application component, and resource provides a specific operation name within that service.
The primary problem ddtrace instrumentation solves is the "black box" nature of complex applications. When a request is slow or fails, it’s hard to pinpoint which part of your code is the bottleneck or the source of the error. Automatic instrumentation covers common libraries (like requests, psycopg2, django), but for bespoke business logic, you need to tell ddtrace what’s important.
Internally, ddtrace uses a tracer object that manages the creation and completion of spans. When you enter a with tracer.trace(...) block, a span is started with a start time. When you exit the block (either normally or due to an exception), the span is finished, its duration is calculated, and it’s queued for sending to the Datadog Agent. The service and resource names become searchable tags on the span, allowing you to filter and analyze your traces effectively.
Here’s how you’d configure the tracer to send data. You’d typically do this once at your application’s entry point.
from ddtrace import config, tracer
from ddtrace.propagation.http import HTTPPropagator
# Configure the tracer
config.service = 'my_python_app'
config.env = 'staging'
config.version = '1.2.0'
config.analytics_enabled = True # Enable distributed tracing for analytics
# Optionally configure the agent host and port if not default
# config.agent_hostname = '127.0.0.1'
# config.agent_port = 8126
# Ensure HTTP propagation is enabled for distributed tracing
tracer.configure(
hostname=config.agent_hostname,
port=config.agent_port,
propagation_style=HTTPPropagator()
)
# Now, any spans created will be associated with 'my_python_app' in staging
The service name you set globally here (my_python_app) will be the default for all spans, but you can override it per span, as shown in the process_order example. The env and version tags are vital for segmenting your traces by deployment environment and application version, making it easier to compare performance across different deployments. Enabling analytics_enabled is key if you want to use Datadog’s APM analytics features, which aggregate trace data for insights.
The most surprising thing about custom spans is how little overhead they actually add when used judiciously. People often worry about performance impact, but the overhead of creating and completing a span is minimal, typically measured in microseconds. The real benefit comes from the insights gained, which far outweighs the slight computational cost. You’re not just measuring time; you’re giving context to that time.
If you want to add metadata to your custom spans beyond just service and resource, you can use the span.set_tag() method.
from ddtrace import tracer
def process_payment(order_id, amount, currency):
with tracer.trace("payment_processing") as span:
span.service = "payment_gateway"
span.resource = "process_credit_card"
span.set_tag("order.id", order_id)
span.set_tag("payment.amount", amount)
span.set_tag("payment.currency", currency)
span.set_tag("payment.method", "credit_card")
# ... actual payment processing logic ...
success = True # Simulate success
if not success:
span.set_tag("error", "Payment failed")
span.set_status("error") # Explicitly set status to error
return success
Here, we’re attaching order.id, payment.amount, payment.currency, and payment.method as tags. These tags become searchable fields in Datadog, allowing you to filter traces based on specific order IDs, amounts, or payment methods. Setting span.set_status("error") and adding an error tag is crucial for marking failed operations, which automatically surfaces them in error tracking views.
The next step after mastering custom spans is understanding distributed tracing context propagation.