Distributed tracing in Azure Functions isn’t just about seeing where your requests went; it’s about understanding the emergent behavior of your microservices.
Let’s see it in action. Imagine a simple HTTP-triggered Azure Function (HttpTriggerFunction) that calls another Azure Function (QueueTriggerFunction) by sending a message to a Service Bus queue. The QueueTriggerFunction then processes this message and writes to a Cosmos DB.
Here’s a simplified view of the code:
HttpTriggerFunction (C#):
using Azure.Messaging.ServiceBus;
using Microsoft.Azure.Functions.Worker;
using Microsoft.Azure.WebJobs.Extensions.OpenApi.Core.Attributes;
using Microsoft.Extensions.Logging;
using Microsoft.Azure.Functions.Worker.Http;
using System.Net;
public class HttpTriggerFunction
{
private readonly ServiceBusSender _serviceBusSender;
private readonly ILogger<HttpTriggerFunction> _logger;
public HttpTriggerFunction(ServiceBusClient serviceBusClient, ILogger<HttpTriggerFunction> logger)
{
_serviceBusSender = serviceBusClient.CreateSender("my-service-bus-queue");
_logger = logger;
}
[Function("HttpTriggerFunction")]
[OpenApiOperation(operationId: "Run", tags: new[] { "http" })]
[OpenApiResponse(statusCode: HttpStatusCode.Accepted, description = "The Accepted response")]
public async Task<HttpResponseData> Run([HttpTrigger(AuthorizationLevel.Function, "get", "post")] HttpRequestData req)
{
_logger.LogInformation("C# HTTP trigger function processed a request.");
var messageContent = "{\"id\": \"12345\", \"data\": \"sample data\"}";
var message = new ServiceBusMessage(messageContent);
await _serviceBusSender.SendMessageAsync(message);
_logger.LogInformation($"Sent message to Service Bus queue: {messageContent}");
var response = req.CreateResponse(HttpStatusCode.Accepted);
response.WriteString("Request accepted and message sent to queue.");
return response;
}
}
QueueTriggerFunction (C#):
using Azure.Cosmos;
using Microsoft.Azure.Functions.Worker;
using Microsoft.Extensions.Logging;
using System.Text.Json;
public class QueueTriggerFunction
{
private readonly CosmosClient _cosmosClient;
private readonly ILogger<QueueTriggerFunction> _logger;
public QueueTriggerFunction(CosmosClient cosmosClient, ILogger<QueueTriggerFunction> logger)
{
_cosmosClient = cosmosClient;
_logger = logger;
}
[Function("QueueTriggerFunction")]
public async Task Run([ServiceBusTrigger("my-service-bus-queue", Connection = "ServiceBusConnection")] ServiceBusReceivedMessage message)
{
_logger.LogInformation($"C# ServiceBus queue trigger function processed message: {message.Body}");
var messageData = JsonSerializer.Deserialize<MyMessageData>(message.Body.ToString());
var container = _cosmosClient.GetDatabase("my-database").GetContainer("my-container");
await container.CreateItemAsync(messageData, new PartitionKey(messageData.id));
_logger.LogInformation($"Message data saved to Cosmos DB.");
}
}
public class MyMessageData
{
public string id { get; set; }
public string data { get; set; }
}
When you enable Application Insights and configure distributed tracing, requests flowing through these functions will be captured. You’ll see an entry for the HttpTriggerFunction execution, and then a subsequent entry for the QueueTriggerFunction execution, linked by a common Operation Id. The trace view in Application Insights will visually represent this flow, showing the duration of each function and the time spent waiting for dependencies like Service Bus and Cosmos DB.
The core problem distributed tracing solves is the "black box" problem in distributed systems. When a request fails or is slow, you don’t just see that an error occurred; you see where it occurred and what it was doing just before. This allows you to pinpoint bottlenecks, identify cascading failures, and understand the end-to-end latency of a transaction across multiple services.
Internally, Azure Functions integrates with Application Insights via the Microsoft.Azure.Functions.Worker.ApplicationInsights NuGet package. When this package is present and configured with an Application Insights instrumentation key, the Functions host automatically emits telemetry. For distributed tracing, it relies on the Request and Dependency telemetry types.
- Request Telemetry: This is emitted for incoming requests to your function (e.g., HTTP triggers). It includes the operation ID, request name, duration, and success status.
- Dependency Telemetry: This is emitted when your function makes an outgoing call to another service (e.g., Service Bus, Cosmos DB, other HTTP endpoints). It captures the dependency type, target, operation name, duration, and success.
Crucially, the Operation Id is propagated across these telemetry events. For outbound calls made by Azure Functions SDKs (like Azure.Messaging.ServiceBus or Azure.Cosmos), the SDKs themselves often automatically inject tracing headers (like traceparent and tracestate for W3C Trace Context) if they detect an Application Insights SDK or a compatible distributed tracing setup. This allows the downstream service (or another Azure Function) to pick up the same Operation Id and continue the trace.
The exact levers you control are primarily within your host.json and local.settings.json (or Azure App Settings).
In host.json, you configure Application Insights:
{
"version": "2.0",
"logging": {
"applicationInsights": {
"samplingSettings": {
"isEnabled": true,
"excludedTypes": "Request"
},
"enableDependencyTracking": true,
"enableLiveMetricsFilters": true
}
},
"extensions": {
"serviceBus": {
"messageHandlerOptions": {
"autoCompleteMessages": false,
"maxConcurrentCalls": 1,
"maxConcurrentSessions": 1
}
}
}
}
The enableDependencyTracking flag is key here. For HttpTrigger and ServiceBusTrigger functions, the host automatically generates Request telemetry. For ServiceBus and CosmosDB calls made using the Azure SDKs within your function code, the enableDependencyTracking setting helps ensure that the Dependency telemetry is captured. If you’re making custom HTTP calls using HttpClient, you’ll need to manually configure HttpClient to participate in distributed tracing, often by using an DelegatingHandler that injects the current Operation Id into outgoing request headers.
The samplingSettings can be adjusted to control how much telemetry is sent, which is important for managing costs and performance in high-throughput scenarios. excludedTypes can prevent certain types of telemetry from being sampled or even collected, if desired.
The most surprising thing about how distributed tracing works with Azure Functions is how seamlessly it integrates with many Azure SDKs, often requiring minimal explicit code. The Azure SDKs, when they detect an Application Insights environment variable (like APPLICATIONINSIGHTS_CONNECTION_STRING or APPINSIGHTS_INSTRUMENTATIONKEY) and the Microsoft.Azure.Functions.Worker.ApplicationInsights package is present, will automatically start and propagate trace context. This means that calls made to Azure Service Bus, Cosmos DB, Event Hubs, and others using their respective SDKs will often show up as dependencies in your traces without you needing to write any specific tracing code for those dependency calls. The Azure Functions host itself acts as a context carrier, and the SDKs are designed to pick up and continue that context.
The next concept you’ll want to explore is correlation. When you have multiple, distinct Operation Ids that are logically part of a larger business transaction, you’ll want to learn how to link them using custom properties and measurements.