Adding manual spans to your Go application with the Elastic APM agent allows you to instrument code paths that the automatic instrumentation might miss, giving you deeper insight into specific operations.
Let’s see it in action. Imagine you have a function that fetches data from an external API, processes it, and then writes it to a database. The automatic instrumentation might capture the overall function duration, but you want to know how long just the API call took, or just the database write.
package main
import (
"fmt"
"net/http"
"time"
"go.elastic.co/apm/v2"
)
func main() {
// Initialize the APM agent (simplified for example)
tracer, err := apm.NewTracer("my-go-app", "1.0.0")
if err != nil {
panic(err)
}
defer tracer.Stop()
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
span, ctx := apm.StartSpanFromContext(ctx, "external_api_call")
span.Label("url", "https://api.example.com/data")
// Simulate API call
time.Sleep(150 * time.Millisecond)
span.End()
span, ctx = apm.StartSpanFromContext(ctx, "process_data")
// Simulate data processing
time.Sleep(100 * time.Millisecond)
span.End()
span, ctx = apm.StartSpanFromContext(ctx, "database_write")
span.Label("table", "users")
// Simulate database write
time.Sleep(200 * time.Millisecond)
span.End()
fmt.Fprintln(w, "Request processed")
})
http.ListenAndServe(":8080", nil)
}
In this example, apm.StartSpanFromContext(ctx, "external_api_call") creates a new span named "external_api_call" that is a child of the current transaction or parent span. We then perform our simulated API work and call span.End() to mark its completion. The Label calls add key-value metadata to the span, which is invaluable for filtering and analysis in Kibana.
The core problem this solves is granularity. Automatic instrumentation captures the "what" – the request entered the handler, a database query was run. Manual instrumentation lets you capture the "how long" for specific, business-critical operations within your code that might not align perfectly with the automatically instrumented boundaries. Think of it as adding checkpoints to your code’s execution path to measure the duration of distinct logical steps.
Internally, the go.elastic.co/apm agent manages a tree of spans for each transaction. When you start a span with apm.StartSpanFromContext, the agent ensures it’s correctly linked to its parent (either the current transaction or another span). When span.End() is called, the agent calculates the duration and attaches any labels before sending it to the APM server.
The primary levers you control are the span names and the labels. Span names should be descriptive and consistent. Instead of span1, span2, use fetch_user_profile, render_template, validate_input. Labels are where you add context. For a database span, db.statement, db.type, table_name are common. For an HTTP client span, http.method, http.url, http.status_code are useful.
The apm.StartSpan function, without a context, creates a root span. This is less common when instrumenting within an existing transaction, but it’s useful for standalone tasks or background jobs that you want to track as their own distinct unit of work, independent of an incoming web request.
The most surprising thing is how little overhead adding many small manual spans typically introduces. The agent is highly optimized for this, and the primary cost is context switching and the brief moment of capturing start/end times and associated metadata. Unless you’re creating thousands of spans per request in a very tight loop, the performance impact is usually negligible compared to the visibility gained.
Next, you’ll want to explore how to propagate these manually created spans across service boundaries using distributed tracing.