Cassandra’s TRACING ON command doesn’t actually trace anything; it just tells the coordinator to trace, and then you have to go fetch the trace data yourself.
Let’s see it in action. Imagine you have a slow query. Your client application might send something like this:
cqlsh> TRACING ON;
cqlsh> SELECT * FROM users WHERE user_id = 123;
The TRACING ON command itself doesn’t immediately produce output. It’s a directive. What happens under the hood is that the cqlsh client adds a special header to the next CQL query it sends. This header signals to the Cassandra coordinator node that it should start recording timing information for that specific request. The coordinator then forwards this tracing request along with the original query to the other nodes involved in fulfilling it. Each node that processes the request, including the coordinator itself, records its own internal timings for various stages of the query execution.
Once the query finishes and returns its results (or an error), the coordinator gathers all the individual trace events recorded by itself and the other participating nodes. It then bundles this trace data and sends it back to the client. The cqlsh client, having received this trace data, will then automatically display it in a human-readable format after the query results.
Here’s what that trace output might look like:
Tracing session: <trace_id>
...
<node_ip>:<port> - <stage_name> (<duration_ms>ms)
<node_ip>:<port> - <stage_name> (<duration_ms>ms)
...
The trace_id is a unique identifier for this specific request’s trace. Each line represents an event recorded by a specific Cassandra node (<node_ip>:<port>) during a particular stage of processing (<stage_name>), along with how long that stage took (<duration_ms>ms).
The fundamental problem TRACING ON solves is diagnosing latency. When a query is slow, you need to know where the time is being spent. Is it network latency between nodes? Is a particular node overloaded? Is a specific stage of query execution (like deserialization, range slicing, or sending results back) taking too long? Tracing provides the granular, per-node, per-stage timing information to pinpoint these bottlenecks.
To understand the mental model, think of a request as a baton in a relay race. The coordinator is the first runner. It passes the baton (the request) to other nodes. Each node performs its leg of the race and then passes the baton back. Tracing is like attaching a stopwatch to the baton at each handover and having each runner record how long their leg took. The coordinator collects all these stopwatch readings and presents them as a timeline.
The key components you control are the level of tracing and the duration it’s enabled. By default, tracing is off. When you issue TRACING ON, you’re enabling it for the next single query. If you want to trace multiple queries, you’d need to re-issue TRACING ON before each one. You can also enable tracing programmatically via your driver, which often provides more control and flexibility for continuous monitoring or targeted tracing.
A common misconception is that TRACING ON itself incurs significant overhead. While tracing does add some overhead, it’s generally negligible for typical operations. The overhead comes from the logging and aggregation of timing data. The actual work of processing the query isn’t significantly impacted unless you’re tracing extremely high-volume, low-latency operations where even a few extra microseconds per request can add up. The real cost is in analyzing the trace data itself, which can become voluminous.
Here’s a crucial detail most people miss: the trace data is ephemeral. By default, Cassandra keeps trace data in memory on the coordinator node for a limited time (controlled by hints_storage_max_size_in_mb in cassandra.yaml and related settings, though this is more for hints than traces directly, the trace storage is managed differently and is often tied to JVM heap or a separate in-memory store). If the coordinator restarts before you retrieve the trace, or if the trace data is evicted due to memory pressure or a high volume of new traces, you’ll lose it. You need to retrieve the trace immediately after the slow query completes. For persistent tracing, you’d typically configure Cassandra to asynchronously write traces to a separate table (often named system_traces).
The next step after identifying a slow query and its bottleneck is to understand how to optimize the query itself or the underlying data model.