Dynatrace can correlate logs, traces, and events, but it doesn’t do it by magic; it relies on shared context injected at the instrumentation level.
Let’s see this in action. Imagine you have a request that failed. You see an error in your logs, a broken trace in Dynatrace’s PurePath, and a corresponding event indicating a problem.
Here’s a typical log entry from an application:
2023-10-27 10:30:15.123 ERROR [http-nio-8080-exec-5] com.example.MyService: Failed to process request: Order ID 12345, User ID abcde
And here’s what the corresponding trace might look like in Dynatrace:
- Service:
MyService - Operation:
POST /orders - Duration: 5.2s
- Error:
NullPointerException - PurePath: A tree of method calls, with the
NullPointerExceptionoccurring deep within aPaymentGatewaycall.
And an event might be generated:
- Title:
Error in Payment Gateway - Severity:
ERROR - Source:
MyService - Timestamp:
2023-10-27 10:30:15.123
The magic happens because Dynatrace’s OneAgent automatically injects context into these different telemetry types. When your application is instrumented, OneAgent adds metadata to outgoing requests and logs. For logs, this often means adding specific key-value pairs that Dynatrace can parse. For traces, it’s inherent in the distributed tracing mechanism itself.
The core problem this solves is debugging in complex, distributed systems. When an error occurs, you don’t want to jump between a log viewer, a tracing tool, and an event dashboard. You want a single pane of glass that shows you the entire failure story. Dynatrace achieves this by linking these disparate pieces of information using common identifiers.
Internally, Dynatrace’s OneAgent, when it instruments your application, captures various pieces of context for each transaction. This includes:
- Trace ID: A unique identifier for the entire distributed transaction.
- Span ID: A unique identifier for a specific operation within a trace.
- Request Attributes: HTTP headers, method names, URL paths, etc.
- User Information: If available, the logged-in user ID.
- Custom Tags: User-defined tags applied to services or requests.
When your application logs data, if OneAgent is configured to capture logs (which it is by default for many frameworks), it attempts to enrich these log lines with the current trace context. This enrichment might involve:
- Automatic Injection: OneAgent might automatically prepend or append trace IDs and span IDs to log lines.
- Log Framework Integration: For common logging frameworks (like Logback, Log4j, SLF4j), OneAgent can hook into the logging process and add MDC (Mapped Diagnostic Context) or similar mechanisms with trace information.
- Custom Instrumentation: If you’re using a custom logging mechanism or need to ensure specific data is logged, you can use Dynatrace’s custom instrumentation APIs to explicitly add trace context to your log messages.
For example, if your application uses Logback, OneAgent might ensure that logback.xml is configured to include %X{dt.trace.id} and %X{dt.span.id} in your log pattern. A typical pattern might look like:
<pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n%X{dt.trace.id}%n%X{dt.span.id}</pattern>
This would result in logs like:
2023-10-27 10:30:15.123 ERROR [http-nio-8080-exec-5] com.example.MyService: Failed to process request: Order ID 12345, User ID abcde
trace-1234567890abcdef
span-fedcba0987654321
Dynatrace’s backend then parses these log lines, extracts the trace and span IDs, and links them to the corresponding PurePath.
When an event is generated (e.g., by Dynatrace’s alerting or by an external system integrated with Dynatrace), it can also be enriched with this same context. If an alert is triggered by a service exhibiting high error rates, Dynatrace can automatically associate the relevant trace IDs and request attributes with that event.
The most surprising thing is that this correlation is not just about matching timestamps. While timestamps are used, the primary linkage is through the unique identifiers injected into the transaction flow. A log message could have the exact same timestamp as an event, but without the shared trace ID, they might not be automatically linked. Conversely, a log message from minutes ago could be linked to a current trace if it contains the correct trace ID.
The next step in understanding this system is exploring how to leverage custom attributes for even deeper correlation.