Dynatrace doesn’t just passively collect metrics; it actively pulls them from your services if you configure it right, and that’s how you’ll get your custom metrics in.
Let’s see this in action. Imagine you’ve got a Python app that’s crunching numbers and you want to track how many records it processes per minute.
from time import sleep
from dt_metrics import DtMetrics
# Assuming DT_TENANT and DT_API_TOKEN are set as environment variables
# and DT_INGEST_URL is pointing to your Dynatrace endpoint (e.g., https://{your-tenant}/api/v2/metrics/ingest)
metrics = DtMetrics()
def process_records():
records_processed_this_minute = 0
while True:
# Simulate processing some records
sleep(0.1)
records_processed_this_minute += 1
# Every 60 seconds, send the metric and reset the counter
if records_processed_this_minute % 600 == 0: # roughly 60 seconds
metric_name = "custom.myapp.records_processed_per_minute"
metric_value = records_processed_this_minute
dimensions = {"environment": "production", "service": "data_cruncher"}
metrics.gauge(metric_name, metric_value, dimensions=dimensions)
print(f"Sent metric: {metric_name}={metric_value} with dimensions {dimensions}")
records_processed_this_minute = 0
sleep(0.04) # Adjust sleep to control loop speed
if __name__ == "__main__":
process_records()
This code uses the dt_metrics Python library, which simplifies sending data in Dynatrace’s Line Protocol format. The core idea is to define a metric name (like custom.myapp.records_processed_per_minute), assign it a value (the count), and attach key-value pairs called dimensions for filtering and grouping (like environment and service).
The magic behind Dynatrace’s custom metrics lies in its ActiveGate or OneAgent acting as a gateway. Instead of your application pushing data to Dynatrace’s API directly, Dynatrace’s agents are configured to scrape or pull metrics from endpoints you expose. For custom metrics, you typically expose an endpoint (often a simple HTTP server) that serves data in the Line Protocol format. The ActiveGate/OneAgent then periodically fetches this data and forwards it to your Dynatrace tenant. This pull model is great because it decouples your application from Dynatrace’s infrastructure, and Dynatrace controls the collection frequency.
Here’s how you’d typically set this up:
-
Expose Metrics Endpoint: Your application needs to serve metrics on a specific port and path. A common pattern is to use a library that handles this, like the
dt_metricslibrary shown above or Prometheus client libraries, which can often be configured to output Line Protocol. For instance, you might exposehttp://localhost:9091/metrics. -
Configure Dynatrace Agent (ActiveGate/OneAgent): You’ll tell your Dynatrace agent to scrape this endpoint. This is done via configuration files. For an ActiveGate, you might edit
config/custom.propertiesand add lines like:com.dynatrace.diagnostics.agent.metric.ingestion.protocols.http.enabled=true com.dynatrace.diagnostics.agent.metric.ingestion.protocols.http.endpoint.<your_endpoint_name>.url=http://localhost:9091/metrics com.dynatrace.diagnostics.agent.metric.ingestion.protocols.http.endpoint.<your_endpoint_name>.type=LINEPROTOCOL com.dynatrace.diagnostics.agent.metric.ingestion.protocols.http.endpoint.<your_endpoint_name>.interval_seconds=60Replace
<your_endpoint_name>with a descriptive name for your metric source (e.g.,myapp_metrics). Theinterval_secondsdictates how often the agent checks for new data. -
Define Metric in Dynatrace (Optional but Recommended): While Dynatrace will discover metrics automatically, defining them in the UI under "Metrics" -> "Custom Metrics" -> "Add Metric" allows you to set descriptions, units, and aggregation behaviors upfront, making them easier to find and analyze later. You’d specify the metric name (
custom.myapp.records_processed_per_minute) and dimensions.
Once configured, the ActiveGate/OneAgent will poll your /metrics endpoint every 60 seconds. It will parse the Line Protocol data, extract the metric name, value, and dimensions, and send it to your Dynatrace tenant. You can then view these metrics in dashboards, set up alerts, and correlate them with other data Dynatrace collects.
The Line Protocol itself is quite simple: metric_name,tag_key1=tag_value1,tag_key2=tag_value2 value field_key=field_value timestamp. For custom metrics ingested via the HTTP endpoint, Dynatrace often simplifies this for you, expecting just metric_name,dimension1=value1,dimension2=value2 metric_value. The library handles the timestamp.
For example, the dt_metrics library might generate a string like this for your records_processed_per_minute metric:
custom.myapp.records_processed_per_minute,environment=production,service=data_cruncher 120
The agent picks this up, and Dynatrace associates it with your host and process.
The one thing most people don’t realize is that the value in Line Protocol can also be a floating-point number or even a boolean, and you can send multiple measurements for the same metric name in a single payload, separated by newlines, which is highly efficient for batching.
After successfully ingesting custom metrics, the next challenge is often correlating them with trace data to understand why your record processing rate might be fluctuating.