Track and Fix Flaky Tests with Datadog CI Test Visibility (2026)

Datadog CI Test Visibility is failing to report test results, leading to a complete blackout of your CI/CD pipeline’s performance.

Common Causes and Fixes:

Datadog Agent Not Running or Misconfigured:
- Diagnosis: Check the Datadog agent status on your CI runners. For Docker, this would be docker ps and looking for the Datadog agent container. For host-based agents, sudo datadog-agent status. If it’s not running or shows errors, investigate agent logs (/var/log/datadog/agent.log on Linux).
- Fix: Ensure the Datadog agent is started and healthy. For Docker, this might involve restarting the container: docker restart <datadog-agent-container-id>. For host agents, sudo datadog-agent start. Verify your datadog.yaml configuration, especially api_key and site.
- Why it works: The Datadog agent is the gateway for all telemetry data, including test results, to reach Datadog. If it’s down or misconfigured, nothing gets sent.
Incorrect Datadog API Key or Site:
- Diagnosis: In your CI environment, inspect the Datadog API key and site configuration. This is often set via environment variables like DD_API_KEY and DD_SITE. Manually try to ping the Datadog API endpoint from your CI runner using curl -v https://api.datadoghq.com/api/v1/validate?api-key=<your-api-key>. A successful response is {"status":"Success"}. If using a government cloud, replace datadoghq.com with your specific site (e.g., datadoghq.eu).
- Fix: Correct the DD_API_KEY environment variable in your CI configuration to your actual Datadog API key. Ensure DD_SITE is set correctly for your Datadog account region (e.g., datadoghq.com for US1, datadoghq.eu for EU1, us3.datadoghq.com for US3, etc.).
- Why it works: The API key authenticates your account, and the site directs the data to the correct Datadog datacenter. An incorrect key or site means your data is sent to the wrong place or rejected.
Datadog Test Runner Configuration Missing or Incorrect:
- Diagnosis: Examine your CI job configuration files (e.g., .gitlab-ci.yml, Jenkinsfile, GitHub Actions workflow .yml). Look for how the Datadog tracer or dd-trace CLI is initialized. Specifically, check for environment variables like DD_TEST_FRAMEWORK, DD_TEST_LEVEL, DD_TEST_SERVICE, and DD_TEST_MODULE.
- Fix: Ensure the necessary Datadog environment variables are exported in your CI script before your tests run. For example, in a Node.js project using Jest:
```
export DD_TEST_FRAMEWORK="jest"
export DD_TEST_SERVICE="my-node-app"
export DD_TEST_MODULE="my-tests"
# Run your tests
npm test
```
  For Python with pytest:
```
export DD_TEST_FRAMEWORK="pytest"
export DD_TEST_SERVICE="my-python-app"
export DD_TEST_MODULE="my-tests"
# Run your tests
pytest
```
- Why it works: These variables instruct the Datadog tracer how to instrument your specific test runner and what metadata to associate with the test results. Without them, the tracer doesn’t know what to capture or where to send it.
Datadog Tracer Not Properly Injected/Initialized:
- Diagnosis: Verify that the Datadog tracer is being loaded and initialized before your test suite starts executing. This often involves modifying your test runner’s entry point or using a wrapper script. For example, if you’re using dd-trace CLI with Node.js: DD_TEST_FRAMEWORK=jest node -r dd-trace/init node_modules/.bin/jest. Check your CI logs for any errors related to dd-trace initialization.
- Fix: Ensure the dd-trace library is installed (npm install --save-dev dd-trace or pip install dd-trace) and that it’s being required or invoked correctly. For Node.js, this might be require('dd-trace').init(); at the top of your test setup file. For Python, it’s often done via environment variable DD_TRACE_ENABLED=true.
- Why it works: The tracer needs to actively hook into your test runner’s execution flow to capture test events. If it’s not initialized or loaded correctly, it can’t intercept or report anything.
Firewall or Network Restrictions Blocking Datadog Agent Communication:
- Diagnosis: From your CI runner, attempt to curl the Datadog intake endpoint for your region. For example, for US1: curl -v https://ingest.logs.datadoghq.com. Check for connection timeouts or explicit rejections. Review your CI environment’s network policies, VPC security groups, or firewall rules.
- Fix: Open outbound connections from your CI runners to Datadog’s intake endpoints on ports 80 and 443. The specific hostnames depend on your Datadog site (e.g., ingest.logs.datadoghq.com, trace.agent.datadoghq.com). Consult Datadog’s documentation for a comprehensive list of IP ranges and hostnames.
- Why it works: The Datadog agent needs to send data to Datadog’s servers. Network restrictions can prevent this essential communication, even if the agent itself is running correctly.
Test Results Not Being Captured by Datadog’s Integration:
- Diagnosis: Check the specific Datadog integration for your test framework. For example, if using Jest, Datadog’s integration might require specific Jest configuration or plugins. Inspect the CI job logs for any output from the dd-trace CLI or Datadog tracer indicating it’s not finding or recognizing your test framework. Look for messages like "No test framework detected."
- Fix: Ensure you’ve installed any necessary Datadog-specific test runner packages or plugins. For example, if Datadog requires a specific Jest reporter, install it (npm install --save-dev @datadog/datadog-jest-reporter) and configure it in your Jest config (jest.config.js).
- Why it works: Datadog’s CI Test Visibility relies on specific integrations to correctly parse and report test results from different frameworks. Without the correct integration, it can’t understand the output or structure of your tests.

The next error you’ll likely encounter is datadog.api.exceptions.RateLimitError if you’ve fixed the reporting but are sending too much data.