DogStatsD, the agent’s statsd protocol endpoint, refused to accept metrics from a source that wasn’t the local machine, indicating a security or configuration misstep.
Here’s why this happens and how to fix it:
1. Incorrect bind_host Configuration:
The most frequent culprit is the bind_host setting in your datadog.yaml file. If this is set to 127.0.0.1 (localhost) or localhost, DogStatsD will only listen for traffic originating from the agent’s own machine. Any metrics sent from another host will be rejected.
-
Diagnosis: Check your
datadog.yamlfile for thedogstatsdsection. Look for thebind_hostparameter. -
Fix: Change
bind_hostto0.0.0.0to allow DogStatsD to bind to all available network interfaces, or to the specific IP address of your agent machine if you want to be more restrictive.dogstatsd: bind_host: 0.0.0.0 # ... other dogstatsd settings -
Why it works: Binding to
0.0.0.0tells the operating system to accept incoming connections on any IP address configured on the host, effectively making DogStatsD visible on the network.
2. Firewall Blocking Traffic:
Even if DogStatsD is configured to listen on all interfaces, a firewall on the agent machine might be preventing external traffic from reaching the DogStatsD port (default is 8125).
- Diagnosis: On the agent machine, use a tool like
ufw(Ubuntu/Debian) orfirewalld(CentOS/RHEL) to check active rules.- For
ufw:sudo ufw status - For
firewalld:sudo firewall-cmd --list-all
- For
- Fix: Open UDP port 8125 (or your configured DogStatsD port) for incoming traffic.
- For
ufw:sudo ufw allow 8125/udp - For
firewalld:sudo firewall-cmd --add-port=8125/udp --permanent && sudo firewall-cmd --reload
- For
- Why it works: This explicitly allows network packets destined for the DogStatsD port to pass through the firewall and reach the agent process.
3. Network Address Translation (NAT) Issues:
If your agent is behind a NAT gateway and you’re trying to send metrics from outside that network, the NAT configuration might not be correctly forwarding UDP traffic on port 8125 to the agent.
- Diagnosis: Verify your NAT rules on your firewall or router. Ensure that UDP traffic on port 8125 from external IPs is being translated and forwarded to the internal IP address of your Datadog agent.
- Fix: Configure your NAT gateway to forward UDP port 8125 to the agent’s IP address.
- Why it works: NAT rewrites the destination IP address of packets entering your private network to the correct internal IP, allowing the agent to receive the metrics.
4. Incorrect Metric Source IP in Datadog UI (Less Common for this Specific Error):
While less common for the "Non-Local Traffic Disabled" error itself, if you’ve manually configured IP allowlists or blocklists within Datadog’s integrations or agent settings, you might inadvertently be blocking traffic from your intended source. This error is usually agent-side, but worth a quick check if the above don’t resolve it.
- Diagnosis: In the Datadog UI, navigate to Agent settings or specific integration configurations that might have IP filtering enabled.
- Fix: Ensure the IP address of the machine sending metrics is not explicitly blocked.
- Why it works: This removes any application-level filtering that might be misinterpreting legitimate traffic as invalid.
5. DogStatsD Configuration use_ms or use_grpc Mismatch:
If you’re using DogStatsD with use_ms (message stream) or use_grpc enabled, and your client library isn’t configured to match, it could lead to connection issues that manifest as traffic being dropped. However, the specific "Non-Local Traffic Disabled" error points more directly to network binding.
- Diagnosis: Check your
datadog.yamlfordogstatsd.use_msanddogstatsd.use_grpc. Then, check your application’s Datadog client library configuration. - Fix: Ensure
use_msoruse_grpcsettings are consistent between the agent configuration and your application’s client. If you’re not intentionally using these features, ensure they are set tofalse(or omitted, asfalseis often the default). - Why it works: This ensures that both the agent and the sending application are speaking the same DogStatsD protocol version or transport mechanism.
6. Agent Not Restarted After Configuration Change:
A common oversight is forgetting to restart the Datadog agent after modifying datadog.yaml. Configuration changes are typically only loaded upon agent startup.
- Diagnosis: No direct diagnostic command, but a strong suspicion if you’ve made changes and the problem persists.
- Fix: Restart the Datadog agent service.
- On Linux:
sudo systemctl restart datadog-agent - On Windows: Restart the "Datadog Agent" service via
services.msc.
- On Linux:
- Why it works: This re-initializes the agent process, forcing it to read the updated
datadog.yamland apply the newbind_hostor other settings.
After applying these fixes, you should see metrics flowing in from your non-local sources. The next error you might encounter, if your setup is complex, is a connection refused error if the DogStatsD endpoint is still not reachable due to a more obscure network routing issue or a misconfigured client.