Elastic Fleet Agent is your new best friend for shipping data to Elasticsearch, but it’s not just a glorified rsyslog.
Let’s watch it in action. Imagine you want to ingest Linux system logs. You’d typically use something like filebeat or rsyslog to grab /var/log/syslog and send it off. With Fleet, it’s more declarative.
First, in Kibana, you’d go to "Fleet" -> "Agents" and enroll a new agent. This gives you a token. On your Linux machine, you install the Fleet agent:
sudo curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.10.4-linux-x86_64.tar.gz
sudo tar xzvf elastic-agent-8.10.4-linux-x86_64.tar.gz
cd elastic-agent-8.10.4-linux-x86_64
sudo ./elastic-agent install --url=https://your-elasticsearch-host:8080 --token=YOUR_ENROLLMENT_TOKEN
This agent isn’t just running; it’s actively checking in with your Fleet Server (which is often just your Elasticsearch node itself, or a dedicated instance). The agent pulls its configuration from Fleet, not from a local file you edit.
Now, in Kibana, you go to "Fleet" -> "Integrations." You find "System" and click "Add and Configure." You choose your agent, give it a name (e.g., my-linux-server-logs), and set the output to your Elasticsearch data stream. You might configure it to read from /var/log/syslog and /var/log/auth.log.
# This is what Fleet *pushes* to the agent, not what you edit locally
# Snippet of what the agent config might look like after Fleet config
- type: system
streams:
- id: syslog
paths:
- /var/log/syslog
- /var/log/syslog.1
- /var/log/messages
- /var/log/messages.1
# ... other log paths
- id: authlog
paths:
- /var/log/auth.log
- /var/log/auth.log.1
# ... other auth log paths
processors:
- dissect:
tokenizer: "%{timestamp} %{hostname} %{program}: %{message}"
field: "message"
Once you save the integration, Fleet tells the enrolled agent to download and apply this configuration. The agent then starts tailing those files, parsing them according to the dissect or grok patterns defined in the integration, and sending the structured data to your Elasticsearch index. The key here is the declarative nature: you tell Fleet what you want, and Fleet tells the agent how to do it.
The problem this solves is the complexity of managing agents across potentially thousands of machines. Instead of SSHing into each server and editing filebeat.yml, you manage everything from a single Kibana UI. The agent is designed to be resilient, automatically reconnecting and resuming data collection if it’s interrupted. It also handles secure communication (TLS) and authentication with your Elasticsearch cluster out-of-the-box.
The one thing most people don’t realize is that the agent has its own internal state for each data stream it’s collecting. If it stops and restarts, it knows exactly which line in which file it was last processing because it persists this offset information. This is managed by the agent itself, not by Elasticsearch, ensuring that no log lines are missed during downtime.
The next step is understanding how to customize these integrations with your own parsing rules or add entirely new data sources.