Linux systems can be surprisingly chatty about security events, but you have to know how to listen.
Let’s see what a Linux system is actually doing when it logs a security event. Imagine a user, "alice," tries to SSH into a server.
# On the server, as root
tail -f /var/log/auth.log
Jul 26 10:00:01 server1 sshd[12345]: Accepted password for alice from 192.168.1.10 port 54321 ssh2
This single line tells a story: the SSH daemon (sshd) accepted a password login for user alice from IP address 192.168.1.10 on port 54321. But that’s just one log file. To get the full picture, we need to understand what generates these logs and how to collect and analyze them.
The core of Linux security logging is the syslog protocol and its implementations. The most common daemon is rsyslogd. It’s configured by /etc/rsyslog.conf and files in /etc/rsyslog.d/. rsyslog acts as a central collector, receiving messages from various system services and applications, and then routing them to different destinations based on rules. These destinations can be local files, remote syslog servers, or even databases.
Beyond rsyslog, many security-sensitive applications and kernel modules generate their own logs or integrate with syslog. The Linux Audit Daemon (auditd) is a prime example. It provides a powerful framework for logging system calls, file access, and security-relevant events. Its configuration is primarily in /etc/audit/auditd.conf and its rules are defined in /etc/audit/rules.d/.
Here’s how auditd might log a file access:
# On the server, as root
tail -f /var/log/audit/audit.log
type=SYSCALL msg=audit(1658803200.123:456): arch=c000003e syscall=2 success=yes exit=3 a0=2 a1=7ffc5c438c50 a2=0 a3=0 items=1 ppid=12345 pid=67890 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 tty=pts0 ses=1 comm="cat" exe="/usr/bin/cat" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key="access_sensitive_file"
type=CWD msg=audit(1658803200.123:456): cwd="/home/alice"
type=PATH msg=audit(1658803200.123:456): name="/etc/shadow" inode=12345 dev=fd:01 mode=0100640 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:shadow_t:s0
This auditd log is much more granular. It shows a SYSCALL event where the cat command (running as user alice, auid=1000) attempted to access /etc/shadow (exe="/usr/bin/cat"). The key="access_sensitive_file" indicates a custom rule was triggered.
To get a robust security monitoring setup, you typically combine these sources. For effective analysis, you’ll want to centralize these logs. A common pattern is to use rsyslog on each host to forward logs to a dedicated log server.
On a client machine, to send all logs to a central server logserver.example.com on UDP port 514:
Edit /etc/rsyslog.d/50-default.conf (or create a new file like 99-remote.conf) and add:
*.* @logserver.example.com:514
On the logserver.example.com, ensure rsyslog is configured to receive logs. In /etc/rsyslog.conf or a file in /etc/rsyslog.d/, you’d have:
module(load="imudp")
input(type="imudp" port="514")
And then define where to store them, for example:
$template RemoteLogs,"/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log"
*.* ?RemoteLogs
This configuration directs all incoming logs from remote hosts to files named after the hostname and program that generated the log, stored under /var/log/remote/.
For auditd logs, you might want to forward them specifically. On the client, in a file like /etc/audit/rules.d/remote.rules, you’d add:
-S open -F path=/etc/shadow -k sensitive_access
-S open -F path=/etc/passwd -k sensitive_access
-a always,exit -F arch=b64 -S execve -k exec_commands
-a always,exit -F arch=x86_64 -S unlink -k file_deletion
-f +2
Then, configure rsyslog to pick up the auditd logs (often written to /var/log/audit/audit.log or a pipe) and forward them. A typical rsyslog rule might look like:
if $programname == 'auditd' then @@logserver.example.com:514
This directs all messages originating from the auditd process to the central log server.
For deeper analysis, tools like Elasticsearch, Logstash, and Kibana (ELK stack) or Splunk are invaluable. Logstash can ingest logs from rsyslog, parse them, and send them to Elasticsearch for indexing. Kibana then provides a web interface for searching, visualizing, and creating dashboards from the indexed data.
The one thing that often trips people up with auditd is its rule syntax. It’s powerful but verbose. For instance, to monitor all writes to /etc/passwd by any user, you’d add a rule like this to /etc/audit/rules.d/custom.rules:
-w /etc/passwd -p wa -k passwd_changes
This rule tells auditd to watch (-w) the file /etc/passwd and log (-p) any write (w) or attribute change (a) operations, tagging them with the key passwd_changes. If you don’t explicitly define the file and permissions you want to monitor, you’ll miss critical events.
Once you have logs centralized and indexed, you can start building alerts. For example, you could set up a Kibana alert to trigger an email or webhook when a specific auditd key like sensitive_access is seen, or when multiple failed SSH login attempts (/var/log/auth.log) occur from the same IP within a short period.
The next step after setting up basic logging is often automating the analysis and response to these security events.