Rootkits can be incredibly stealthy, but their presence often leaves subtle traces in how the operating system manages processes and files.

Let’s say you suspect a rootkit on your Linux system. The first thing to do is to stop all non-essential services and isolate the machine from the network. Then, boot from a known-good, trusted live CD or USB drive. This ensures that any malicious code running on the system’s main installation doesn’t interfere with your detection efforts.

Your primary tool will be chroot. This command allows you to change the root directory for a process, effectively making the infected system’s filesystem appear as the root to your live environment.

# Mount the infected system's root partition
sudo mount /dev/sdXY /mnt

# Chroot into the infected system
sudo chroot /mnt

Once inside the chroot environment, you can start looking for anomalies.

1. Suspicious Processes: Rootkits often hide processes. The standard ps aux might not show them. A common technique is to compare the output of ps with the contents of /proc.

  • Diagnosis: Run ps aux and then manually check each PID listed against the directories present in /proc. A PID listed by ps but without a corresponding directory in /proc is a strong indicator of a hidden process. Alternatively, use a tool like lsof to inspect open files for each process.
    ps aux > /tmp/ps_output.txt
    ls /proc | sort -n > /tmp/proc_dirs.txt
    # Manually compare or script this
    comm -23 /tmp/ps_output.txt /tmp/proc_dirs.txt # This is a simplified example, actual comparison needs PID extraction
    
  • Fix: If you find a suspicious PID, you can try to terminate it using kill -9 <PID>. However, rootkits are designed to resist this. If kill fails, you’ll need to remove the rootkit’s kernel module or binary later.
  • Why it works: Rootkits hook into the kernel’s process management functions. By directly inspecting the /proc filesystem, which is managed by the kernel itself, you bypass the rootkit’s hooks that modify ps output.

2. Hidden Files and Directories: Rootkits can hide files and directories from standard filesystem commands.

  • Diagnosis: Use ls -laR recursively and look for files that have unusual permissions, ownership, or timestamps. Also, compare the output of ls with file listings from a known good backup or another identical system. Tools like find with specific criteria can help.
    find / -xdev -ls 2>/dev/null | grep " Nov 2023" # Example: look for files modified around a specific time
    
    A more robust method is using stat on known system binaries and comparing them to what you see on the infected system.
    # On a known good system:
    stat /bin/ls
    # On the infected system (chrooted):
    stat /bin/ls
    
    Look for differences in inode numbers, timestamps, or file sizes.
  • Fix: Once identified, delete the hidden files/directories using rm -rf <path/to/file>. Be extremely cautious, as deleting legitimate system files can render the system unbootable.
  • Why it works: Rootkits often intercept system calls related to directory listings (readdir) and file stat calls. By using tools that might interact with the filesystem at a lower level or by comparing against known good states, you can uncover these discrepancies.

3. Modified System Binaries: Rootkits often replace or modify core system utilities (like ls, ps, netstat) to hide their presence.

  • Diagnosis: Use checksums. Compare the MD5 or SHA256 checksums of critical binaries on the infected system with those from a known clean source (e.g., a package repository or a trusted installation).
    # Inside chroot:
    md5sum /bin/ls
    md5sum /usr/bin/ssh
    # Compare with output from a clean system or package manager query
    dpkg -V coreutils # For Debian/Ubuntu systems
    rpm -V coreutils # For Red Hat/CentOS systems
    
  • Fix: Reinstall the compromised packages from a trusted source.
    # Inside chroot, for Debian/Ubuntu:
    apt-get --reinstall install coreutils
    # For Red Hat/CentOS:
    yum reinstall coreutils
    
  • Why it works: Rootkits modify these binaries to make them report false information. Verifying the integrity of these files using cryptographic hashes ensures they haven’t been tampered with.

4. Kernel Module Manipulation: Many rootkits operate as kernel modules to gain deep system access.

  • Diagnosis: Check loaded kernel modules. A rootkit module might have a generic name or be disguised.
    # Inside chroot:
    lsmod
    
    Compare the output with a list of expected modules for your kernel version. A common technique is to look for modules that don’t appear in the output of modprobe -l or have unusual sizes. You can also check /lib/modules/$(uname -r)/ for suspicious .ko files.
  • Fix: Unload the suspicious module using rmmod <module_name>. If it’s persistent, you might need to prevent it from loading on boot by removing its .ko file from /lib/modules/<kernel_version>/ and updating module dependencies (depmod -a).
  • Why it works: Rootkits hook into the kernel’s module loading/unloading and listing mechanisms. By directly examining the module files or using tools that query the kernel’s internal state, you can find them.

5. Network Connections: Rootkits can establish covert network channels.

  • Diagnosis: Use netstat -tulnp (or ss -tulnp) and look for listening ports or established connections that you don’t recognize, especially those associated with unusual PIDs. Some rootkits can even hide their network activity from these tools.
    # Inside chroot:
    netstat -tulnp
    
    If netstat seems suspicious, try lsof -i which might reveal connections that netstat misses.
  • Fix: Identify the PID associated with the suspicious connection and terminate it (kill -9 <PID>). Then, remove the rootkit’s executable.
  • Why it works: Rootkits hook into the kernel’s network stack functions. By using tools that bypass or directly query these hooks, you can identify unauthorized network activity.

6. Log File Tampering: Rootkits often attempt to erase their tracks by modifying or deleting log files.

  • Diagnosis: Look for gaps in log files (/var/log/auth.log, /var/log/syslog, /var/log/wtmp, /var/log/btmp). Check file modification times. If logs are missing or truncated, it’s a major red flag.
    # Inside chroot:
    ls -l /var/log/
    stat /var/log/syslog
    
  • Fix: Restore log files from a known good backup. Prevent future tampering by implementing log auditing and secure log storage.
  • Why it works: Rootkits intercept file operations, including writes and deletions to log files. By checking metadata or comparing with backups, you can identify these attempts.

After performing these checks and cleaning up any identified malicious components, you will need to exit the chroot environment and reboot the system.

# Exit chroot
exit
# Unmount the infected system's partition
sudo umount /mnt

The next error you’ll likely encounter is a failure to boot due to missing or corrupted bootloader components, as the rootkit might have tampered with them.

Want structured learning?

Take the full Cdk course →