Detect and Prevent Port Scanning on Linux Servers (2026)

Port scanning is your server’s unsolicited admirer, trying to find open doors to sneak through.

Here’s how a port scan actually looks in practice, if you’ve got tcpdump handy. Imagine a scanner on 192.168.1.100 poking at your server 192.168.1.50.

# On the scanning machine (192.168.1.100)
nmap -p 1-1024 192.168.1.50

# On your server (192.168.1.50), watch this:
sudo tcpdump -ni any 'host 192.168.1.100 and tcp'

You’ll see a flurry of SYN packets from 192.168.1.100 to various ports on 192.168.1.50. If a port is open, your server dutifully replies with a SYN-ACK. If it’s closed, it sends back an RST. This rapid-fire probing is the telltale sign.

The most effective way to detect and prevent this is with iptables and a little helper called fail2ban. fail2ban watches your logs for suspicious patterns, like too many failed connection attempts or, in this case, too many connection probes from a single IP address. When it sees a pattern, it tells iptables to block that IP for a while.

Common Causes and Fixes

Simple SYN Scans (most common): The scanner just sends SYN packets and looks for SYN-ACKs.
- Diagnosis: Look for a single source IP hitting many destination ports on your server in quick succession.
```
# On your server, install and configure fail2ban if you haven't already
sudo apt-get update && sudo apt-get install fail2ban
sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local
```
  Edit /etc/fail2ban/jail.local and add a new jail specifically for port scanning:
```
[syn-scan]
enabled = true
port = 1-65535
filter = synscan
logpath = /var/log/syslog # Or your specific firewall log if you redirect
maxretry = 10 # Number of SYN packets to tolerate from one IP in the scan window
findtime = 60 # Scan window in seconds
bantime = 3600 # Ban duration in seconds (1 hour)
action = iptables-multiport[name=SYN-SCAN, port="1-65535", protocol=tcp]
```
  You’ll also need to create a custom filter file at /etc/fail2ban/filter.d/synscan.conf:
```
[Definition]
failregex = ^%(_daemon_prefix)s.*SRC=<HOST>.*ATTEMPT to.*$
ignoreregex =
```
  Note: The failregex above is a simplified example. A more robust filter would look for the specific packet patterns. For a truly robust solution, you might need to log packet information to a file that fail2ban can parse, or use more advanced iptables modules.
  - Why it works: fail2ban monitors system logs (or custom logs). When the synscan filter detects a high rate of SYN packets from a single source IP within the findtime window (e.g., 10 SYN packets in 60 seconds), it triggers the iptables-multiport action to add a rule blocking that IP for bantime seconds.
FIN, NULL, and Xmas Scans: These scans send packets with unusual flag combinations (FIN only, no flags, or URG, PSH, FIN flags set) to try and bypass simple firewalls that only check for SYN packets. Closed ports on Linux typically respond with an RST, while open ports tend to ignore these malformed packets.
- Diagnosis: This is harder to catch with basic log monitoring. You’d typically see a lot of unanswered probes from a single IP to various ports. iptables has a module called xtables-addons that can help. Install it:
```
sudo apt-get install xtables-addons-common
```
  Then, load the physdev module and add rules to iptables:
```
sudo modprobe xt_physdev
sudo iptables -A INPUT -m physdev --physdev-in eth0 -p tcp --tcp-flags FIN,URG,PSH NULL -j DROP
sudo iptables -A INPUT -m physdev --physdev-in eth0 -p tcp --tcp-flags FIN FIN -j DROP
sudo iptables -A INPUT -m physdev --physdev-in eth0 -p tcp --tcp-flags FIN,SYN FIN -j DROP
sudo iptables -A INPUT -m physdev --physdev-in eth0 -p tcp --tcp-flags SYN,RST SYN -j ACCEPT # Allow normal SYN
sudo iptables -A INPUT -m physdev --physdev-in eth0 -p tcp --tcp-flags ACK,FIN FIN -j DROP
sudo iptables -A INPUT -m physdev --physdev-in eth0 -p tcp --tcp-flags ACK,PSH PSH -j DROP
sudo iptables -A INPUT -m physdev --physdev-in eth0 -p tcp --tcp-flags ACK,URG URG -j DROP
sudo iptables -A INPUT -m physdev --physdev-in eth0 -p tcp --tcp-flags FIN,ACK FIN -j DROP
sudo iptables -A INPUT -m physdev --physdev-in eth0 -p tcp --tcp-flags SYN,ACK SYN -j ACCEPT # Allow normal SYN/ACK handshake
```
  Note: The specific flags and order can be complex. These rules aim to drop packets that are malformed or don’t adhere to standard TCP states.
  - Why it works: These iptables rules specifically target TCP packets with unusual or illegal flag combinations. By dropping them at the firewall level before they even reach the application or trigger a response, they effectively neutralize these scan types.
Stealth Scans (e.g., ACK scan): These scans send ACK packets and look at the RST responses. They don’t establish a connection and are often used to map firewall rulesets, not necessarily to find open ports directly.
- Diagnosis: Similar to FIN scans, you’ll see many probes from one IP without a full connection handshake. Again, xtables-addons is useful.
```
sudo iptables -A INPUT -m state --state INVALID -j DROP
```
  - Why it works: This rule marks packets that don’t conform to valid TCP connection states as INVALID and drops them. ACK scans often produce such invalid states, effectively blocking them.
Port Knocking (Legitimate but can be mimicked): Sometimes, a legitimate service might be configured to only open its port after a sequence of "knocks" on other ports. A scanner might try to brute-force this sequence.
- Diagnosis: If you have port knocking set up, you’ll see rapid connection attempts to specific, usually non-standard, ports from an IP, followed by an attempt to connect to the actual service port.
- Fix: Ensure your port knocking daemon is configured with strong timeouts and a reasonable number of retries. For example, using knockd:
```
[options]
logfile = /var/log/knockd.log
use-syslog
# Set a maximum of 5 attempts within 20 seconds
max_retries = 5
timeout = 20

[openSSH]
sequence = 1111,2222,3333
```
  - Why it works: The max_retries and timeout parameters in knockd limit how quickly an attacker can try different sequences, making brute-forcing impractical.
Nmap Scripting Engine (NSE) Scans: Nmap can be scripted to perform much more sophisticated scans, including fingerprinting OS versions and detecting vulnerabilities.
- Diagnosis: These scans might appear as a mix of the above, but with more detailed probes. You might see specific service banner requests or attempts to elicit specific error messages.
- Fix: A combination of fail2ban with good iptables rules is your best bet. Ensure your fail2ban jails are tuned to catch unusual traffic patterns. For instance, a broader synscan jail with a lower maxretry might catch more aggressive NSE scripts.
```
[syn-scan]
enabled = true
port = 1-65535
filter = synscan
logpath = /var/log/syslog
maxretry = 5 # Lowered for more aggressive scans
findtime = 30 # Shorter window
bantime = 7200 # Longer ban
action = iptables-multiport[name=SYN-SCAN, port="1-65535", protocol=tcp]
```
  - Why it works: By reducing the maxretry and findtime, fail2ban becomes more sensitive to rapid, albeit less numerous, probes characteristic of sophisticated scripts. A longer bantime provides more breathing room.
Source IP Spoofing: Attackers may try to spoof their source IP to hide their true identity or to bypass IP-based bans.
- Diagnosis: You might see traffic from internal IP addresses (e.g., 192.168.x.x, 10.x.x.x, 172.16.x.x - 172.31.x.x) arriving on your external interface, or traffic with a source IP that isn’t routable on your network.
- Fix: Implement ingress filtering (BCP 38) on your edge routers or firewalls. On Linux, you can add rules to drop packets arriving on an external interface that have source IPs belonging to internal networks.
```
# Assuming eth0 is your external interface
sudo iptables -A INPUT -i eth0 -s 192.168.0.0/16 -j DROP
sudo iptables -A INPUT -i eth0 -s 10.0.0.0/8 -j DROP
sudo iptables -A INPUT -i eth0 -s 172.16.0.0/12 -j DROP
```
  - Why it works: These rules prevent packets with private IP addresses from entering your network on an interface that should only receive public IP addresses, thus blocking spoofed internal IPs.

After implementing these iptables rules and configuring fail2ban, the next error you might see is related to fail2ban’s own management, such as "fail2ban.service: Failed to parse configuration file /etc/fail2ban/jail.local".