Network ACLs are stateless firewalls at the subnet level, which means they don’t remember if an incoming packet was part of an established outgoing connection. This often leads to unexpected blocks because you have to explicitly allow traffic in both directions for any communication.
Let’s say you’ve got an EC2 instance that’s not responding to SSH from your local machine, and you suspect a Network ACL (NACL) is the culprit.
Common Causes and Fixes
-
Missing Outbound Rule for Established Connections:
- Diagnosis: On your EC2 instance’s subnet, check the NACL’s outbound rules. You’ll likely see a rule allowing SSH (port 22) outbound, but not the ephemeral ports that the return traffic from your SSH connection will use.
- Fix: Add an outbound rule to the NACL.
- Rule #:
100(or any number lower than your explicit deny rules, higher than your implicit deny*) - Type: Custom TCP
- Port Range:
1024-65535 - Destination:
0.0.0.0/0 - Protocol:
6 (TCP)
- Rule #:
- Why it works: When your local machine sends an SSH connection request (SYN packet) to your EC2 instance on port 22, the EC2 instance responds with a SYN-ACK. This return traffic originates from an ephemeral port on the EC2 instance and is destined for an ephemeral port on your local machine. The NACL needs to explicitly allow this outbound ephemeral traffic for the connection to be established.
-
Missing Inbound Rule for Return Traffic:
- Diagnosis: If you can SSH out from your EC2 instance to somewhere else, but not in to your EC2 instance, check the NACL’s inbound rules. You might have a rule allowing SSH (port 22) inbound, but not the ephemeral ports used for the response from your EC2 instance.
- Fix: Add an inbound rule to the NACL.
- Rule #:
100(or similar) - Type: Custom TCP
- Port Range:
1024-65535 - Source:
0.0.0.0/0 - Protocol:
6 (TCP)
- Rule #:
- Why it works: This allows the return traffic from your local machine (which initiated the SSH connection on an ephemeral port) to reach your EC2 instance on its ephemeral port. Since NACLs are stateless, they don’t automatically associate this inbound response with the initial outbound request.
-
Incorrect Port Range for Outbound Traffic:
- Diagnosis: Similar to #1, but specifically check if the outbound ephemeral port range is too restrictive. For example, if you only allowed
49152-65535outbound but your system is using ports in the1024-65535range. - Fix: Ensure the outbound ephemeral port range is broad enough.
- Rule #:
100 - Type: Custom TCP
- Port Range:
1024-65535 - Destination:
0.0.0.0/0 - Protocol:
6 (TCP)
- Rule #:
- Why it works: This covers the standard range of ephemeral ports used by TCP connections for return traffic.
- Diagnosis: Similar to #1, but specifically check if the outbound ephemeral port range is too restrictive. For example, if you only allowed
-
Rule Order Issues with Explicit Denies:
- Diagnosis: NACLs evaluate rules in order from lowest to highest number. If you have an explicit deny rule (e.g., port
22withDENY) that comes before your allow rule for port22, traffic will be blocked. - Fix: Ensure your allow rules have lower rule numbers than any deny rules that might affect the same traffic.
- Example:
- Rule #
100: Allow TCP port22fromyour_ip/32 - Rule #
110: Allow TCP port1024-65535fromyour_ip/32 - Rule #
300: Deny TCP port22from0.0.0.0/0
- Rule #
- Example:
- Why it works: The NACL processes rules sequentially. By placing the allow rule with a lower number, it’s evaluated and matched before any broader deny rule that might otherwise block it.
- Diagnosis: NACLs evaluate rules in order from lowest to highest number. If you have an explicit deny rule (e.g., port
-
Blocking UDP Traffic for DNS Lookups:
- Diagnosis: If your EC2 instance can’t resolve domain names (e.g.,
ping google.comfails with "Temporary failure in name resolution"), check NACL rules for UDP port 53. - Fix: Add inbound and outbound rules for UDP port 53.
- Inbound:
- Rule #:
110 - Type: Custom UDP
- Port Range:
53 - Source:
0.0.0.0/0(or your VPC CIDR if you have an internal DNS resolver) - Protocol:
17 (UDP)
- Rule #:
- Outbound:
- Rule #:
110 - Type: Custom UDP
- Port Range:
53 - Destination:
0.0.0.0/0(or your VPC CIDR/DNS resolver IP) - Protocol:
17 (UDP)
- Rule #:
- Inbound:
- Why it works: DNS queries typically use UDP port 53. Since NACLs are stateless, you need to allow this traffic in both directions for the query to go out and the response to come back.
- Diagnosis: If your EC2 instance can’t resolve domain names (e.g.,
-
Incorrect Source/Destination IP Addresses:
- Diagnosis: Double-check that the source IP in inbound rules and destination IP in outbound rules are correctly specified. Using
/32for single IPs and your VPC CIDR (e.g.,10.0.0.0/16) for internal traffic is common. - Fix: Correct any typos or CIDR block misconfigurations. For example, if you’re trying to SSH from
203.0.113.5and your NACL inbound rule for port 22 has192.168.1.0/24as the source, it won’t work. Change the source to203.0.113.5/32. - Why it works: NACLs perform IP address matching. If the source or destination IP doesn’t match the rule’s CIDR block, the traffic is not allowed.
- Diagnosis: Double-check that the source IP in inbound rules and destination IP in outbound rules are correctly specified. Using
After applying these corrections, you might next encounter ELB-HealthCheck-Failed errors if your instance is part of a load balancer, as ELBs also use health checks that need to be allowed through NACLs.