I'm having a strange routing problem within a AWS VPC that I just can't get my head around.
VPC: 10.16.20.128/25 Subnet: 10.16.20.128/25 Routing table: 0.0.0.0 to Internet gateway, 10.16.20.128/25 local Network ACL: standard (rule 100 allow all, rule * deny all) I have 2 linux/debian instances. The first one is 10.16.20.132 is also has a VPN connection to an external VPN server for the subnet 10.16.0.0/16 (so a superset of the local network). It also has net.ipv4.ip_forward=1 enabled in /etc/sysctl, which is required for routing.
Both instances can connect to the internet and are able to ping each other. The first instance is able to ping hosts at the other side of the VPN and it's confirmed that the VPN server has a route back to the subnet. In both cases, tcpdump shows all the packets as I would expect them.
When I add a route to the 10.16.0.0/32 subnet on the second host and route it via the first host, I can't ping hosts at the other side of the VPN. The part where I can't get my head around is that I can't even see the packages arrive at the first host using tcpdump. I can see the packets leave host 2, but they never arrive at host 1.
Host 1 config
# sysctl -p net.ipv4.ip_forward = 1 # ip add ... 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc pfifo_fast state UP group default qlen 1000 link/ether 02:33:71:2f:34:66 brd ff:ff:ff:ff:ff:ff inet 10.16.20.132/25 brd 10.16.20.255 scope global dynamic eth0 valid_lft 2466sec preferred_lft 2466sec inet6 fe80::33:71ff:fe2f:3466/64 scope link valid_lft forever preferred_lft forever 3: ip_vti0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000 link/ipip 0.0.0.0 brd 0.0.0.0 4: vti1@NONE: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 8981 qdisc noqueue state UNKNOWN group default qlen 1000 link/ipip 10.16.20.132 peer <ip.of.vpn.server> inet6 fe80::5efe:a10:1484/64 scope link valid_lft forever preferred_lft forever # ip route default via 10.16.20.129 dev eth0 10.16.0.0/16 dev vti1 scope link 10.16.20.128/25 dev eth0 proto kernel scope link src 10.16.20.132 # iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination Chain FORWARD (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination # ping 10.16.20.177 <-- other instance, works PING 10.16.20.177 (10.16.20.177) 56(84) bytes of data. 64 bytes from 10.16.20.177: icmp_seq=1 ttl=64 time=0.362 ms # ping 10.16.0.2 <-- host at other side of the VPN connection, works PING 10.16.0.2 (10.16.0.2) 56(84) bytes of data. 64 bytes from 10.16.0.2: icmp_seq=1 ttl=62 time=12.0 ms Host 2 config
# ip add ... 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc pfifo_fast state UP group default qlen 1000 link/ether 02:77:24:63:66:c2 brd ff:ff:ff:ff:ff:ff inet 10.16.20.177/25 brd 10.16.20.255 scope global dynamic eth0 valid_lft 2100sec preferred_lft 2100sec inet6 fe80::77:24ff:fe63:66c2/64 scope link valid_lft forever preferred_lft forever # ip route default via 10.16.20.129 dev eth0 10.16.20.128/25 dev eth0 proto kernel scope link src 10.16.20.177 10.16.0.0/16 via 10.16.20.132 dev eth0 # ping 10.16.20.132 <-- instance 1, works PING 10.16.20.132 (10.16.20.132) 56(84) bytes of data. 64 bytes from 10.16.20.132: icmp_seq=1 ttl=64 time=0.373 ms # ping 10.16.0.2 <-- host at other side of the VPN connection, DOESN'T WORK PING 10.16.0.2 (10.16.0.2) 56(84) bytes of data. ^C --- 10.16.0.2 ping statistics --- 4 packets transmitted, 0 received, 100% packet loss, time 58ms tcpdump on host 1 when host 2 is pinging
# tcpdump -nn -a -i any not port 22 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes # when pinging host 1 from host 2 07:44:42.518133 IP 10.16.20.177 > 10.16.20.132: ICMP echo request, id 9234, seq 1, length 64 07:44:42.518161 IP 10.16.20.132 > 10.16.20.177: ICMP echo reply, id 9234, seq 1, length 64 # when pinging host at other side of the VPN connection from host 2 <big empty void here> tcpdump on host 2 when host 2 is pinging
# tcpdump -e -i eth0 -nn not port 22 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes # when pinging host 1 from host 2 07:50:04.858467 02:77:24:63:66:c2 > 02:33:71:2f:34:66, ethertype IPv4 (0x0800), length 98: 10.16.20.177 > 10.16.20.132: ICMP echo request, id 9239, seq 1, length 64 07:50:04.858859 02:33:71:2f:34:66 > 02:77:24:63:66:c2, ethertype IPv4 (0x0800), length 98: 10.16.20.132 > 10.16.20.177: ICMP echo reply, id 9239, seq 1, length 64 # when pinging host at other side of the VPN connection from host 2 07:53:00.123557 02:77:24:63:66:c2 > 02:33:71:2f:34:66, ethertype IPv4 (0x0800), length 98: 10.16.20.177 > 10.16.0.2: ICMP echo request, id 9241, seq 5, length 64 07:53:01.147564 02:77:24:63:66:c2 > 02:33:71:2f:34:66, ethertype IPv4 (0x0800), length 98: 10.16.20.177 > 10.16.0.2: ICMP echo request, id 9241, seq 6, length 64 07:53:01.275527 02:77:24:63:66:c2 > 02:33:71:2f:34:66, ethertype ARP (0x0806), length 42: Request who-has 10.16.20.132 tell 10.16.20.177, length 28 07:53:01.275641 02:33:71:2f:34:66 > 02:77:24:63:66:c2, ethertype ARP (0x0806), length 56: Reply 10.16.20.132 is-at 02:33:71:2f:34:66, length 42 07:53:02.171582 02:77:24:63:66:c2 > 02:33:71:2f:34:66, ethertype IPv4 (0x0800), length 98: 10.16.20.177 > 10.16.0.2: ICMP echo request, id 9241, seq 7, length 64 07:53:03.195571 02:77:24:63:66:c2 > 02:33:71:2f:34:66, ethertype IPv4 (0x0800), length 98: 10.16.20.177 > 10.16.0.2: ICMP echo request, id 9241, seq 8, length 64 tcpdump on host 1 when host at other side of vpn is pinging host 2
tcpdump -e -i -n -i any not port 22 tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes 08:11:56.672255 In ethertype IPv4 (0x0800), length 100: (tos 0x0, ttl 62, id 61024, offset 0, flags [DF], proto ICMP (1), length 84) 10.16.0.11 > 10.16.20.177: ICMP echo request, id 4413, seq 0, length 64 08:11:56.672344 Out 02:33:71:2f:34:66 ethertype IPv4 (0x0800), length 100: (tos 0x0, ttl 61, id 61024, offset 0, flags [DF], proto ICMP (1), length 84) 10.16.0.11 > 10.16.20.177: ICMP echo request, id 4413, seq 0, length 64 tcpdump on host 1 when host at other side of vpn is pinging host 2
tcpdump -e -i eth0 -nn not port 22 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes <big empty void here> So it seems that specifically traffic that actually passed the AWS subnet / network stack but has a source or target IP outside of the subnet seems to disappear somewhere between the two hosts in the subnet. The fact that I don't see the packets arrive at the other host (in either direction) makes me think that this might be AWS related, but I can't really rule that out. Anyone has any idea what I can still look at?
No comments:
Post a Comment