Hi all!
Early in my career, so please bare with me :). I've previously implemented an NGINX Load Balancer with IP Transparency for both TCP/UDP and HTTPS streams. Worked great in our previous setup (PFSense Firewall + a Cisco Catalyst Switch).
A fellow Network Engineer and I worked on a project to replace the switch with a newer Catalyst and to replace the PFSense with a Fortigate. Not much has changed on the architecture front other than our networks getting assigned new VLANs. Ever since the change, my IP Transparency stopped working on the upstream servers. The Load Balancer is correctly listing the remote address, is correctly sending packets with the remote addr as the source address, but the upstream server is not showing the source as the remote addr. After countless efforts to isolate the issue, we are stuck. Below are some outputs from conntrack -L
1.) Here's the packet observed on the Load Balancer seeing traffic from the Remote Client:CLOSE_WAIT src=<Remote Client Public IP> dst=<Load Balancer Private IP> sport=57447 dport=443 src=<Load Balancer Private IP> dst=<Remote Client Public IP> sport=443 dport=57447 [ASSURED] mark=0 use=1
2.) Here's the packet being sent to the upstream server, observed from the Load Balancer
110 TIME_WAIT src=<Remote Client Public IP> dst=<Upstream Server #1> sport=57447 dport=443 src=<Upstream Server #1> dst=<Load Balancer Private IP> sport=443 dport=57447 [ASSURED] mark=0 use=1
3.) Here's the packet being sent to the upstream server, observed from the upstream server.
TIME_WAIT src=<Load Balancer Private IP> dst=<Upstream Server #1> sport=57447 dport=443 src=<Upstream Server #1> dst=<Load Balancer Private IP> sport=443 dport=57447 [ASSURED] mark=0 use=1
This, in my opinion, confirms that IP Transparency/IP Spoofing is working on the Load Balancer. We have tried to disable any Reverse Path Filtering setting on the Firewall, and I can't find any enabled spoofing protection setting on the Catalyst. Does anyone have another place to look? I also took tcpdumps on both the Load Balancer and Upstream Server, but WireShark wasn't showing any of the packets that would match #2. Any help would be greatly appreciated.
Some other notes: Yes, I have rebuilt the server from scratch on the latest CentOS 7x + NGINX releases. Yes, I've also restored from an .ova of a known good state. I've also confirmed via their troubleshooting docs that this setup is working correctly. The upstream servers and load balancer are on the same VLAN and subnet.
No comments:
Post a Comment