HTTPS traffic blackholes over S2S VPN

Hi folks,

I’ve been working on a site standup and I’ve got a fairly interesting problem I could use some perspective on. If you wouldn’t mind lending me your minds for a few, I would greatly appreciate it.

Topology: HQ Firewall <— S2S routed IPsec Tunnel —> Remote Firewall

Routes are exchanged over the tunnel between the two firewalls, running OSPF in Area 0.0.0.20 (normal area). Both endpoints are redistributing any static routes.

HQ Core is a pair of Nexus 9ks, with a number of static route remote office prefixes that point clients to the HQ Firewall to then egress via VPN tunnel. (Don’t ask me why they didn’t set them up with OSPF...).

Remote office core is a pair of FlexFabric 5900s. All routing in this office is handled via OSPF, Area 10 between the internal site routers. The remote firewall participates and receives prefixes from Area 10 and pushes them out in Area 20 (VPN S2S).

What I see: I see the routes are getting learned on both firewalls. The remote gets HQ’s prefixes and HQ gets remote’s prefixes. That’s great! I can pass ICMP traffic from the HQ end to a host on the remote office end and get responses, and vice-versa. I can RDP to computers in the remote office. So traffic is passing.

However, when I go to try logging into iLO or VMware at this site, the HTTPS traffic just seems to die. The browser—all of the browsers—I’ve tried just seems to spin and eventually it times out. A packet capture shows ACKs, SYNs, eventually ACK PSH... but eventually it appears as if the connection times out after a while. I do see RSTs being sent back to the client on the remote end, but the local end does not see those RSTs.

I’ve worked with the firewall vendor to ensure there is no sort of traffic inspection, IDS/IPS involvement, firewall rule blocking or changing flags/states, or NAT rules redirecting. They’ve officially pointed the finger at routing because they did not see anything that indicated the firewalls were involved.

If I SSLVPN to the site directly, HTTPS traffic works just fine. The only think I can think of that could be happening is that maybe there is some sort of misconfiguration in my routing or S2S setup that the packet is either getting misdirected or dropped. Oddly, I do see the packet emerge on the remote office end, see the reply come back through the VPN.

Would any of you happen to have run into this sort of issue before? I’m really at my wit’s end and I’ve got to have this site up and functional for Monday, so I’m doing my best not to lose my cool and be objective.

Thanks for any tips you can offer in advance. If you would find it helpful to see code or RIB/FIB of any of the devices in question, I can get post those up. Many thanks to you all in advance.

/edit: late night grammar fixing