IPSEC is the bane of my existence because of ISPs f***ing up their DDoS mitigation appliances

I've been troubleshooting connectivity issues between two locations for a few hours now. Turns out it's someone in-between throttling ESP (IPSEC) traffic. I know this because as soon as I force-enable UDP encapsulation of the ESP traffic, the throttling issue goes away.

I was getting 10KB/s (yes you read that right) between the locations using non-encapsulated ESP. As soon as enabling UDP encapsulation, I was maxing out my line speed.

I've seen this at least a few dozen times in my career. Some Tier 2/3 ISP gets the bright idea to install a DDoS mitigation appliance or traffic shaping appliance to save on their bandwidth bills and misconfigures it to heavily restrict non TCP/UDP/ICMP traffic. Eventually enough people complain and they fix the issue but every so often it creeps back up.

So, ProTIP: Always force enable UDP encapsulation on IPSEC traffic for site-to-site.