Friday, March 13, 2020

Major Packet Loss to Certain Geographical Regions - ISP Related?

Hey Everyone,

I've been analyzing this issue for quite a bit and it's sort of baffling me.

I work for a company that deals with large data sets everyday. We send large medical images through IPSec VPN tunnels. This mostly requires us to send terabytes of data around the country daily with certain hub servers that offer gig/gig fiber strategically placed geographically. We have uncapped bandwidth at our main Tierpoint datacenter, with what they call an ISP "blend" for internet access.

Basically, what I have been finding is that during our peak hours 10 AM - 4 PM, packet loss drastically increases but only to certain geographical regions. Even sites with lower latency metrics, packet loss will be greater. We have basically every ISP out there and I am at a loss.

I've been running Ping Plotter to track these metrics, and some days the packet seems to flip flop from one site to the other. Some sites I can visibly see one node through a trace route consistently dropping our packets, and I have no idea why. Off hours I am able to essentially send data at 40-50x speeds with seemingly no other traffic around it.

I have tried lowering MTU, clamping TCP MSS, upgrading my firewall's firmware, upgrading our IPSec algorithms, and nothing seems to help.

For what it's worth we use a FortiGate 200E with CPU/memory usage never really touching 20%.

Any ideas?



No comments:

Post a Comment