Friday, October 29, 2021

A Post About Packet Mismatches

Our company recently deployed two cellular devices which would act as bridges for downstream FortiGate firewalls. These firewalls would then build policy-based IPsec tunnels between each other.

The tunnel is unable to establish, although all IKE phases match, as well as traffic selectors. So, we thought, "can we even ping the other side of this tunnel?" We set up our ping -t and saw some pings go through, and others report a message

MISCOMPARE AT OFFSET 13 - TIME=118ms

MISCOMPARE AT OFFSET 13 - TIME=113ms

MISCOMPARE AT OFFSET 13 - TIME=108ms

So we thought this was odd, which led us to getting packet captures on either side of the tunnel with active pings running. What we discovered was pretty interesting. The packet leaving Site A would actually change (which we could see in the raw packet data shown in hex characters using Wireshark) when received by Site B. And sure enough, visa-versa. Site B would reply using the incorrect packet data which Site A would drop.

What this boils down to is something changing the packet during transit. We ran these same tests on different ISP networks and had no issue. Which leads us to believe that it is a carrier-related issue. We now have a scheduled call with some of their engineers to dig a little deeper into the issue.

It sure breaks up the monotony of our usual day-to-day so I thought I'd share it with you all for your own interest. Before today, I had never seen miscomparisons in a ping!



No comments:

Post a Comment