Wednesday, October 13, 2021

Figuring out why a packet went missing

I'm investigating an issue where someone at a specific location goes to a website to submit an order, submits that order which then opens an SQL query to a data base outside that location (the server) which then send back an SQL query that will commit that order back to the original location (the client). The communication up to the point of the commit packet is all fine when looking at a packet capture at the the server side and the client side, no packets missing and 3 way handshake is good, but the packet to commit the change is lost every times you attempt do submit the order. I've attempting this multiple times in testing and the tcp stream is the same each time.

I've confirmed that at the server location the packet is leaving the egress point ie the firewall in this case by captures at that location.

I attempted to confirm if the packet at least gets to the clients side router/firewall but the ubiquiti device on site can't capture all the packets (captures 4 out of about 15 that the client device can capture in the specific TCP stream. and its always the same 4 packets weirdly).

I'm really at a loss now to figure out how to figure out who is dropping this packet and all the retransmits.

Here is the TCP stream with the client on top and the server below. The packet that does the commit is the 1808 length packet. Note this length is an anomaly of tcpdump running on the firewall putting packets 18155 and 18156 together when it records the length. The packet is within a 1500 MTU still in reality. That packet along with the other retransmit never show up at the client side. On the client side he just TCP keep alives for about 60 seconds and then gives up. Its like once this packet is lost they no longer can see each other.
Also I know the source ports are different in this image I was doing a bunch of testing and didn't keep two captures in the same test unfortunately. Each time the resulting stream is the same.



No comments:

Post a Comment