Tuesday, May 4, 2021

Need help diagnosing packet loss (Please take pity, I've been googling as much as I can)

Good afternoon. If you take a moment to read my story, I would be very grateful.

I work for an ISP and was tasked with creating our own in-house speed test server.

They gave me a server that they put an Intel 82599es 10 gigabit ethernet adapter which is configured with an internal private interface (mainly for ssh from local network machines) and the public interface with a public IP.

Software for speed testing includes iperf3 and the open source "librespeed speedtest" server which uses Apache and php. IPTABLES is also being used to only allow customer ip addresses to connect to the speedtest server.

The problem I'm facing, is that I have packet loss when pinging the public interface only. This causes the speedtest webpage and iperf3 connections to commonly have to be reloaded 2-3 times before a connection is made.

For some reason, I average between 11-13% packet loss. If I ping OUT from the PUBLIC interface, there are no issues. The private interface has no issues in or out. It's only when I do a ping from something else to the public interface.

When I do a continuous ping, there will be numerous passes. Then, for a short time, every other ping will fail. This burst of intermittent failure is fairly consistent. Seemingly happening after every 35-55ish successful pings. Theres always a round of successful pings, and then 3-4 failed ones, then a round of successful pings, etc.

This packet loss happens with or without iptables enabled.

Here's a picture of the ping returns. Notice the icmp sequence number and how it begins to fail every other ping. http://imgur.com/a/wKKrnZh

Any help or ideas would be appreciated. I've been pulling my hair out trying to fix this. If you need more information, please let me know and I will try to get it for you.

Thank you in advance!



No comments:

Post a Comment