Can only send a maximum of ~3mbps from our head office to site with a CentOS router either side. All other traffic to/from elsewhere saturates the link -- Any ideas?

Hi all,

Been looking at this loosely for a couple months now. We can't send more than 3mbps/tcp to one of our sites from the head office yet both can saturate their respective links up/down just fine for any other communications. For the nginx reverse proxy, and the inter-site VPN (tcp OR udp) this has been a slow nightmare. For testing I've been using iperf3 with various window sizes and mtu changes with no budging on this ~3mbps limit with direct tests.

Scope of problem

• NBN FTTP 250/100 service
• * CentOS gateway (Public IP on onboard gbit interface) LAN behind it on different interface
• * Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168]
• • Driver: r8169

• NBN Wireless 75/10 service

• • CentOS gateway (Public IP on onboard gbit interface) LAN behind it on different interface
• • Intel Corporation Ethernet Connection I217-LM [8086:153a]
• • Driver: e1000e

---

MTU is set to 1500 on each side which is has been appropriate for communicating to any other public host just fine, saturating connections as expected.

Both services (Tested direct on the router's themselves) can saturate their respective up/downlinks to public iperf3 testing services and any on speedtest.net. As far as I'm concerned, the network connections are great and have been serving well beyond expectation and the routers are doing a fine job.

• Sending from the NBN Wireless service router to the FTTP service manages to saturate its 10mbps upload speed which is max, so that's good.

• Sending from the FTTP service router to the NBN Wireless service seems to cap out at around 3mbps/tcp instead of getting anywhere near the NBN Wireless connection's max 75mbps downstream.

• • iperf3 shows zero retries for these tests, like the FTTP router isn't even trying.

Both routers are running the stock CentOS 7 sysctl window sizes and rmem/wmem window scaling. Playing with it in memory hasn't resulted in any change and they've been rebooted back to default since.

Both services public IPs are same /22 subnet of the ISP and pass through their border network gateway (BNG02) however this problem has been ongoing since before this change were made. Right now the two services a beautiful symmetrical single hop route between each-other through BNG02.

This problem only seems to be present on direct communication from the FTTP router's public interface to the NBN Wireless router's public interface. The most important part for our infrastructure.. and to our only other office. The one place that these links matter.

Some gotchas I've noticed

• Port Forwarding the iperf3 port to an internal host (5201/tcp) and running the test through the FTTP gateway, via an internal centos7 machine successfully saturates the NBN Wireless connection's downstream (What we want to see from the FTTP router itself)

• • I tried applying all the same iptables rules (WAN interface name matches and copied all the sysctl rules to this newly spawned VM... the next iperf3 it did from inside the FTTP LAN was still perfect to the NBN Wireless connection.

Plus, live booting the FTTP router itself into an Archlinux usb stick temporarily and ran iperf3 again and was also able to max out the NBN Wireless link again.. this time on the router metal itself.

Plugging in a little USB3 ethernet adapter into the FTTP router and changing the iptables rules to use that as the WAN and dhcp'ing on it (Appeared as enp0s20u1) to make that the public-ip wan interface? Perfect iperf tests again.

All of this seems to stem from the onboard Realtek [10ec:8168] Gigabit Ethernet Controller.. but I am not sure what just yet.

Been completely mind blown about this for a few months and taking the occasional glance hasn't gotten me very far. Our ISP has been very helpful in trying everything in the toolbox to get some decent connectivity but everything seems to point to something unique to the FTTP CentOS and maybe the driver? But port forwarding to an internal machine and iperf'ing to that is perfectly fine.. through the same wan interface in the end. Or live-booting into a newer distro on the same interface, still good.