Monday, August 10, 2020

Slow Speeds Across Multi-Gigabit WAN Connections

Hello great fellow networkers; I need some help here.

I have been working on an issue for many months now I cannot seem to make any sense of. We have two datacenters on opposite sides of the United States. Both datacenters are interconnected via a few point to point connections as follows:

  • A) Zayo WAVE 10Gbps
  • B) Zayo L2 5Gbps
  • C) Cogent L2 5Gbps
  • D) GRE Tunnel(Level3 and CTL)

When I send traffic from datacenter A(West Coast) to datacenter B(East Coast) I get approx 700Mbps(iperf3 tcp single stream test). When I send traffic from datacenter B to datacenter A I get approx 150Mbps(iperf3 tcp single stream test).

Please see THIS diagram; The lines represent the following:

  • Green - An iperf3 test, single stream, TCP from VM-A to a server directly plugged into datacenter B's ASR 1006 that achieves 8.5Gbps both directions. Interfaces on ASR1006 reflect that the traffic is hitting the device as expected.
  • Purple - An iperf3 test, single stream TCP from VM-B to VM-C where VM-C's default gateway is the ASR 1006 at datacenter A. Results are 9.2Gbps both directions. Interfaces on ASR1006 reflect that the traffic is hitting the device as expected.
  • Red - An iperf3 test single stream TCP from VM-A to VM-B that only gets 150Mbps one way and 700Mbps the other way. If I do -P 10 to run 10 parallel streams, I can achieve 4.5Gbps(When using WAN link C)
  • Pink - An iperf3 test single stream TCP from VM-B to the test server plugged directly into datacenter B's ASR 1006 gets 150Mbps one way and 700Mbps the other way.

The WAN LINK is where the 4 connections between the datacenters exist. It is OSPF on the 4 links with BGP on top for the actual routing. It does not make a difference what link is being used on the WAN connection. The speeds are the same.

I have thoroughly gone through every piece of networking equipment that is in play here. There are no CRCs, no pausing, no queuing, no fragmenting, no over subscription, light levels are good, no dropping on the carrier side, etc... I've contacted all 3 providers of the WAN links and they've sent techs to do throughput testing and achieved the speeds we pay for. I've done packet captures and I see lots of "TCP ACKed Unseen Segement" even though tracepaths show that the path is symmetrical. The packet captures have come from datacenter B's ASR 1006. I've consulted this with many other co-workers, friends, etc and we cannot figure out what may be causing this issue. I will provide any additional information you may need to assist me with this issue.



No comments:

Post a Comment