I'm at a loss here... (tcp performance issues across an ocean)

So, I'm having a helluva time on this.

The basic problem statement is that performance between A and Z are grossly asymettric, (11Mbps in one direction, 55Mbps in the other). And even the better of the two is no where near good enough. 100Mbps should be well within reach. Utilization on these paths is low.

PCAPs show that the culprit is bursts of packet loss. Okay. But we've checked all the counters we have available to us and every device swears it's not dropping the packets. What's most peculiar, and what I hope you guys might be able to help me out with is there's a particular pattern to the packet loss.

Here's our topology:

+---------------------+ +---------------------+ | Cloud Provider | +--------------------------------------------+ +----------------------------------------+ | Cloud Provider | | region A | | Our site B | | Our Site Y | | Region Z | | +--------------+ | | +---------------+ +--------------+ | +--------------+ | +---------------+ +--------------+ | | +--------------+ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Several | | | | | | | | | | | | | Host-A | | | | SW | | ASR1k | | | Switched | | | ASR1k | | SW (Force10)| | | | Host-Z | | | | +--------------+ (Cisco 4500) +-----+ +----------+ Hops +-----------+ +---+ +--------+ | | | | | | | | | | | | | (Force10) | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | +--------------+ | | +-------+-------+ +--------------+ | +--------------+ | +---------------+ +-------+------+ | | +--------------+ | | | | | | | | | | | +---------------------+ | | | | | | +---------------------+ | +----+----+ | | +----+----+ | | | Capture | | | | Capture | | | | Point | | | | Point | | | | B / | | | | Y / | | | | Host B | | | | Host Y | | | +---------+ | | +---------+ | +--------------------------------------------+ +----------------------------------------+ 

RTT for the whole path is about 75ms. Most of that is crossing an ocean between the "several switched hops".

Tests are ran like this iperf3 -c Host -n10M -b100M -p1501

equivalent results are found copying files via scp

Testing from A to Z gives me this:

A to Z

which is similar enough to what we get when we test from B to Z:

B to Z

(both captures taken at Y)

Very consistently we get about 300-350kB into the TCP slow start process and we drop a shitload of packets, so TCP goes into "bad touch" mode and stays there for the rest of the flow. But, broadly, there's little to no loss through the rest of the flow. In this case there was more trouble at the 4s mark, but that's not common.

Pcaps prove that this loss is happening somewhere between B and Y.

So interface counters all along the path claim zero drops. But maybe there's some carrier equipment doing the deed or we've got a "mystery hop" everyone's forgotten about or something. What confuses the hell out of me is this:

A to Y

Here we drop plenty of packets, but not 'till we're 2mB into the flow, and by then TCP can take it in stride.

Whatever is dropping the packets between B and Y isn't dropping them in the same way when they're destined for Y. And this is a consistent difference. If it was just a semi-random thing, that'd make more sense I think.

What the hell could cause this? Looking at the A-Z tests, a reasonable guess might be that there's some policing or buffer-filling going on. Some hop along the path is (deliberately or otherwise) imposing some hard limit in terms of either throughput or packets-per-second.

But any limit that might be exceeded in the A-Z tests is being far exceeded in the A-Y tests.

Hosts at A, B, Y, and Z are all in the same VRF, and there is only one path that connects them. The only L3 hops are those depicted, and they're not doing anything interesting in terms of QoS, etc... Same thing with the L2 hops. All configs are quite basic. RTT between A and Y is maybe 1ms less than A to Z. All circuits are 1G, except the server @ B, whose in-path interface is 100M (the capture interface is 1G)

Has anyone ran into anything like this? Is there a particular kind of buffer you find on L2 or L1 gear (does L1 stuff even have buffers? I wouldn't think so) that might treat flows in this way?

(and yes, I know we need capture points elsewhere along the path... working that, but it's easier said than done :/)