SMB throughput optimization over VPN (high latency)

Hello friends,

We are trying to squeeze out every bit of performance from SMB over VPN. We all know that SMB is very chit-chatty so latency really kills the performance. We are digging into optimization for VPN and so far we only tried disabling bandwidth throttling on high latency network (HKLM\System\CurrentControlSet\Services\LanmanServer\Parameters\DisableBandwidthThrottling). This helped a bit with performance, but there are still spikes and troughs which we are trying to figure out the reason for.

E.g: This is a test over VPN. We have 10G links on prem and we are testing from a GCP VM which again has over 1Gbps downlink and 7-800Mbps uplink. Latency is around 70-80ms between these two

.Here are the iperf numbers:

single stream: 12-15 Mbps

8 Steams: ~ 90 Mbps

16 Streams: ~ 180 Mbps

And now SMB: Its usually doing around 100Mbps, but there are few spikes going up to 200Mbps. Graph

And some runs go even upto 500Mbps. Graph

But then the very next run is at snails pace barely holding around 10Mbps. Graph

Between these runs nothing changes on network side, same latency (~75ms), same storage (which is backed by SSD flash, no slowdown there at all). So curious where the spikes in performance are coming from. What else can we do to optimize this further.

I've posted to r/sysadmin but posting it here as well since someone here most likely has more experience with this sort of thing. If its not allowed, please feel free to remove it.

Cheers and thanks all for help in advanced.