iPerf throughput testing - Cisco UCS

Running a new HyperFlex cluster connected to UCS. Hosts have ESXi installed. Hosts are using VIC 1457 with 25Gb uplinks to UCS. I've deployed 2 Ubuntu VMs (8 vCPU, 24Gb RAM) and I'm running Iperf3 between the two. VMs are pinned to the same FI (except for the final test where they are on the same host). Latency is <0.3ms between VMs. No other VMs except the HX storage controllers on these hosts. These are the results I'm getting:

+-------------------------+-----------+-----------+------------+---------------+ | VM MTU | Iperf Test | -t 30 | -t 30 -R | -t 30 -P 4 | -t 30 -P 4 -R | +-------------------------+-----------+-----------+------------+---------------+ | 1500 | 7.97 Gbps | 8.45 Gbps | 6.08 Gbps | 5.56 Gbps | | 3000 | 17.4 Gbps | 16.6 Gbps | 19.2 Gbps | 18.8 Gbps | | 9000 | 24.7 Gbps | 24.7 Gbps | 24.7 Gbps | 24.7 Gbps | | 1500 - VMs on same host | 34.2 Gbps | 38.3 Gbps | 42.3 Gbps | 43.1 Gbps | +-------------------------+-----------+-----------+------------+---------------+ 

As you can see, performance is good with jumbo MTUs. I would expect better results for a standard MTU. To me, it looks as though enabling jumbo MTUs is masking an underlying issue.

I've tried running multiple instances of iperf as recommended here but still the combined bandwidth equals the same as the single instance tests. I'm using TCP tests as UDP Iperf3 tests seem to be limited by the VM CPU (UDP is maxing between 2-4Gbps and I'm seeing CPU at 100% in top).

iPerf3 isn't reporting any retransmissions and there are no buffer drops on the hosts. I've applied the recommended ESXi performance-tuned ethernet adaptor policy to the hosts to increase the Rx Queues to 8, buffers to 4096 and enable RSS.

I've tested with Iperf2 - similar results.

Where could the bottleneck be? Could I be hitting some kind of hardware pps limit? These are brand new M5 C-series hosts and UCS 6454 FIs. I would be expecting better performance using a standard MTU. I've spoken to our Cisco account team and they aren't able to provide any benchmark values or expected performance figures.