Wednesday, November 15, 2017

Troubleshooting sluggishness of 10GbE network cards connected via Thunderbolt 2

I seem to be running into an issue where 10GbE network cards connected via Thunderbolt 2 are incapable of achieving full bandwidth potential. I'm preparing to deploy 10GbE on several iMacs and this issue is throwing me for a loop.

The cards are Intel 82599ES cards in Thunderbolt-2 PCIe chassis. I have a few classic MacPro workstations, and with their cards connected directly to PCIe they are able to achieve full line speed. The iMacs -- whose 10GbE cards are connected to a Thunderbolt PCIe enclosure -- are not. I'd gladly welcome thoughts as to what I'm overlooking.

Below is my test data. Connectivity method was via SFP+ DAC cable connected straight to each unit. No switch, though for the heck of it I briefly tested with a Dell N4032. Jumbo frames enabled (MTU 9000). I've also rotated several other 82599s into the test (I have several) with no significant change in result.


---Test 0 (Control)

Unit1: LinuxBox (Ubuntu 16.04.3) with Intel 82599ES via PCIe.

Unit2: MacPro5,1 (OSX 10.11.6) with Intel 82599ES via PCIe

iPerf2 Uplink result: 10.6 Gbits/sec

iPerf2 Downlink result: 9.87 Gbits/sec

---Test 1

Unit1: LinuxBox (Ubuntu 16.04.3) with Intel 82599ES via PCIe.

Unit2: iMac 2015 (OSX 10.11.6) with Intel 82599ES via TBolt2

iPerf2 Uplink: 6.14 Gbits/sec

iPerf2 Downlink: 5.46 Gbits/sec

---Test 2

Unit1: MacPro5,1 (OSX 10.11.6) with Intel 82599ES via PCIe

Unit2: iMac 2015 (OSX 10.11.6) with Intel 82599ES via TBolt2

iPerf2 Uplink: 5.97 Gbits/sec

iPerf2 Downlink: 5.28 Gbits/sec

---Test 3

Unit1: LinuxBox (Ubuntu 16.04.3) with Intel 82599ES via PCIe.

Unit2: iMac 2015 (Windows 10 Creators Edition via BootCamp) with Intel 82599ES via TBolt2

iPerf2 Uplink: 4.27 Gbits/sec

iPerf2 Downlink: 4.40 Gbits/sec


OSX systems are running SmallTree 3.3.15 drivers. Windows system running Intel Network Adapter drivers v22.9.

I've tried to rule out as much as possible. I've also tested with different iMacs in case the machine itself had some sort of issue, and I even loaded Windows on one of them in case it was somehow an OS thing.

I'm pretty certain Thunderbolt 2 can do better than 6 Gbit per second, but I don't know what more I can do to troubleshoot this problem. Suggestions welcomed. Thank you!



No comments:

Post a Comment