RDMA/SET - In over my head

Hey guys,

​

Company helpdesk guy here. Still in school and still learning so I'm not going to pretend like I 100% understand this but I've been recruited to fix a mess from a previous employees implementation (Yay!). We have 2 servers in a hyper-converged setup to utilize S2D for our database both equipped with Mellanox Connectx-3 Pro NICs. The job presented to me is this: We don't think RDMA working, and we don't know why. Find out.

​

Unfortunately this previous employee didn't believe in documentation.. so I don't have a clue what he did exactly.

​

So, with what limited knowledge I have and what I've been able to research online.. here's what I've found

- Servers are running 2016 Datacenter

- Both of our servers have the DCB feature enabled

- Mellanox NICs are RDMA/RoCE capable

- Inside Hyper-V Manager under Virtual Switch Manager there is a vNIC named "SETswitch"

- Cluster-01 has a SETswitch pointed to Mellanox ConnectX-3 #2 adapter.

- Cluster-02 has a SETswitch pointed to Mellanox ConnectX-3 #1 adapter

​

I've used PowerShell to look at the way it is currently configured, and have attempted to follow along with a Microsoft doc that I believe the previous employee used to create this setup

​

https://docs.microsoft.com/en-us/windows-server/virtualization/hyper-v-virtual-switch/rdma-and-switch-embedded-teaming#bkmk_modes

​

PS C:\Windows\system32> Get-VMSwitch Name SwitchType NetAdapterInterfaceDescription ---- ---------- ------------------------------ SETswitch External Teamed-Interface NIC4 - Backbone External Intel(R) Gigabit 4P I350-t rNDC #3 NIC2 - 10.0.99.X External Intel(R) Gigabit 4P I350-t rNDC 

​

PS C:\Windows\system32> Get-VMSwitchTeam -Name "SETswitch" Name NetAdapterInterfaceDescription TeamingMode LoadBalancingAlgorithm ---- ------------------------------ ----------- ---------------------- SETswitch {Mellanox ConnectX-3 Pro Ethernet Adapter #2, Mellanox ConnectX-3 Pro Ethernet Adapter} SwitchIndependent Dynamic 

​

PS C:\Windows\system32> Get-NetAdapterRdma Name InterfaceDescription Enabled ---- -------------------- ------- vEthernet (NIC2 - 10.0... Hyper-V Virtual Ethernet Adapter #5 False vEthernet (NIC4 - Back... Hyper-V Virtual Ethernet Adapter #4 False vEthernet (SMB_2) Hyper-V Virtual Ethernet Adapter #3 True vEthernet (SMB_1) Hyper-V Virtual Ethernet Adapter #2 True vEthernet (SETswitch) Hyper-V Virtual Ethernet Adapter False SLOT 3 Mellanox ConnectX-3 Pro Ethernet Adapter True SLOT 3 2 Mellanox ConnectX-3 Pro Ethernet Adap... True 

​

PS C:\Windows\system32> Get-SmbClientNetworkInterface Interface Index RSS Capable RDMA Capable Speed Friendly Name --------------- ----------- ------------ ----- ------------- 10 True False 20 Gbps vEthernet (SETswitch) 14 True False 20 Gbps vEthernet (SMB_1) 20 True False 20 Gbps vEthernet (SMB_2) 

​

My best guess is that the previous person who created this virtual SETswitch pointed the SET directly to the Mellanox NICs rather than pointing SMB_1 & _2 to each of the Mellanox NICs. My understanding is the SETswitch should contain SMB_1 and SMB_2.

​

Hopefully I've been able to explain this well enough that it is understandable. Let me know if I need to clarify anything. If anyone can give me direction (Additional commands I can research and run) or if you see anything obvious as to why the SETswitch is not returning as RDMA capable (Even though the host NICs are showing enabled) would be great.

​