Friday, October 1, 2021

Connecting three servers with QSFP+

Hi!

I have a chance to get three DL380 G9s, each with a HPE 544 dual-port QSFP+ FlexLOM adapter. The entire system should be a (probably Proxmox based) virtualization cluster with two main VM hosts (with all of the client VMs) and a third smaller "office"/"admin" VM host, and I'd like to use the 40Gbit links for replication inside the cluster.

However, I'm not sure whether I also need two QSFP+ switches (i.e., server{1,2,3}-port1 on switch1, server{1,2,3}-port2 on switch2) or if I can manage to connect them into the cluster with a physical loop (i.e. server1-port1 <-> server2-port1, server2-port2 <-> server3-port1, server3-port2 <-> server1-port2) and use Linux's bridging and STP to achieve redundant links.

It is ok if there are performance degradation in case of failures (e.g. if the cable between vmhost1 and vmhost2 fails, it is ok that the CPU on officehost can't handle 40Gbit in & out), but it would be great if it could utilize the 40Gbit links in the fully-functional case. This also raises the question, if the STP tree is global (i.e., all bridges use the same tree and ignore the same links) or local (i.e. vmhost1 uses its direct link to vmhost2, but falls back on the link via the officehost bridge).?

Do you have any recommendations about the most cost-effective and power efficient way to connect the three hosts?

Thanks!



No comments:

Post a Comment