Saturday, November 24, 2018

Cheap whitebox 10GbE switch for non-profit? (x-post)

x-post from r/homelab

I'm a volunteer at two non-profits that share an office-space and server room.

We have several Dell servers running a Proxmox cluster, and we virtualise everything on top of this.

The servers have Mellanox ConnectX-3 cards in them, with SFP SR optics - but we can replace with DACs if needed.

I need a "cheap" 10GbE switch to tie it all together.

A second-hand Arista 7150S seemed like a good option - 24-ports, SFP+, and I assume one of the cheaper 10GbE options from Arista (and we have some other old Arista gear), and it supports VRF to help keep the tenants separated (they process credit card payments on-site, and also handle some sensitive personal info).

However, they're very keen to save pennies - and in this case, volunteer time is cheap/free - so DIY switch came up. Basically, spending money on hardware isn't so great, but spending money on time is OK.

The first thing I thought of was to buy an x86 box, some more Mellanox ConnectX-3's, and load it with VyOS. This video seems like a good starting guide.

But then, what about an ONIE-compatible whitebox switch?

In terms of ready-to-go software it seems like there's Cumulus Linux and SoNiC, but Open Networking Linux requires you to build a lot of it yourself, and OpenSwitch isn't ready yet. Is that all correct?

I see that a Cumulus license costs money (and remember it's hard to get money around here) - but you can get Cumulus VX and run that in a VM - so we could conceivably virtualise that on an x86 box that had SFP+ cards, right? Any issues going down that route?

Also, seems like Cumulus might tie in nicely to our SaltStack infrastructure.

What are our other options in terms of hardware for running Cumulus?

I looked on Aliexpress, and I found things like this or this but I can't find any options with SFP+ ports? (I assume this will only run Cumulus VX).

What are our hardware options for running Cumulus on bare-metal? (I read that 8/16 port vs 48 port was about the same cost, due to ASICs, but still not sure what the cheapest option here is, hardware wise).



No comments:

Post a Comment