DANOS Project - early findings

Hi networking,

Just though I'd share some of my (very) basic findings toying around with the recently open-sourced AT&T project DANOS in case you're interested in giving it a spin.

tl;dr; - don't make the same mistake I did trying to run it on Xen/xcp-ng.

So far, my experience has been very positive, but by no means have I explored all the features available or have been able to stress it. For more information, here's a link to the project:

https://www.danosproject.org/

(If this isn't the correct place, please let me know where it would be more appropriate as I am relatively new to Reddit. I did have a peek at /r/NFV but that place seems to be dead)

I'll start out with being able to install from the downloadable iso on various local hypervisors/clusters I have at my disposal:

KVM - Xeon E5 W, Intel NICs, Local Storage, Centos 7
oVirt - 3 Node, Xeon E5 W, Intel NICs, iSCSI, oVirt 4.3.7
xcp-ng - 3 Node, Xeon E5 W, Intel NICs, NFS, xcp-ng 8.0.0
ESXi - Xeon E5 W, Intel NICs, Local Storage, ESXi 6.7
Hyper-V* - Core i7-7800X, Intel NIC, Local Storage, Windows 10 Pro

*Sorry, out of real servers at home, and power budget :)

These findings were based on 4 CPU, 4GB RAM, 10GB SSD VMs. I've found that the router likes at least 2 cores to run. My first inclination is that it uses a core for management while the rest are used for processing packets.

Installed from ISO, booted from disk

KVM - Pass
oVirt - Pass
xcp-ng - Pass
ESXi - Pass
Hyper-V - Pass

Dataplane Up, Reachable (DHCP IP, Default Route, ICMP, SSH)

KVM - Pass
oVirt - Pass
xcp-ng - FAIL
ESXi - Pass
Hyper-V - Pass

The only system that the dataplane does not run on is the Xen based hypervisor, which is where I started. It will boot in HVM mode, but not in PV mode. But if the dataplane doesn't start, it's useless unless you want to have a peek at the CLI. I have tried everything I know with Xen and I couldn't get anywhere. I wasted a lot of time on this because I didn't RTFM like an idiot. Once I moved to other hypervisors, everything worked great.

I *assume* since I've had a good experience with all the KVM testing I've done, that Proxmox would run DANOS just fine, but I've got a lot running in my clusters, so that will have to wait.

FWIW, I've run VyOS on all of the above +Proxmox with much higher loads and more features running - no issues with dataplane on Xen. The dataplane itself is noticeably different from VyOS and that's where AT&T seems to have focused much of their work after the acquisition.

I plan to run some performance tests with what I've got - basic routing protocols, specifically loading a few full internet tables, throughput, compatibility, IPSec etc.

Today I will try all 3 major cloud providers and will follow up in this thread on how I make out. I suspect that since AWS is Xen-based, DANOS will not run on AWS on regular EC2 instances, but I'm interested to give it a shot on their new KVM-based Nitro hypervisor and might have better success there.

Anyone else's experience with more advanced configurations or performance testing would be appreciated! I'll share whatever I find.