Philosophy on right-sizing a Cloud DC / ISP PoP

Dear sub,

I am working with a cloud provider / internet service provider in the SME market.

We are planning to build a new location and I am tasked with designing the new DC and ISP networks.

At the moment I feel somewhat overwhelmed by the available options and looking for some guidance and seconds opinions.

​

Cornerstones of my though process on requirements:

- The initial size with be about 4 racks and grow about to 10-20 racks in the next three years.

- We are serving the SME market but require a somewhat enterprise-ish setup to achieve high uptime.

- Growth comes in surges and is hard to plan ahead. So the network should be scalable and easy-to-understand.

- The ISP network-side uses a distributed PoP design, every PoP has two core routers and varying numbers of routers for peering/access and the like.

- Our existing DC locations have layer2 spine-leaf DC networks implemented, as it spans only a couple of racks.

- A layer3 spine-leaf network is high on the wishlist though.

- The DC network will host both VMware NSX with VXLAN as well as Openstack with VXLAN, where in both occations software VTEPs are used "within" the cloud environments.

- There will be some 'traditional' workloads, to something like VXLAN on the physical network is required.

- Budget is tight as always, so we are not even looking at Cisco, maybe at Arista/Juniper but more likely towards affordable vendors like FS.com and Mikrotik.

​

Now here's the struggle:

The spine leaf network in itself will be rather small (2 spines, initially 8 leafs) and might grow into a 40-ish amount of leafs (2 per rack).

Is it worth the 'overhead' going all L3, or on the other hand is it worth the 'risk' of building an L2 network (again)?

Most L3 spine-leaf designs go eBGP all the way. Spines into one private ASN, leafs into another private ASN (or more). Can I / should I dual-use my beefy spine switches to also act as core routers in the ISP PoP? We carry only a small amount of routes internally. I sense trouble having both our public ASN and the private spine-ASN on the same boxes. Could be a management nightmare, even if technically possible. But it is appealing from a budget perspective.

The latest and greates in L3 spine-leafs seems to be the introduction of EVPN. Does that make sense for a small deployment or should we stick with 'only' VXLAN?

We do have access to two independent DC rooms at the new location and we can utilize racks in both DCs. Given the small footprint it seems a total overkill to build two completely independent networks in terms of required components. But spanning the spine-leaf network over both rooms bears a cost in the many required cross connects.

I appreciate any thoughts and suggestions. I got a feeling to have driven into a mental corner on how to right-size this :-)