Best way to build overlay SDN-capable network on FOSS

Hi Folks, I'm representing a network solution architect team in huge country-wide organization with numerous (~1000) branch offices and reseller points. Yes, we going to migrate an old setup with newer approaches and tech in mind. Currently we have a mixture of IPsec/OpenVPN/ugly RA and site-to-site connections. We have redundant RASes distributed over few locations. The whole idea is to provide access to some Central services (hosted in AWS) and have access from HQ to particular IP-ready equipment in remote location. That's the baseline we have now.

The way this network operates now is less efficient you might think . There is numerous outages, fat finger mistakes, HW failures, long restoration time, and much manual config required on remote side (I meant both remote hands and central initiated provisioning).

We going to build new implementation which involves few new concepts (SDN), techniques (DCI, VXLAN) and technologies (OvS/WireGuard/EVPN) . In general we going to create an overlay on top of existing underlay backbone, and soon decommission old gateway hardware. We'll start with shipping appliances which will host kvm/docker images with preinstalled software which going to build up connection to out HQ. Then we will use this appliance as reverse-proxy for accessing remote resources and finally will switch on-premise to use appliance as main GW. Most remote locations has internet-routable ipv4 address, so appliance will serve also as NAT GW (needed bw is very small).

Here is summary of connect requirements we've defined: 1. automation provisioning of RA outbound tunnel (remote side is requester) 2. direct IP connectivity inside created overlay (branch to HQ and vice-versa) (VRouter functionality) 3. possibility to isolate and to grant spoke-to-spoke flows 4. ability to tunnel L2 to remote location 5. ability to tunnel trunk with 10 VID to remote location 6. appliance redundancy for SW components (HA inside host kvm) 7. cryptography to secure tunnel creation

So, finally a questions part: - what open-source software products you can suggest (OvS)? - what SDN controller to use (we looking at Tungsten Fabric, old OpenContrai from Juniper)? - what CI/CD stack to implement (ansible/terraform) - what proactive monitoring approach to use

Any kind of input from you will be very valuable for us. Understand me correctly, we now only in the phase of considering possible options. Grats!