Load balancing over two ISPs

Diagram: https://i.imgur.com/Xuox0p9.png

I'd like to balance traffic from the clients over two ISPs, so I'm wondering if the design in the diagram is valid or am I just over engineering it. I'd like to be able to use bandwidth of the both ISPs, but also get traffic locally on the correct firewall (two groups of the devices are in different cities). And to send traffic to different SaaS services etc. depending on which ISP has better response time. Seems that for example ISP 2 has smaller latency towards sharepoint.com.

Idea is to advertise first /24 with shorter AS path to one ISP and longer to other, then vice versa at the other city. ISPs advertise default + specific routes and the plan here is to import/export the default route between our VRFs. In the upper city I'd prefer the upper city ISP 1 default route over the ISP 2 default route. So that in the event that ISP 1 router is disconnected, I would still have default route in the ISP 1 VRF.

Firewalls are Fortigates so I can bundle both ISP uplinks to a single "SD-WAN interface" (as everything needs to be SD-WAN nowadays) and then apply rules to direct clients over ISP 1 or 2 depending on the traffic / response times of the ISPs and NAT the clients to those two /24 networks depending on which link is selected.

Other option is to just use single VRF and get routes from both ISPs and then just throw a coin which one to prefer by default (the one with lower latency towards M365 services probably). Simpler design but I'd lose the ability to direct traffic over best links and to utilize both ISPs. In the event one goes down we could activate stricter shaping policies to visitor network / windows updates / etc to keep the traffic below what a single ISP could handle.

​

Any thoughts?