Looking for the right solution to replace internal MPLS & VPN routers with something more publicly accessible to get less "on-prem"

Hi all, I'm just in the beginning stages of some brainstorming and research..

I work IT at a very small financial institution and we have had a pretty on-premises-centric infrastructure for the last 10 years (most everything had been on-site such as AD, Exchange, SQL servers, other systems) but now the last 3 years we have had more and more services pulled out and going hosted and we've significantly reduced server hardware quite a bit.

One of the things that has stuck around are our connections to our main 3rd party vendor's datacenter, of which there are two gateways: 1, a primary MPLS router connected to private fiber lines; and 2, a failover VPN router which goes over one of our Internet connections. These physical connection points exist inside the building and therefore tie us to the building in that if users need to work remote, they must first remote into the company's environment from a company provided laptop to their individual company workstation (we use Citrix VDA) and then from there, they access the apps which connect to the datacenter.

What I am trying to figure out is if there is a common solution/implementation which would bring those connection points out of the building to be accessible over the Internet. My thought is that I would like to free up the dependency on connecting into the company's internal network before being able to access the apps that access the remote datacenter. This would eventually get rid of the 1:1 ratio of laptop to PC, and have people just be able to use one system (laptop or PC) if necessary, and be able to securely access everything over the internet.

Also, if the building burns down, people will still have access to all services.

Edit: I suppose the main thing would be checking what access methods the vendor actually supports, but I also am just trying to figure out what specific things to ask for so I get us heading down the right path.

Edit 2: wild shot in the dark, I assume I just need to ask if they support a public VPN connection, web-based, so there is no hardware necessary.