VPN behind Double NAT issues

Been breaking my head over this one for a few days now, so figured I'd ask here as there seem to be quite a few knowledgeable people on here.

A brief explanation on the topic at hand. Due to the way the "ONLY" ISP around configures their switches we're having to run our VPN behind a double NAT.

The server side has 100/20 VDSL The client side used for testing has 80/20 VDSL but problems occur on fiber networks all the same. Ping is low (~10) at all times

The modem is serving 192.168.2.x whilst the server behind which is the internal network is serving 10.0.0.x. There's no way to configure the modem in bridge mode because some equipment used for IP phones is also ran off there. Reason it's not behind the internal network has to do with some external management functionality that the ISP desperately doesn't want to give up on and seems to depend on the modem being in the state that it is, so there's not much room to wiggle there.

Initially (about a year and half ago) this worked fine with proper port forwarding, etc. But about a month or two ago the modem was replaced and the external IP address changed. Ever since then it's been very problematic and downright awful.

VPN is provided by good old Windows Server 2016. It's just a basic L2TP VPN with nothing fancy. Something that like I said, has worked well for a substantial period of time.

Now I've isolated the problem down to the NAT interface in RRAS, without that configured the VPN is blazing and browsing through folders on network shares is a breeze. Opening files works like the server was right next to you.

Obviously without NAT configured (the server has two physical NICs) there would be no internet access on the internal 10.0.0.x network so that simply has to run. But once I add the NAT interface the file browsing just stops dead in its tracks. Browsing the internet, watching youtube, etc. all works flawlessy (using the external gateway) but it takes well over a minute to open any folder on the remote network. Whilst opening a folder and waiting for it, the internet in the background continues to work flawlessly.

Remove the NAT interface from RRAS and everything is all fine and dandy again.

Now I'm aware that a VPN behind a double NAT will never work perfectly but this is a bit much isn't it?

I've played with MTU sizes, tried a different protocol (PPTP), went as far as completely reconfiguring the on premise networking side of things (which means I'll now have to sacrifice a weekend to reconfigure part of the server..) all to no avail. Am I missing something blatantly obvious here or what gives?

I should probably mention the server runs on VMware and as such is virtual.