DMVPN over MPLS: dual-hub, dual-isp, dual-ivrf, dual-fvrf

Hi all you smart networking people, I've got a good one for ya ...

This is a 3-tier DMVPN deployment:

• Central Hubs: Dual-hub, dual-cloud (PE devices) at HQ, running MPLS/ISIS/OSPF/BGP (3 tunnels, 1 WAN)
  • 1 Global tunnel to Azure running MPLS over the tunnel
  • 1 MGMT VRF tunnel to spoke
  • 1 CUST VRF tunnel to spoke
• Regional Hub: Single device in Azure (CSR), acting as a spoke for HQ and also a Hub for downstream spokes (5 tunnels, 1 WAN)
  • All Tunnels use tunnel vrf OUTSIDE which is an FVRF created to get rid of the global routing table. The outside interface sits in this VRF
  • 1 Global Tunnel to HQ running MPLS over the tunnel, no vrf forwarding
  • 2 Tunnels to the spoke(s) for each ivrf (MGMT and CUST) using vrf forwarding MGMT | CUST
• Spoke(s): Dual-stack of ISR's, one acting the primary and one the backup using iBGP for failover between them (6 tunnels, 2 WANs, 4 VRFs). Also no global routing table. Each WAN interface uses vrf forwarding ISP1 | ISP2
  • 1 Tunnel with 2 NHRP maps to HQ via IVRF MGMT and FVRF ISP1
  • 1 Tunnel with 2 NHRP maps to HQ via IVRF MGMT and FVRF ISP2
  • The above are using different tunnel keys, NHRP auth, tunnel source and tunnel protection
  • 1 Tunnel to Azure via IVRF MGMT and FVRF ISP1
  • 1 Tunnel to Azure via IVRF MGMT and FVRF ISP2
  • Also using different tunnel keys, NHRP auth, tunnel source and tunnel protection
  • 1 Tunnel to Azure via IVRF CUST and FVRF ISP1
  • 1 Tunnel to Azure via IVRF CUST and FVRF ISP2
  • different tunnel keys ... blah blah
  • I have 4 isakmp profiles setup under 4 ipsec profiles - MGMT and CUST both have a primary and a backup and the shared keyword is being used on the tunnel interfaces, regarding which ipsec profile they need for protection. The isakmp profiles point to specific F/IVRF's to separate the VRF in the Phase II SA.
  • All tunnels in every location use RSA via in-house PKI

And here's the problem ...

The Spoke is using a cellular interface for ISP2 which is behind NAT. So I think "ok, well NAT-T is kind of ugly but works ok", so I go for it. Come to find out, Azure also does a static NAT in front of the CSR :(

I know it's a long-shot but does anyone know of a "trick" or something you've come across in the past?

I've turned on debug and find a few repeating offenders:

ISAKMP-ERROR: (1076):phase 2 SA policy not acceptable! (local 10.116.34.170)

ISAKMP-ERROR: (1076):IPSec policy invalidated proposal with error 32

ISAKMP-PAK: (1076):sending packet to x.x.x.x my_port 4500 peer_port 4500 (I) QM_IDLE

Any assistance is appreciated. I know it's kind of a cluster - I can send out configs if that would help.

Thanks, Guys & Gals!

​