IPSec ESP Troubleshooting on the internet

I have a IPSec VPN issue.

I have a diagram that will help with this.

In my office, we have a VPN Firewall, connected to dual internet service providers.

At a remote site we have a VPN Firewall connected to just one ISP

• If my traffic traverses ISP #1, I can Ping, SSH and HTTPs to the WAN IP no problem. My VPN establishes, but I get 60% packet loss if I try to ping across the VPN link. The VPN has IPs on both Ends.

• If my traffic traverses ISP #2, I can Ping, SSH and HTTPs to the WAN IP no problem. My VPN establishes, and I get 0% packet loss across the VPN link.

• If I try to ping the remote office, from my office. I see packets leaving my office to the remote office, but the packets never make it to the remote office's WAN interface.

• If I try to ping my office from the remote office, I see packets arriving at my firewall and the responses. However the responses never arrive at the remote office.

So given what I see, if encrypted IPSec traffic leaves my office to the remote office through ISP #1, it fails. Through ISP #2 it works. All other traffic seems to work fine.

If this was a SSL VPN, I could just tcptraceroute and see where it fails, but I don't have an equivalent for IP Protocol 50.

I'm trying to figure out a way to show this to the ISP, because I know they are going to say PING/TRACEROUTE works, so everything else should work.

Has anyone else had a problem like this?