Site to Site VPN/ASA issue

I have a strange issue that I cannot seem to figure out. At work, we are deploying a site to site VPN with a Cisco ASA 5508 and a stack of two Cisco 9300s. Our point to point fiber circuit is not ready yet, so we need to use the existing connection. Before I connect the ASA to the demarc I can ping from the switch to the ASA without an issue. When connected the ASA builds the tunnel just fine. The ASA can ping anything at the main site, but pings between the switch and ASA fail about half of the time making the connection unusable.

We are just passing one subnet over the tunnel, and it does not appear anywhere else in our network. The firewall can still reach everything just fine on both the internet at at the main site, but anything on the switch cannot.

I will be back on site tomorrow to work on it further, as it is not a downtime tolerant site through the week. I was wondering if somebody had any suggestions? I have tried different ports and cables. Im not seeing any issues with the config, and NAT appears to be working as intended.

Thanks in advance!