Site-to-Site VPN drops and latency significantly increases anytime files are downloaded behind Cisco ASA.

I have a Cisco ASA 5506-X that has an outside interface being provided internet from a fiber 100/100mbps line. Over the past month or so something odd has changed.

Anytime I initiate a download on an inside client behind my inside interface OR even a laptop directly connected to an inside-test interface on the ASA (that routes out the same outside line), my VPN tunnel suddenly drops multiple packets and or ping requests are 800+ ms. This is normally 42ms. If I'm running constant 4.2.2.2 ping requests, these are normally at 6ms but will spike way beyond that. The second the download is stopped, traffic resumes back to normal.

Per TAC I upgraded to 9.10(1)30. No Firepower services running. There are no interface errors. I have a backup internet line and if I connect to that interface there are no issues.

I have yet to plug my laptop directly into the ISP as I'm having a hard time finding downtime but wanted to see if anyone had thoughts on what may be causing this. Seems very odd this hasn't always been a problem.

When I can plan an outage I plan on connecting my laptop directly to the Outside interface and testing. Then plugging directly into the ISP without the ASA in the picture. Finally configuring the outside interface on a different port on the ASA. Any suggestions on what I can test before then would be appreciated.