RDP disconnect issues with site to site VPN, Azure to SonicWALL

With a site to site (SonicWALL to Azure) VPN up and running I can access the network in both directions without issue.  But when I RDP into a Windows Server running on Azure I get frequent disconnects - the session will drop, then auto reconnect, on the first attempt.

With a point to site VPN from Windows 10 to Azure (on the same machines) there is no issue with RDP.

This leads me to think that everything is fine with the local OS and the Azure VM and that the issue is somewhere in the site to site VPN configuration. Does that sound right?

Any idea what could be causing this and, more importantly, thoughts on how to fix it?

edit - just to be clear, the timeouts are happening while I'm actively using the connection, not if I let it sit unused.