VPN Tunnel woes - ASA to ISR, return traffic coming from wrong IP

So I've been setting up a VPN tunnel from my ISR to a remote ASA.
We can get all the configs matching, pass traffic, but eventually it drops, maybe after hours, maybe after days. When it drops, I'm seeing packets encapsulate from the ISR, decaps on the ASA, then encaps again on the ASA, but they never arrive back on a return trip.

The ISR is receiving instead packets from the next hop router on the ASA side.
So if my ISR is Y.Y.Y.Y, ASA is X.X.X.Y, then I'm getting packets from X.X.X.Z.

Error on ISR: %CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has invalid spi for destaddr=Y.Y.Y.Y, prot=##, spi=0x#######(#######), srcaddr=X.X.X.Z, input interface=GigabitEthernet0/0

Can't find anything that's causing this in routing on the remote side, these are obviously the responding packets from X.X.X.Y, but why are they sourced from the other IP?