[cisco] P3DMVPN SA's Dropping between two spokes - NHRP Peer session stays up

Hi all,

Out of the blue yesterday two of my spokes in my P3DMVPN stopped being able to communicate between themselves. All other spokes are still able to communicate with each other AND communicate with the two spokes having the problem. The Hub is still able to communicate with both spokes as well.

Let's call the spokes Tom and Jerry

Tom and Jerry learn about each other through NHRP and share routes using iBGP, as do the rest of the Spokes.

After a bunch of troubleshooting various possibilities, I've narrowed down the symptoms a bit:

If there is no NHRP peer session between Tom and Jerry and I send some packets (ping) from Tom through the tunnel, the Session establishes, and ~15 pings will go through until they stop. From there on out, no communication between the two will work over the tunnel until I clear the dmvpn peer session. Then the same pattern repeats. Pings work for 15 frames or so, and then they stop until I manually clear the session or the timer hits and they clear on their own.

#debug cryp ikev2 error
#debug cryp ipsec error
#terminal mon

Shows the SA go down when I clear the dmvpn peer session

Tom#clear dmv sess peer Jerry 

%IKEV2-5-SA_DOWN: SA DOWN

And then comes up after I start a ping

But without fail, a few seconds in, the SA goes down again, with a few errors I can't make reason of.

Tom#ping Jerry repeat 1 Type escape sequence to abort. Sending 1, 100-byte ICMP Echos to Jerry, timeout is 2 seconds: ! Success rate is 100 percent (1/1), round-trip min/avg/max = 24/24/24 ms Oct 23 17:18:51: %IKEV2-5-RECV_CONNECTION_REQUEST: Received a IKE_INIT_SA request Oct 23 17:18:51.267: IKEv2-ERROR:Failed to retrieve Certificate Issuer list Oct 23 17:18:51.275: IKEv2-ERROR:Failed to retrieve Certificate Issuer list Oct 23 17:18:51.279: insert of map into mapdb AVL failed, map + ace pair already exists on the mapdb Oct 23 17:18:51: %IKEV2-5-OSAL_INITIATE_TUNNEL: Received request to establish an IPsec tunnel; local traffic selector = Address Range: Tom-Tom Protocol: 47 Port Range: 0-65535 ; remote traffic selector = Address Range: Jerry-Jerry 7 Protocol: 47 Port Range: 0-65535 Oct 23 17:18:51.451: IPSEC(ipsec_get_crypto_session_id): Invalid Payload Id Oct 23 17:18:51.451: IKEv2-ERROR:Error constructing config reply Oct 23 17:18:51: %IKEV2-5-SA_UP: SA UP Oct 23 17:18:51.451: IPSEC(ipsec_get_crypto_session_id): Invalid Payload Id Oct 23 17:18:51.639: IPSEC(ipsec_get_crypto_session_id): Invalid Payload Id Oct 23 17:18:51: %IKEV2-5-SA_UP: SA UP Oct 23 17:18:51.639: IPSEC(ipsec_get_crypto_session_id): Invalid Payload Id Oct 23 17:18:51.651: IPSEC: sa null Oct 23 17:18:51.651: IPSEC(send_delete_notify_kmi): not sending KEY_ENGINE_DELETE_SAS Oct 23 17:18:51: %IKEV2-5-SA_DOWN: SA DOWN 

This is ONLY happening between Tom and Jerry and I'm worried about the rate of hair loss I'm currently experiencing....help.