Nexus - interface Eth1/4 has gone down. Reason: Echo Function Failed

Hello everyone,

​

Had an event yesterday that caused an outage in our DC. I was wondering if anyone could provide some more insight into what may have caused this to happen.

​

Here's what I know -

​

No changes made to environment

Legacy design of Nexus 3k's 2 Core 2 Service Edge

Core's lost sight of SE's which caused some SVI's to be unreachable

Digging deeper into the SE's I found the following BFD logs -

​

2019 Oct 15 15:53:42 DAL-SE-1 %BFD-5-SESSION_STATE_DOWN: BFD session 1090519093 to neighbor 10.255.255.93 on interface Eth1/4 has gone down. Reason: Echo Function Failed.

2019 Oct 15 15:53:46 DAL-SE-1 %BFD-5-SESSION_REMOVED: BFD session to neighbor 10.255.255.93 on interface Eth1/4 has been removed

2019 Oct 15 15:56:44 DAL-SE-1 %BFD-5-SESSION_STATE_DOWN: BFD session 1090519085 to neighbor 10.255.255.77 on interface Eth1/3 has gone down. Reason: Echo Function Failed.

2019 Oct 15 15:56:48 DAL-SE-1 %BFD-5-SESSION_REMOVED: BFD session to neighbor 10.255.255.77 on interface Eth1/3 has been removed

2019 Oct 15 16:01:05 DAL-SE-1 %BFD-5-SESSION_MOVED: BFD session 0x4100003d: Installed on LC 1

2019 Oct 15 16:01:05 DAL-SE-1 %BFD-5-SESSION_CREATED: BFD session to neighbor 10.255.255.93 on interface Eth1/4 has been created

2019 Oct 15 16:01:05 DAL-SE-1 %BFD-5-SESSION_MOVED: BFD session 0x4100003e: Installed on LC 1

2019 Oct 15 16:01:05 DAL-SE-1 %BFD-5-SESSION_CREATED: BFD session to neighbor 10.255.255.77 on interface Eth1/3 has been created

2019 Oct 15 16:01:10 DAL-SE-1 %BFD-5-SESSION_STATE_UP: BFD session 1090519101 to neighbor 10.255.255.93 on interface Eth1/4 is up.

2019 Oct 15 16:01:12 DAL-SE-1 %BFD-5-SESSION_ACTIVE_PARAMS_CHANGE: Local parameter of BFD session 0x4100003d has changed Disc 0x4100003d [[protocol 1 if_name Eth1/4 if_index 0x1a003000 iod 0xa 5effff0a:0:0:0=10.255.255.94 -> 5dffff0a:0:0

:0=10.255.255.93]] TX(2000000): RX(2000000): ST(2000000), Mult(3), Ver(1)

2019 Oct 15 16:01:12 DAL-SE-1 %BFD-5-SESSION_ACTIVE_PARAMS_CHANGE: Local parameter of BFD session 0x4100003d has changed Disc 0x4100003d [[protocol 1 if_name Eth1/4 if_index 0x1a003000 iod 0xa 5effff0a:0:0:0=10.255.255.94 -> 5dffff0a:0:0

:0=10.255.255.93]] TX(2000000): RX(2000000): ST(2000000), Mult(3), Ver(1)

2019 Oct 15 16:01:13 DAL-SE-1 %BFD-5-SESSION_ACTIVE_PARAMS_CHANGE: Local parameter of BFD session 0x4100003e has changed Disc 0x4100003e [[protocol 1 if_name Eth1/3 if_index 0x1a002000 iod 0x9 4effff0a:0:0:0=10.255.255.78 -> 4dffff0a:0:0

:0=10.255.255.77]] TX(2000000): RX(2000000): ST(2000000), Mult(3), Ver(1)

2019 Oct 15 16:01:13 DAL-SE-1 %BFD-5-SESSION_STATE_UP: BFD session 1090519102 to neighbor 10.255.255.77 on interface Eth1/3 is up.

2019 Oct 15 16:01:13 DAL-SE-1 %BFD-5-SESSION_ACTIVE_PARAMS_CHANGE: Local parameter of BFD session 0x4100003e has changed Disc 0x4100003e [[protocol 1 if_name Eth1/3 if_index 0x1a002000 iod 0x9 4effff0a:0:0:0=10.255.255.78 -> 4dffff0a:0:0

:0=10.255.255.77]] TX(2000000): RX(2000000): ST(2000000), Mult(3), Ver(1)

​

​

I'm not terribly familiar with BFD but have checked the firmware we're running and there's no bugs related. Any insight would be appreciated or if you need more data please let me know.