We have an on going issue with one of our 2960x stacks. Around every 3 days or so a switch appears to leave the switch stack. Here are the logs leading up and following the removal of the stack:
2019-09-07T05:09:04.324-07:00 swhostname/swhostname/syslog local7.warning PST: %SUPQ-4-CPUHB_RECV_STARVE: Still seeing receive queue stuck after throttling (swhostname-1) 2019-09-07T05:30:52.226-07:00 swhostname/swhostname/syslog local7.crit PST: %FIB-2-FIBDISABLE: Fatal error, slot 4: XDR disabled 2019-09-07T05:56:25.894-07:00 swhostname/swhostname/syslog local7.warning PST: %STACKMGR-4-SWITCH_REMOVED: Switch 1 has been REMOVED from the stack (swhostname-3) 2019-09-07T06:14:54.228-07:00 swhostname/swhostname/syslog local7.err PST: %PLATFORM_RPC-3-MSG_THROTTLED: RPC Msg Dropped by throttle mechanism: type 1, class 21, max_msg 32, total throttled 0 2019-09-07T06:14:54.229-07:00 swhostname/swhostname/syslog local7.err -Traceback= 555EECz 29086C8z 23CECF0z 23D0090z 23D03ACz 2E31644z 2E2D340z 2019-09-07T06:17:55.246-07:00 swhostname/swhostname/syslog local7.err PST: %PLATFORM_RPC-3-MSG_THROTTLED: RPC Msg Dropped by throttle mechanism: type 3, class 21, max_msg 32, total throttled 1 2019-09-07T06:17:55.246-07:00 swhostname/swhostname/syslog local7.err -Traceback= 555EECz 29086C8z 23CECF0z 23D0090z 23D0394z 2E31644z 2E2D340z 2019-09-07T06:20:55.248-07:00 swhostname/swhostname/syslog local7.err PST: %PLATFORM_RPC-3-MSG_THROTTLED: RPC Msg Drthrottle mechanism: type 3, class 21, max_msg 32, total throttled 3 2019-09-07T06:20:55.249-07:00 swhostname/swhostname/syslog local7.err -Traceback= 555EECz 29086C8z 23CECF0z 23D0090z 23D0394z 2E31644z 2E2D340z 2019-09-07T06:23:55.228-07:00 swhostname/swhostname/syslog local7.err PST: %PLATFORM_RPC-3-MSG_THROTTLED: RPC Msg Dropped by throttle mechanism: type 3, class 21, max_msg 32, total throttled 5 2019-09-07T06:23:55.229-07:00 swhostname/swhostname/syslog local7.err -Traceback= 555EECz 29086C8z 23CECF0z 23D0090z 23D0394z 2E31644z 2E2D340z 2019-09-07T06:26:55.226-07:00 swhostname/swhostname/syslog local7.err PST: %PLATFORM_RPC-3-MSG_THROTTLED: RPC Msg Dropped by throttle mechanism: type 3, class 21, max_msg 32, total throttled 7 2019-09-07T06:26:55.226-07:00 swhostname/swhostname/syslog local7.err -Traceback= 555EECz 29086C8z 23CECF0z 23D0090z 23D0394z 2E31644z 2E2D340z 2019-09-07T06:29:55.224-07:00 swhostname/swhostname/syslog local7.err PST: %PLATFORM_RPC-3-MSG_THROTTLED: RPC Msg Dropped by throttle mechanism: type 3, class 21, max_msg 32, total throttled 9 2019-09-07T06:29:55.225-07:00 swhostname/swhostname/syslog local7.err -Traceback= 555EECz 29086C8z 23CECF0z 23D0090z 23D0394z 2E31644z 2E2D340z
Thing is when you check the switch via a 'show switch' command it appears to be in the stack. In addition to that I appears to be forwarding traffic.
When i issue a "show interface status" it freezes my vty sesssion. I'm at a lost what is going on, just posting this to see if someone else has had a similar experience?
The "fix" for this is to power cycle the entire stack. The reload command is not effective as it does not appear to pass the reload command to the phantom switch. After power-cycle everything is good for another 3-4 days until we see the '%STACKMGR-4-SWITCH_REMOVED: Switch 1 has been REMOVED from the stack (swhostname-3)' in the logs.
Stack is currently on 15.2(2)E6. Any insight would be appreciated.
No comments:
Post a Comment