Tuesday, August 14, 2018

Cisco 3850 pings timeout unless I add a static route

I had an issue with two Cisco switch stacks today that stumped me. Network diagram and CLI output at the end.

A client is retiring their MPLS in favor of site to site VPNs. VPNs between the firewalls have been setup but when I tried to ping from the SC switch to the OH switch the pings time out. OH to SC has the same result. Doing a packet capture on the local SC firewall, I see the ICMP request leave *and* the OH's reply come back. The firewall says it forwards the reply to the local SC switch, and has the switch's MAC address as the destination. But the switch says it never received a reply and that the request timed out.

See the CLI output, the switch at 10.10.60.1 is set to send traffic to the default gateway of 10.10.60.254, the firewall. If I add a static route to OH via the default gateway 10.10.60.254 the pings start working. If I remove the DG the pings stop working. I've verified neither firewall has any routes pointing to the old MPLS router. Things I tried:

  • Rebooting the firewalls and switches at both sides (no affect)
  • Adding a /24 route to the OH side (pings still failed in both directions)
  • Pinging the SC server at 10.10.60.10 from OH (pings were successful)
  • Pinging an OH Cisco router at 10.20.36.15 from the SC server (pings were successful)
  • Manually sourcing all pings from the respective interface (SC 10.10.60.1 and OH 10.10.36.1) (made no different)

The switches were sold to the client by their phone vendor and the client has not updated them since install, I noted the firmware dates in the network diagram. Any ideas why adding a seemingly unnecessary static route to the SC allowed pings in both directions to complete successfully?

Network diagram and CLI output



No comments:

Post a Comment