Network working, but not working... but working!

Hey Guys!

Today has been a reaaaally weird day at work and I'd like to share what happened today so you can share some light of what could be the cause of the issue. TL;DR at the end. Please let me know about any misspelled word or bad wording.

Environment:

• 1 Cisco 3924 Router, IOS version 15.5
• 40+ Cisco 2960 Switches, Mixed IOS versions between 12 and 15
• Router-on-a-Stick
• All switches' SVIs are in the same VLAN
• Multicast Routing enabled
• 50+ FortiAPs

This morning, for some unknown reason the WiFi stopped working for a few seconds. We didn't pay to much attention, but after a couple hours we started to notice that 90% of the switches appeared as down in the network monitor (PRTG), the APs were not working properly, but internet connectivity never failed on the end users' PCs connected via Ethernet.

After some troubleshooting we noticed that we could connect via SSH to the router and the switches that still appeared as up in PRTG. From the router we couldn't SSH to the switches that were down, but we could connect to those from one of the still "up" switches. After we connected to one of the "down" switches, it appeared as up in PRTG and then we could ping it and connect from any PC on a different VLAN.

We restarted the router and everything started working again, but after a coupled hours the issue came back.

This time, only one switch appeared as up. We could do the same process of connect to that switch and then SSH to the other for it to appear as up. We disabled Multicast and the issue persisted, so I connected my laptop to the Switches VLAN to ping every device. The ping was successful, all devices are up and we can connect to them.

So, my question is:

WTF??? What could be making the switches to appear as down when they are up? Why are they answering only to another switch? What has anything to do with multicasting? Why the ping corrected everything?

Any insight will be appreciated.

​

TL;DR: We couldn't connect to our switches from any VLAN and they appeared as down, but network connectivity was still working. After disabling multicasting on the router and pinging them from their VLAN, everything works.

Edit

Added Router and Switches models and IOS version