What is scarier than 100's of alarms in your NMS? When your NMS becomes inaccessible.

Yesterday night/today morning was 'fun'.

I was doing something while looking at our NMS after work, when suddenly I notice a small almost invisible ajax popup about not being able to refresh the data in the NMS. I tried ctrl+f5 and our NMS did a 404. Half our DC entered into err-disable.

We spent the entire night going from switch to switch doing shut/no shut repeatedly until stuff started to work. Next week we will do diagnosis ASAP.