As many of European people will know, we're experiencing some serious heat at the moment.
Yesterday two of our data centre air con units gave up causing our data centre to go from a reasonable 18-22 Degrees C to 60+ in just a couple of hours. The argon suppression units "exploded" causing much confusion, leaving people thinking something had literally blown up. "The floor lifted" I was told by some office workers. I, putting two and two together figured the smell of burning plasting (actually the smell of argon) and the "explosion" was enough to evacuate the surrounding offices. Lives are far more important.
Eventually I learned there wasn't a fire or an explosion, in fact the suppression units literally blast out the good stuff and the smell is a bit like burning plastic.
Anyway this resulted in an attempt to gracefully shut down as many servers as possible then failing devices to an alternate data centre where required.
At this point I'd like to add, Nexus 7ks literally just chill in that kind of temp. They reported "minor" temp alarms and happily continued to forward traffic. In fact, all of our networking equipment continued without a sweat. Some fortigates hitting near 80c. My part here was to shut down any slave devices with the hope of preserving a unit or two and reduce the contribution to "global warming".
We had electricians and engineers bringing all kinds of industrial fans to channel the hot air out. That helped drop 10 degrees or so.
Eventually the air con engineers turned up and sorted it promptly. The temperature was back to normal in 30 mins or so. A day later systems are almost back to normal. Some databases kicking off about something or other. ..Sensitive things ay?
Oh and sadly we lost a fortianalyzer. God rest it's connectionless body.
All in all, not something you'd expect, so it poses a couple of questions, how resilient is your network and how would you respond to such an event?
Finally, my dear networking redditors, what is your data centre disaster?
No comments:
Post a Comment