Tuesday, December 12, 2017

MMA - Midnight maintenance Adventure, seeking IOS advice!

Midnight Maintenance Adventure (MMA)

Imagine, you opened a plain brown box and unpack a sleek 3650-8X24UQ, get real excited and deploy it removing a perfectly stable 2960XR from production. Things work fine for about 48 hours, then the 16.3.3 IOS switch spontaneously crashes. 110 customers file support tickets. Support department head throws a fireball at you doing 50 HP of damage.

You do a 2am upgrade to 16.3.5b. Network engineer calls you at 7am to say it looks bad, you revert to 16.3.3 and your mac address table immediately goes from 320 MACs back to 740. Things look OK, but you know 16.3.3 has issues.

You desperately ask on a forum, what about 3.7.5, it works so well on the 3850 switches?!? You are warned about a demon named ROMmon. You are advised to move on swearing-like-a-sailor-kicking-and-screaming. Rereading the advice, the person somewhere in the depths of the Internet has edited their post, and the information is different. It just says file a TAC ticket and the swearing-like-a-sailor-kicking-and-screaming line was removed from their post. You are confused it is 4am.

You file a TAC ticket. Turns out 16.3.3 is cursed with Entropy exhaustion (CSCvc88106) and Memory Loss (CSCvb48912). You look at the memory graphs you are diligently graphing and predict you have until 10am before the next spontaneous reboot.

Asking the wise TAC sage what version to run, they respond with, "As Cisco TAC engineers we cannot recommend any specific code release." Confusion spell gets worse.

Make a choice, adventurer, what do you do?

  • 16.3.3 - stay with memory leak, reboot every 48 hours + port isolation busted (CSCun82902)
  • 16.3.5b - half the mac addresses on network become unreachable - doesn't actually crash
  • 16.6.2 - dark horse, yet to deploy. Downloading right now.
  • 16.2.2 - old version, has other bugs
  • 3.7.5 - not offered as a download, try to brick device with 3850 IOS and risk future ROMmon incantation

bonus option Do you turn off IPv6 everywhere for good luck?



No comments:

Post a Comment