Friday, May 17, 2019

Change Controls Gone Awry; Or how to make a 5 minute fix take 12 days.

My employer has recently started requiring change controls for any configuration change. Typing config t and hitting enter without a change control could result in termination. Each change requires an implementation plan, a validation plan, and a rollback plan. 40 character minimum each. The board only meets on Wednesdays to approve changes, and the changes have to be in 48 hours before the start of the board meeting. Implementation, actually making the config changes is only allowed on the weekends.

Want to change a port description? Change control with 3 tasks with a 40 character minimum describing the change, a validation plan, a roll back plan, and a detailed description of the potential impact of the change. Want to shut no shut a err disabled port? Same deal... You get it. All inactive ports are disabled. To enable any of these ports you will need a change control.

To put this in perspective if I want to disable a port and change a port description to "Rogue DHCP Device! - Do Not Enable!" I need to know 1 week in advance and submit 3 documents, with a total of 120 characters describing the change. If I don't meet these requirements it doesn't even make it to the board for review and if I ignore the process I could be terminated.

On the fly troubleshooting steps like altering duplex settings or port speeds and the like for malfunctioning endpoint devices is impossible. We probably have over 200 network cells on our campus alone. We also have off campus stacks.

Have a port die? User setup is a pc > ip phone > drop > pp > switchport? Old fix: Move them to a new switchport, enable it, get them back online, confirm, and then go back to the cell to troubleshoot the problem port. Not anymore!

Sorry Mr. CFO, all the unused ports are disabled.. You say the problem started yesterday before you went home? OK, What day is it? Tuesday the 14th? Sorry, earliest you will have access to your desk phone and a functional PC is Monday the 27th, 12 days from now. You didn't really need any of that to do your job right?

This feels insane. Am I missing something? Anyone seen anything similar, or have good advice on how to approach a situation like this? Let it burn?



No comments:

Post a Comment