Wednesday, November 6, 2019

Null0 route caused big outage?

Trying to understand what went wrong. We were trying to advertise a summarized route to our wan. The site has several subnets in the 10.10.0.0 space. For ex. 10.10.0.0/24, 10.10.1.0/24, 10.10.2.0/24, 10.10.3.0/24.

No other location had any subnet in the 10.10.0.0/16 range.

For this reason I said that we could eliminate all these advertisements and replace it with one 10.10.0.0/16 summary route.

So we put in the network statement in bgp for the /16 route, but it wouldn’t advertise. That’s when I remembered that you need a matching route in your rib or it won’t advertise.

So I entered this command at the wan edge router.

ip route 10.10.0.0 255.255.0.0 null0 

As soon as I hit enter, I immediately lost connection to the router, and our LAN went down. Phone also started blowing up that remote sites were all down.

I could not ping a neighboring local subnet. It was even on the same switch as me with intervlan routing. I even did show ip route to said subnet and it showed directly connected /24 subnet, so that /16 wasn’t the best path.

We rushed to the wan Router and I consoled in with locals, tacacs wasn’t reachable even though that was all on a separate vrf.

I no’ed out the null0 route, and that fixed everything.

I am just flabbergasted on what could have happened. This was my first week on the job too and I’m worried big time. The senior engineer let me do it without change management because he agreed it shouldn’t cause any impact. He was just as surprised as me. Help!



No comments:

Post a Comment