Was doing some Saturday night maintenance last night (yay... fun.)
Part of the maintenance involved moving our Internet circuit to a different router. We unplugged from old router, and moved the circuit to the new.
BGP came up right away, and we learned the default route from our provider, and a 'show ip bgp neighbor advertised routes' shows we're sending the correct prefix.
But Internet access was hard down. (Yes we're a smaller company without pi ip space, without our own asn. Yes I've been complaining and saying we needs isp redundancy but told they won't pay for it.)
Anyway, traceroute to Google goes all the way to the new router and dies there.
We go over the configs hard looking for anything we got wrong... but everything looks right.
Then we start looking at netflow and see our traffic is very clearly egressing, but no return traffic is coming back from the ISP. Doing 'show interface human-readable' confirms this as input packets is practically nothing pretty much just the bgp keep alives.
It's like they black holed us. And on the bgp looking glass of course you can only see the super net advertised by our isp, so there's no way to see if they're getting our smaller advertisement.
At this point we call their noc but it's painful talking to their Saturday night crew as they can't understand what the problem is "you're not accepting our route, you're not sending return traffic" is completely alien language to them, and they just bounce us back and forth between two desks.
While waiting to actually get someone everything just starts working.
We had bgp debugging turned on, and nothing happened when it started working. Not a single syslog threw on our router. But now return traffic is coming in and everything works.
Our isp closed the ticket with no info and their noc manager said "you can call back Monday but I'm just saying, you're not going to get an RFO because we never saw anything wrong, and once I get off the phone with you, we aren't going to look at this any further."
What do you guys think happened to us?
No comments:
Post a Comment