Note to the reader: A random engineer contacted me after having the exact same problem as me. He saw my ASN as being the only one handled properly by Cox's IRR records and upstream filtering, yet Cox was telling him that they absolutely do not, can not, and would not add his prefixes to their IRR records in identical fashion. My hope is that anybody having my issue may find this story before either giving up or going down the rabbit hole as far as him and I did.
Late last year I was working on fixing up the Internet Routing Registry (IRR) records for the BGP ASN I operate (AS27580). I was trying to clean up some old records in various IRR databases (RADB, Level3, etc) left behind by the previous owner of 27580, and was getting my IRRExplorer (http://irrexplorer.nlnog.net) table to show up all green (the way us engineers like it).
I peer with Cox for upstream connectivity and I noticed Cox had IRR route objects in the Level3 database for my prefixes; objects which Cox had created themselves and which notated an origin of AS22773 (Cox's AS). These are incorrect records (incorrect origin AS) which is why they were flagged by IRRExplorer, but the creation of them is Cox's standard way of provisioning a new BGP customer. By Cox adding route objects with origin: AS22733 it builds a RSPL such that Level3’s filtergen allows all prefixes matching origin: AS22773 to be announced to AS3356 from AS22733, regardless of what the actual correct origin AS might be. This can be seen by running whois -h filtergen.level3.net AS22773. I promptly contacted my Cox rep and opened a ticket to have Cox remove their route records in the Level3 database.
November 2019
I spoke with a Cox engineer and informed them I managed my own IRR records in the ARIN database and that Level3/Centurylink (L3CL) should be able to use those to properly filter at their peerings with Cox (which is technically incorrect, since Cox would need to list my ASN in their own records, but I didn't know this at the time). It was their standard procedure to add route records to the Level3 database and the Cox engineer told me my routing may break if they removed it.
I had him remove the route record for one of my unused test prefixes so we could check the result. Sure enough, L3CL stopped accepting my test prefix once the Cox record was removed. I had the engineer leave it that way so I could try to figure out if my own IRR records were incorrect. I did a little bit of research and adjustments to no avail, got busy with other things, and forgot about it for 'a while'.
July 2020
A few small world events cropped up (heh), business slowed down a bit, and I finally had some time on my hands to solve this ever-looming issue. Due mostly to the fact I had forgotten almost everything about this problem, I approached it a different way.
My AS only peers with Hurricane Electric (HE) for IPv6 (tunnel broker service) and I noticed HE was not learning my IPv4 advertisements through their Cox peering at all, but instead learned them through L3CL. I contacted HE and asked them why they were not learning my prefixes through their peerings with Cox.
HE told me Cox is not properly adding customer's ASNs to their IRR records and it is resulting in approx 20% of Cox's IPv4 prefixes to be refused by HE along with other upstream providers. HE shot off an email to some backbone contacts at Cox and looped me into the thread. I was able to use this thread, and the pressure coming from HE, to get Cox to agree to add me (AS27580) to their AS-COX-TRANSIT as-set record (a record which hadn't been updated in over a decade).
The AS-COX-TRANSIT as-set is exported in Cox's main AS22773 record, so this should have fixed the issue; and for the HE-to-Cox peering, it did, but not for the L3CL-to-Cox peering.
After some more research and tinkering I was able to get Cox to open a ticket with L3CL, only to find out L3CL was not honoring Cox's referenced AS-COX-TRANSIT object for filtering. L3CL fixed this issue and BANG, my test prefix started showing up in L3CL's looking glasses. I had Cox remove all their route objects in Level3's DB referencing my prefixes and I was finally able to get a clean report from IRRExplorer and all the provider looking glasses.
Conclusion
AFAIK Cox is still following bad procedure when peering with their customers by adding route objects to the Level3 database with Cox's AS (AS22773) as the origin. It seems like they are doing this as a simple workaround to how Level3 is performing filtering on the Cox peerings.
What I believe Cox SHOULD be doing, which is what HE does by using the recursive and hierarchical nature of IRR, is adding their customer's ASNs (or as-set records if the customer has downstream ASNs) to an as-set object (ie: AS-COX-TRANSIT) which is exported in the AS22773 AS object.
In speaking with other people who have had this same problem with Cox, it seems you just need to contact the correct people at Cox to get this done. Normal support channels such as account managers and support engineers seem to dead end.
TLDR: In trying to fix my IRR records, I prompted Hurricane Electric (one of my providers) to escalate my problem to Cox (another one of my providers), then got Cox to escalate to their own upstream Level3/Centuylink (not my provider) and fix Cox's 15-year-old broken and incorrect IRR records; allowing Level3/Centuylink to properly filter Cox's customer (my) prefixes.
No comments:
Post a Comment