Xfinity SecureEdge for Business transparently intercepts Port 53 DNS and breaks DNSSEC

About 3 weeks ago I did a speed upgrade on my Xfinity Business line and they threw in SecurityEdge for Business as part of the package. I have no idea what it is or what it does... There's no place to configure it and it's not real clear what it is based on any literature I could find.

On Mar 17th I started noticing a huge number of outbound DNS queries taking a LONG time to return, I started doing some digging. When I switched some systems over to DNS over TLS and redirecting to 1.1.1.1 / 1.0.0.1 the issue would go away.

I started doing more testing from one of our web servers that hosts a recursive DNS server for looking up RBLs and such (since they rate limit using public name servers usually results in some sort of a block).

What I found was queries to root name servers were returning IP addresses.... That's... not possible.

[root@web ~]# dig google.com @198.41.0.4 +trace ; <<>> DiG 9.11.4-P2-RedHat-9.11.4-9.P2.el7 <<>> google.com @198.41.0.4 +trace ;; global options: +cmd . 600 IN NS i.root-servers.net. . 600 IN NS j.root-servers.net. . 600 IN NS k.root-servers.net. . 600 IN NS l.root-servers.net. . 600 IN NS m.root-servers.net. . 600 IN NS b.root-servers.net. . 600 IN NS c.root-servers.net. . 600 IN NS d.root-servers.net. . 600 IN NS e.root-servers.net. . 600 IN NS f.root-servers.net. . 600 IN NS g.root-servers.net. . 600 IN NS h.root-servers.net. . 600 IN NS a.root-servers.net. . 600 IN RRSIG NS 8 0 518400 20200331050000 20200318040000 33853 . v/tMTMhCpk16kk2iM6ckfFftGalf7yKrrgmOHZkWPIUA97vfkR0YqRtPOzXe8wy9GxR7OXMUKweqfHpgmK/tduGh3a8qdaZ69rFI+bhARgg8r+2TnsLDMgGaJL1s3VvjF10l4pKJ7NILeXz1BtoowxHh9u4ug2Z5SWVGp+NLdXpVjWNFtk3HJlyFYftFoeFJpN+W7yisfNQ3M/zj5Mn/qjFz00dh+1B2aFicUiOErlSV3LuHvKi5dMji1pCnDSkB/nMnRcOXC844G2WWt401p8eSBJ3Ycz3HO+f881PJbxo0QJQ/CH91z09yUPn/LShvZz1NIWt+XAYfaOPz v6ksKA== ;; Received 1086 bytes from 198.41.0.4#53(198.41.0.4) in 37 ms google.com. 93 IN A 172.217.10.142 ;; Received 55 bytes from 192.203.230.10#53(e.root-servers.net) in 4 ms 

Started doing more digging and doing timing tests. I was seeing traceroutes to the root name servers in the realm of 20-40ms but I was getting DNS query responses back anywhere from 3 seconds to never. However, when I would get one quickly, it was WAY too quick, in the < 10ms range. This told me it had to be the modem.

Called Comcast Business and started talking it through with them, we went through the usual steps and on a hunch I asked them to remove SecurityEdge. After the change was made and it was removed, I started getting full proper traces..

[root@web ~]# dig google.com @198.41.0.4 +trace ; <<>> DiG 9.11.4-P2-RedHat-9.11.4-9.P2.el7 <<>> google.com @198.41.0.4 +trace ;; global options: +cmd . 518400 IN NS e.root-servers.net. . 518400 IN NS h.root-servers.net. . 518400 IN NS l.root-servers.net. . 518400 IN NS i.root-servers.net. . 518400 IN NS a.root-servers.net. . 518400 IN NS d.root-servers.net. . 518400 IN NS c.root-servers.net. . 518400 IN NS b.root-servers.net. . 518400 IN NS j.root-servers.net. . 518400 IN NS k.root-servers.net. . 518400 IN NS g.root-servers.net. . 518400 IN NS m.root-servers.net. . 518400 IN NS f.root-servers.net. . 518400 IN RRSIG NS 8 0 518400 20200331170000 20200318160000 33853 . qgasYmvTaMw/ft2FJz7Ze3a8EYdfzDR3E/n9ffoT8zkgJZhW74Yf1Tdnyt7zJUoZjZSL0px3bOccsey7rwAAt7PG3PKsG50hINxFU/G65DdLn5Fe0E3wqLh7J2oix+own3AHEUyntF3nuL/surpqvvZpLoS+DU4enbMfJlZfKSu2/73I+n6tx57gGWnekkFlgq7JVBS6MDry5UsFR4C3GwBInUqcFiQQATVi6s9+xcWmTWhUOLtZa9JyStBDWanch24001hD51VLFix7DOnA1+oG9IcdQjqO4WTbzk2TgfRGNvax6IPeVWwLOTaDfpH/1UjfqI6OVNldnXSE xBsI6g== ;; Received 1097 bytes from 198.41.0.4#53(198.41.0.4) in 17 ms com. 172800 IN NS a.gtld-servers.net. com. 172800 IN NS b.gtld-servers.net. com. 172800 IN NS c.gtld-servers.net. com. 172800 IN NS d.gtld-servers.net. com. 172800 IN NS e.gtld-servers.net. com. 172800 IN NS f.gtld-servers.net. com. 172800 IN NS g.gtld-servers.net. com. 172800 IN NS h.gtld-servers.net. com. 172800 IN NS i.gtld-servers.net. com. 172800 IN NS j.gtld-servers.net. com. 172800 IN NS k.gtld-servers.net. com. 172800 IN NS l.gtld-servers.net. com. 172800 IN NS m.gtld-servers.net. com. 86400 IN DS 30909 8 2 E2D3C916F6DEEAC73294E8268FB5885044A833FC5459588F4A9184CF C41A5766 com. 86400 IN RRSIG DS 8 1 86400 20200331170000 20200318160000 33853 . IqOJ6nE+fKiwc8jNJy+qBpMo2fMSJSYGRbfNO6sz4VejsuoYGDuEdrb4 g/bcwebIXaCWIn/d3pOQaf7f0jweWvykYr4uyKj6Q1fu+ppvzLHyvLxw +OmqOStuZXXgw/kiMEyEFaRGuFShZd74clSc/LJnOjtRXZ3vIb1LSXZZ cTT9nBKIgCe/yS/cbZwWLdkoK4q0vqEJgcdIhdrUsghfti+EVAieq/W/ lYuafNiOdh474NuPdJLM1FRdYey49TLVdyUoZ8n3M+JmRygPLEqH4RAk BFN5Z0DZsWEj7Ny/gAxnxApvM3w1Bog9X4Zl9DvI5DV53Ek4U2b7GCd3 ijCY4Q== ;; Received 1170 bytes from 198.97.190.53#53(h.root-servers.net) in 20 ms google.com. 172800 IN NS ns2.google.com. google.com. 172800 IN NS ns1.google.com. google.com. 172800 IN NS ns3.google.com. google.com. 172800 IN NS ns4.google.com. CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86400 IN NSEC3 1 1 0 - CK0Q1GIN43N1ARRC9OSM6QPQR81H5M9A NS SOA RRSIG DNSKEY NSEC3PARAM CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86400 IN RRSIG NSEC3 8 2 86400 20200322044927 20200315033927 56311 com. pKi3j2T+MmOgxjdmTcZS3YYGSfTSSb0jX5woxUr9roiXvsiM6gxczhHa 43lZFia30VmrYsRNrA43ddnO03iC0bAU0QOfsMSZ0SasKx6fAb+Ynj0H Z/MlenueBOVWr11KlixRNF5hZgLIl+c/+nVM48BkKM6Xfoju4j8+Wedm Nm9phbpnEyd+awJ31vZJAvXDfwOT1SAqqKEq2F06iYoR4g== S84BDVKNH5AGDSI7F5J0O3NPRHU0G7JQ.com. 86400 IN NSEC3 1 1 0 - S84EDELLAUPA96DT12TJKJN32334NGL3 NS DS RRSIG S84BDVKNH5AGDSI7F5J0O3NPRHU0G7JQ.com. 86400 IN RRSIG NSEC3 8 2 86400 20200323044922 20200316033922 56311 com. dFM5sEzSJFsZb1+NeVpo9AeapUhEs/PM/sXDlQO9Mg0wCKLr5HzR3iTK pJ2bUaxuM24osIK/DndpkUQ+TBQF8uXxc9Trrq9kIlzfrylYuRWpOJSy lNXlEkwy51hcGC7i3h5yTDU7ARKQJwquX3BvzTITfbdRbkXCNMichVPg 25PwuWoHZIdsEuiKoWIYCbiUhNeWNhHggvqJ+zxC3+dd6A== ;; Received 836 bytes from 192.43.172.30#53(i.gtld-servers.net) in 18 ms google.com. 300 IN A 172.217.13.238 ;; Received 55 bytes from 216.239.36.10#53(ns3.google.com) in 25 ms