Trying to track down an issue with a Verizon internet circuit that started this past week. Symptoms are intermittent access inbound and outbound, depending on the originating/destination IP. Have already worked with Verizon to isolate the issue beyond our equipment by plugging in a laptop configured with the WAN IP of the CPE into the DMARC and reproducing the issue. Here's where the traceroutes come in. I feel like I almost have a smoking gun here, but the results (to me) seem somewhat inconsistent. I'm using the Cogent looking glass site to perform traceroutes to the WAN IP of the CPE from various locations across the US.
Traceroutes to the upstream router (default gateway of CPE) succeed no matter where they originate from, so I'm guessing there is some sort of routing issue in their network with the WAN IP on our CPE (it's a /30). We have a /27 that we use on the LAN side of the CPE and on our Firewall for NAT, but I don't believe the issue is with it. We have no ACLs in place that would prohibit ICMP to the CPE.
Here's what I have so far:
These locations fail consistently:
US-Miami - To Upstream Router
traceroute to *.*.*.1 (*.*.*.1), 30 hops max, 60 byte packets 1 gi0-0-0-18.221.agr13.mia01.atlas.cogentco.com (66.28.3.217) 1.131 ms 1.143 ms 2 te0-7-0-1.ccr21.mia01.atlas.cogentco.com (154.54.6.57) 0.628 ms 0.684 ms 3 be3401.ccr21.mia03.atlas.cogentco.com (154.54.47.30) 0.587 ms 0.624 ms 4 0.ae13.BR1.MIA19.ALTER.NET (204.255.168.29) 0.497 ms 9.549 ms 5 0.xe-10-0-0.XL1.CMH2.ALTER.NET (152.63.65.38) 43.065 ms 0.xe-10-0-0.XL2.CMH2.ALTER.NET (152.63.65.42) 39.050 ms 6 29.xe-4-2-0.GW4.CMH2.ALTER.NET (*.*.*.1) 39.530 ms 43.200 ms
US-Miami - To CPE WAN interface
traceroute to *.*.*.2 (*.*.*.2), 30 hops max, 60 byte packets 1 gi0-0-0-18.221.agr13.mia01.atlas.cogentco.com (66.28.3.217) 0.741 ms 0.743 ms 2 te0-7-0-1.ccr21.mia01.atlas.cogentco.com (154.54.6.57) 0.743 ms 0.745 ms 3 be3400.ccr21.mia03.atlas.cogentco.com (154.54.47.18) 0.688 ms be3401.ccr21.mia03.atlas.cogentco.com (154.54.47.30) 0.735 ms 4 0.ae13.BR1.MIA19.ALTER.NET (204.255.168.29) 0.366 ms 0.441 ms 5 0.xe-10-0-0.XL2.CMH2.ALTER.NET (152.63.65.42) 81.959 ms 0.xe-10-0-0.XL1.CMH2.ALTER.NET (152.63.65.38) 43.093 ms 6 0.xe-10-0-0.GW4.CMH2.ALTER.NET (152.63.64.74) 43.186 ms 0.xe-11-0-0.GW4.CMH2.ALTER.NET (152.63.64.78) 39.216 ms 7 * *
US-Philadelphia - To Upstream Router
traceroute to *.*.*.1 (*.*.*.1), 30 hops max, 60 byte packets 1 gi0-5-0-10.3.rcr21.phl01.atlas.cogentco.com (66.250.250.105) 0.735 ms 0.744 ms 2 be2364.ccr41.jfk02.atlas.cogentco.com (154.54.3.141) 3.122 ms 3.153 ms 3 be2056.ccr21.jfk10.atlas.cogentco.com (154.54.44.218) 3.493 ms 3.455 ms 4 0.ae17.BR2.NYC4.ALTER.NET (204.255.168.113) 3.126 ms 3.176 ms 5 0.xe-10-0-0.XL2.CMH2.ALTER.NET (152.63.65.42) 24.957 ms 24.973 ms 6 29.xe-4-2-0.GW4.CMH2.ALTER.NET (*.*.*.1) 25.979 ms 26.073 ms
US-Philadelphia - To CPE WAN interface
traceroute to *.*.*.2 (*.*.*.2), 30 hops max, 60 byte packets 1 gi0-5-0-10.3.rcr21.phl01.atlas.cogentco.com (66.250.250.105) 0.777 ms 0.781 ms 2 be2364.ccr41.jfk02.atlas.cogentco.com (154.54.3.141) 3.171 ms 3.181 ms 3 be2056.ccr21.jfk10.atlas.cogentco.com (154.54.44.218) 3.468 ms be3294.ccr31.jfk05.atlas.cogentco.com (154.54.47.218) 3.303 ms 4 0.ae17.BR2.NYC4.ALTER.NET (204.255.168.113) 3.299 ms 3.273 ms 5 0.xe-3-2-0.XL1.CMH2.ALTER.NET (152.63.65.45) 25.120 ms 0.xe-10-0-0.XL2.CMH2.ALTER.NET (152.63.65.42) 24.924 ms 6 0.xe-10-0-0.GW4.CMH2.ALTER.NET (152.63.64.74) 25.261 ms 0.xe-11-0-0.GW4.CMH2.ALTER.NET (152.63.64.78) 25.700 ms 7 * *
US-Chicago - To Upstream Router
traceroute to *.*.*.1 (*.*.*.1), 30 hops max, 60 byte packets 1 gi0-0-0-15.99.agr21.ord01.atlas.cogentco.com (66.250.250.89) 0.683 ms 2.254 ms 2 be2522.ccr42.ord01.atlas.cogentco.com (154.54.81.61) 0.804 ms be2521.ccr41.ord01.atlas.cogentco.com (154.54.80.253) 0.638 ms 3 be2765.ccr41.ord03.atlas.cogentco.com (154.54.45.18) 0.833 ms be2766.ccr41.ord03.atlas.cogentco.com (154.54.46.178) 0.799 ms 4 0.xe-2-3-0.BR3.CHI13.ALTER.NET (204.255.168.61) 0.569 ms 0.533 ms 5 0.xe-10-0-0.XL1.CMH2.ALTER.NET (152.63.65.38) 12.365 ms 0.xe-3-2-0.XL2.CMH2.ALTER.NET (152.63.65.46) 12.272 ms 6 29.xe-4-2-0.GW4.CMH2.ALTER.NET (*.*.*.1) 13.271 ms 13.323 ms
US-Chicago - To CPE WAN interface
traceroute to *.*.*.2 (*.*.*.2), 30 hops max, 60 byte packets 1 gi0-0-0-15.99.agr21.ord01.atlas.cogentco.com (66.250.250.89) 0.879 ms 0.890 ms 2 be2522.ccr42.ord01.atlas.cogentco.com (154.54.81.61) 0.718 ms be2521.ccr41.ord01.atlas.cogentco.com (154.54.80.253) 0.724 ms 3 be2766.ccr41.ord03.atlas.cogentco.com (154.54.46.178) 0.840 ms be2765.ccr41.ord03.atlas.cogentco.com (154.54.45.18) 0.694 ms 4 0.xe-2-3-0.BR3.CHI13.ALTER.NET (204.255.168.61) 0.569 ms 0.543 ms 5 0.xe-3-2-0.XL2.CMH2.ALTER.NET (152.63.65.46) 12.229 ms 0.xe-10-0-0.XL1.CMH2.ALTER.NET (152.63.65.38) 12.405 ms 6 0.xe-11-0-0.GW4.CMH2.ALTER.NET (152.63.64.78) 12.543 ms 12.521 ms 7 * *
These locations succeed consistently:
US-Boston - To Upstream Router
traceroute to *.*.*.1 (*.*.*.1), 30 hops max, 60 byte packets 1 gi0-0-0-15.216.agr22.bos01.atlas.cogentco.com (66.250.250.25) 0.722 ms 0.726 ms 2 te0-7-0-1-0.ccr32.bos01.atlas.cogentco.com (154.54.80.33) 0.668 ms 0.702 ms 3 be3472.ccr42.jfk02.atlas.cogentco.com (154.54.46.34) 6.353 ms 6.007 ms 4 be3295.ccr31.jfk05.atlas.cogentco.com (154.54.80.2) 6.250 ms be2056.ccr21.jfk10.atlas.cogentco.com (154.54.44.218) 6.355 ms 5 0.ae13.BR3.NYC4.ALTER.NET (204.255.168.121) 6.103 ms 6.213 ms 6 0.xe-10-0-0.XL2.CMH2.ALTER.NET (152.63.65.42) 27.056 ms 27.032 ms 7 29.xe-4-2-0.GW4.CMH2.ALTER.NET (*.*.*.1) 26.854 ms 27.004 ms
US-Boston - To CPE WAN interface
traceroute to *.*.*.2 (*.*.*.2), 30 hops max, 60 byte packets 1 gi0-0-0-15.216.agr22.bos01.atlas.cogentco.com (66.250.250.25) 0.634 ms 0.690 ms 2 te0-7-0-1-0.ccr31.bos01.atlas.cogentco.com (154.54.80.9) 0.458 ms te0-7-0-1-0.ccr32.bos01.atlas.cogentco.com (154.54.80.33) 0.449 ms 3 be3471.ccr41.jfk02.atlas.cogentco.com (154.54.40.154) 6.357 ms be3472.ccr42.jfk02.atlas.cogentco.com (154.54.46.34) 6.291 ms 4 be3294.ccr31.jfk05.atlas.cogentco.com (154.54.47.218) 6.387 ms be2057.ccr21.jfk10.atlas.cogentco.com (154.54.80.178) 6.398 ms 5 0.ae17.BR2.NYC4.ALTER.NET (204.255.168.113) 6.102 ms 6.119 ms 6 0.xe-10-0-0.XL2.CMH2.ALTER.NET (152.63.65.42) 27.078 ms 27.092 ms 7 0.xe-11-0-0.GW4.CMH2.ALTER.NET (152.63.64.78) 26.670 ms 26.681 ms 8 gw.customer.alter.net (*.*.*.2) 42.464 ms 42.463 ms
US-Minneapolis - To Upstream Router
traceroute to *.*.*.1 (*.*.*.1), 30 hops max, 60 byte packets 1 gi0-5-0-12.22.rcr21.msp01.atlas.cogentco.com (66.250.250.177) 0.762 ms 0.804 ms 2 be2411.ccr41.ord01.atlas.cogentco.com (154.54.24.34) 11.005 ms 11.171 ms 3 be2765.ccr41.ord03.atlas.cogentco.com (154.54.45.18) 13.657 ms be2766.ccr41.ord03.atlas.cogentco.com (154.54.46.178) 13.514 ms 4 0.xe-2-3-0.BR3.CHI13.ALTER.NET (204.255.168.61) 15.970 ms 13.460 ms 5 0.xe-3-2-0.XL2.CMH2.ALTER.NET (152.63.65.46) 25.119 ms 27.539 ms 6 29.xe-4-2-0.GW4.CMH2.ALTER.NET (*.*.*.1) 25.438 ms 27.852 ms
US-Minneapolis - To CPE WAN interface
traceroute to *.*.*.2 (*.*.*.2), 30 hops max, 60 byte packets 1 gi0-5-0-12.22.rcr21.msp01.atlas.cogentco.com (66.250.250.177) 0.790 ms 0.792 ms 2 be2411.ccr41.ord01.atlas.cogentco.com (154.54.24.34) 10.982 ms be2410.ccr42.ord01.atlas.cogentco.com (154.54.7.229) 10.974 ms 3 be2766.ccr41.ord03.atlas.cogentco.com (154.54.46.178) 13.478 ms be2765.ccr41.ord03.atlas.cogentco.com (154.54.45.18) 11.300 ms 4 0.xe-2-3-0.BR3.CHI13.ALTER.NET (204.255.168.61) 13.462 ms 15.920 ms 5 0.xe-3-2-0.XL2.CMH2.ALTER.NET (152.63.65.46) 27.569 ms 0.xe-10-0-0.XL1.CMH2.ALTER.NET (152.63.65.38) 25.042 ms 6 0.xe-11-0-0.GW4.CMH2.ALTER.NET (152.63.64.78) 25.001 ms 22.631 ms 7 gw.customer.alter.net (*.*.*.2) 40.651 ms 40.669 ms
There are other examples of successes/failures, but I'm trying to keep this as brief as possible. The traffic always seems to make it to GW4.CMH2.ALTER.NET, but depending on where the traffic is originating from, it doesn't make it to (or back from?) the next hop (CPE).
This ticket has gone basically nowhere for about 4 days. If anyone has any advice on where I should direct the repair engineer's attention, it'd be greatly appreciated. When doing outbound traceroutes to unreachable systems it never makes it past our CPE. When I had the laptop plugged in to the DMARC for testing (in place of the CPE), it didn't register ANY hops when tracing to unreachable systems (I was tracing out to 8.8.8.8, 8.8.4.4, 4.2.2.1, 4.2.2.2 etc to keep things simple). When I had the laptop plugged in to the CPE in place of the firewall for testing (configured with an IP from our /27) I would get different results as I cycled through our IPs (for example, if I had it configured with .2 I may not be able to reach 4.2.2.1, but if I changed the laptop to .3, I could reach 4.2.2.1 but then other IPs would still be unreachable).
I believe at one point while I was on the phone with them, the technician stated that he was unable to ping our CPE from their gateway unless he changed the source IP for the ping to the IP that we have set as the default gateway on the CPE (i.e. pinging from ...1 to *...2 instead of whatever the default source IP of their gateway is to *..*.2). He didn't seem to think this was a big deal, although it did seem odd to me.
TIA
No comments:
Post a Comment