Thursday, January 10, 2019

Unable to ping a few specific IPs from my circuit all of a sudden

We have a circuit with Cogent at our datacenter. Starting Sunday around 3:30AM we noticed a VPN to one of our clients on Comcast went down. Troubleshooting, we couldn't ping in either direction. Traceroutes in either direction show it drops off right around the last hop (makes it into the other providers network before dropping off).

Then on Monday we noticed another IP on a completely different Comcast line in another state we couldn't ping. At this point we're thinking either Comcast or Cogent issue. But then a third IP popped up that we wouldn't reach either, on a completely different provider.

We've been troubleshooting with Cogent all week, back and forth. Them saying it's our equipment. So of course our first step was to test bypassing our firewall. Our Cogent fiber connects to a switch with basically no config on it, then goes into our HA Sonicwalls. So I plugged my laptop into the switch, assigned an available IP on our /25 block, and I still couldn't ping either of the three IPs. From our Cogent gateway the IPs can be pinged no problem, but not from any IP in our usable range.

At this point Cogent's blaming Comcast still (even though there's another provider IP we can't reach). But that would make no sense that Comcast would be blocking our IP - when I have an entire /25 block and I can't reach these IPS from ANY IP on our block. For further testing, Cogent assigned a test IP block to our interface. Still can't ping using one of those IPs. The guy put the test block on a different interface, pings fine.

At this point, myself, and Cogent support are completely baffled. They want the next step to be taking our switch/Sonicwall out of the equation, meaning tomorrow I'm going to have to do a maintenance window and connect Cogent up to a media converter then to my laptop and do the tests again, but I'm expecting the same results seeing as how there's nothing wrong with our switch - would make no sense for a dumb switch to block three random IPs all of a sudden. But after that it's going to go back to Cogent, get their NOC guys involved, and figure out wtf is going on.

Has anyone ever seen anything like this??? The entire issue makes absolutely no sense what so ever.



No comments:

Post a Comment