Situation: Lots of ISP finger-pointing between me (MSP) and Comcast regarding a customer's internet connection. They are on coax (I know, I know, upgrading to fiber scheduled in about 90 days) at 150/20. There are no other ISP options in the area except AT&T DSL at 3/1. No thanks.
About every 12 hours or so, but not exactly every 12 hours, the site internet connection will drop. Monitoring data from the firewalls show that the Comcast modem/all-in-one thing stops responding to pings on its gateway address. Firewall has public IP on its WAN interface so no double-NAT going on.
The Comcast modem's US/DS and ONLINE lights would go out and begin flashing in the past, indicating an upstream problem. Power cycling the modem would fix the problem for another day or so. Now, it's doing something different wherein the US/DS lights stay lit suggesting that it's maintaining upstream connection but it won't pass any data. Nor can the firewall ping it on its gateway IP address.
There are a couple switches between the firewall and the Comcast gateway due to stupid demarc decisions that are beyond my control. The switches are connect the firewall to the gateway (on a dedicated VLAN) by a combination of 10GBASE-LR SMF and 10GBASE-SR. Other traffic across those same links on other VLANs is working fine with no reported or observed issues.
We insisted on a modem replacement which Comcast took a month (yes a literal month) to arrange. They finally did so yesterday and, after they did so, we had smooth sailing for nearly 24 glorious hours. Then this morning, the connection died again. Rebooting the gateway did not fix the issue this time. We had to reboot it, wait for it to come all the way up, and THEN unplug the CAT6 cable from the switch and plug it into a different port. Only then did the gateway begin passing traffic. We've done this with 3 different ports on 2 different switches so I really doubt the switches are at fault.
At this point my customer is beyond frustrated. Comcast is insisting that their new modem is fine and there's no problem, even though the firewall reports wild swings in response time and packet loss pinging the gateway at random times during the day and then hard-down outages about once a day or so.
What I'd like for this problem, and others, is a device that I can plug in at certain points of a customer's site to be remotely accessible and to pull diagnostic data from. I'm thinking a NUC or something like that that can run network monitoring software (Like EMCO Ping Monitor maybe) and possibly has cellular capability so I can see what's happening during an outage. Does anybody have ideas for a device and what software you'd load on it for this kind of thing?
No comments:
Post a Comment