Tuesday, January 26, 2021

My boss can't afford a professional and I'm the most tech-savvy employee she has, though only by default. I have a month to fix her outages or else we have to use cellular hotspots for the biggest money-making event this quarter. Our ISP has officially given up on us. Help with a speedrun?

I'm not exactly a genius, everything I know about networking comes from Linus Tech Tips videos. That said, here's the situation as of when I was set to the task:

  • At peak, we have 10-20 employees and several dozen customers using the internet at once.
  • Internet (Spectrum) comes in via a coax cable that gets split four ways. Three go to different modems, one goes to who knows where. Or at least I assume it works like that, because I need to at least pretend I understand something about all this.
  • Of those three modems, one is an Arris 1602 and the others are Arris 1670 modem/router combo units. I have no idea if those are good enough as modems for our network requirements, but the router on the 1670 we use (the 2.5gz network is reserved for guests) just dies during peak, and the connection drops out- not slow internet, the network just drops your device.
  • The other 1670 only really exists for employees can try to connect to it when the other networks fail. It doesn't do a great job of that. It does connect to a RasPi TV showing ads and a mysterious, unmarked grey box that nobody knows the purpose of. It also connects to a PoE wall plug injector, which only has one plug for power out. Why is this going into an already-powered router? I don't know, but wish I did.
  • The 1602 wasn't actually being used, but it was plugged in, connected to an inline PoE injector which itself powered a Ruckus r500 (the only router that looks like it can handle this many customers, which naturally was only used by half of the office staff). For some reason, it didn't actually provide data, so instead of fixing this the Spectrum guy apparently decided to just plug it into a 1670 while leaving the other cable uselessly connected.
  • There's a Cisco 24-port switch of some kind that says "10g/100g" as well as "PoE" on it, so I assume it's good for something, but it's not being used for any of the actual networking. I think it's used for the phone lines, but I don't know for sure because this whole building is an absolute mess of cables. This is a retrofitted warehouse, so no actual ceilings, just steel girders with wires hanging off of them. None of the cables are marked in any way.
  • The front office are all equipped with laptops, but can't use wifi due to aforementioned issues. Instead, they run on a massive tangle of wires, one of which is a cheap 5-port switch daisychained off of another cheap 5-port switch all the way back at the main hub/rat's nest.
  • None of this mess is in a server room or anything, it's just mounted on a wall by the door. And by mounted, I mean some of it is screwed into the drywall and some is just dangling by it's own cables. The cables are strapped to power lines whenever they're going the same direction.
  • The public router still uses the default admin password.
  • Not really network related, but there are UPS systems everywhere, and none of it is actually applied to anything. Most of the equipment is tied to one with a dead battery and nonfunctioning battery plugs, so it's just a really heavy power strip. Another is only connected to a printer- the battery isn't even inside the device for this one, it's just sitting to the side with bare wires exposed. Another goes into a room where the only powered devices are laptops. I think (hope) one of them powers the security cameras, but it sure doesn't power the alarms.

The actual issue is that, aside from the public network not working under load, the main issue is that sometimes the networks just stop working. It's either "Connected, no internet", or trying to connect gives you an infinite loading screen. It happens with no warning and lasts anywhere from a few seconds to the better part of an hour. On some of the older devices we have, a disconnect breaks the internal antenna so that it won't pick up signals at all until it's rebooted. Obviously, that's not great when you have business-critical stuff that depends on the internet. Spectrum insists they don't see any outages on their end, so either they're wrong or the mess here is to blame. You can guess which I'm leaning towards.

What my instincts are telling me to do is rip everything apart, then just go modem ==> switch ==> everything else, but this is a business environment and IDK if that's a good idea or would even solve my problem. I'm coming in early tomorrow to try and get more info when no customers are there to be annoyed by falling dust and cat 5e cables. If there's anything I should try to find out let me know.

Edit: we probably need some kind of monitoring system or... something, but I wouldn't even know where to start.



No comments:

Post a Comment