Saturday, July 21, 2018

Next Steps for a Tier 1 tech to Troubleshoot a FUBAR network?

This situation is a bit over my head. The short and sweet of it is that I'm a Help Desk tech with 5 months of experience in IT and due to recent departures of our 2 most experienced techs, we're lacking people with the skills necessary to troubleshoot a complicated network setup.

 

The situation is complicated by a hybrid network management setup where the ISP handles some of the internal network while we manage some of the other parts of it such as the WiFi and Sonicwalls. Also there is no clear documentation for either side on what gets plugged into where.

 

If I had a wishlist it would probably be credentials to access all the managed switches on premises and use a tool like Auvik to at least get a logical map of the network for me. Although I'm not sure if every managed switch on premises makes use of SNMP, at the very least I doubt the $30-$40 TP-Link switches are capable of that.

 

The main complaints from the client is that 2 buildings have Point of Sale systems that will sporadically become disconnected from LAN and will need to be switched over to WiFi (iPads really). The network is spread between 3 buildings. Buildings 1 & 2 are on the same local network under Sonicwall A, Building 3 is under Sonicwall B connected using Point to Point VPN between the Sonicwalls, so basically using the internet to link the two networks(different subnets) under the same domain.

 

Building 3 might have some network issues but generally not considered huge since PoS systems are only in buildings 1 and 2. I've been using PingPlotter on a free trial basis to try to narrow down the issue. Thus far I would say Building 3 has the least issues as far as latency spikes are concerned. Although Building 3 had issues where web browsing was impossible until DNS was pointed to Google DNS for the workstations. The DC/DNS server is located in Building 1, suggesting that subnet B under Sonicwall B was having issues communicating with subnet A under Sonicwall A which is where the DC/DNS is.

 

I've also noted that when the POC calls to note that they're having network issues it generally takes quite a while to remotely connect to their servers.

 

Because of poor documentation I'm having issues differentiating between Building 1 resources and Building 2, but let's say the subnet of Sonicwall A is overall pretty bad with internal latency of 200-500ms max in any given 3-hour period whereas Building 3 under Sonicwall B subnet is very happy at about 23-40ms max latency at the same time scope.

 

The only notable thing I've seen is that the latency graph for TP-Link switch 1 & 2 mirror each other almost exactly. Switch 1 and 4 are located in the same building within 30 feet of each other, but switch 1 shows high latency whereas switch 4 has almost none.

 

Client POC wants to know what the long-term plan is and I don't have an answer for him with what I have to work with so far. I just figure we'll have to set aside a weekend to compile an actual map of their network and figure out what plugs into where. May have to ask the ISP to have us both there at the same time.

 

Kind of wondering if I overlooked anything in terms of remote data gathering. I've used iperf a few days ago to test internal network bandwidth to see if the router was a chokepoint for another different client site by running it from one computer to another and wanted to see if this might force symptoms to appear if I can narrow down the potential location of the problem, at least logically on the network?



No comments:

Post a Comment