Saturday, May 15, 2021

Slow network

Hi!

We have a remote site that has been complaining about the network being very slow from time to time. They have a 1Gbps DIA and a 40/40Mbps MPLS coming to our DC. We use Meraki SD-WAN to connect to them. Internet traffic is forced to the 1Gbps circuit and VPN traffic is using the MPLS.

The main issue is that sometimes it takes forever to run a Cognos report. It often times out as well. Cognos is located in the same server subnet as DNS, DHCP, AD, file server, etc. In our DC.

We have over 100 sites designed the same way and no one is complaining. They can run reports no problems.

By troubleshooting with a user over there, I found out that when the slowness is present, it also takes forever to load a new Web page. I ran some speed tests to the internet and got over 800Mbps. They use our internal DNS so I presume the DNS queries times out or something. I tried to force the VPN traffic to the 1Gbps for a few days, same problem. I ran some pings and trace routes to several of our internal servers and didn't find anything strange there either. Everything looks good in our Meraki dashboard and in our monitoring system. They are no where near saturating the MPLS or internet circuit. Same thing in our DC, no congestion or peaks whatsoever.

One thing I noticed though is that if I turn on their client VPN (Anyconnect) slowness disappears. Reports are fast, surfing the web is also pretty fast and the second I turn it off, everything slows down. Client VPN traffic arrives to our DC firewalls through the internet. Non-Client VPN branch traffic arrives on our Meraki VPN concentrator connected to our Core through the internet or MPLS depending on what we decide in the branch Meraki's.

So if slowness disappear when the client VPN is on it means there is a routing issue somewhere but all our other branches are OK..

I've ran some packet captures when I was troubleshooting. I saw a lot of DUP TCP packets from the Cognos server when we were running the report. On VPN I can't see them because everything is encrypted.

The problem is intermittent. Everything could run smoothly for a few days.

Any ideas on how to tackle this? Should I use a pc over there and install some tools on it to gather some info's?

We also have SolarWinds NPM at our disposal. The branch is on the other side of the country so I can't go there and there is no tech to assist me.

Thank you all!



No comments:

Post a Comment