Thursday, August 12, 2021

Upload slow - multiple branches - cannot isolate issue

So, here's what i've done since this radioactive ticket graced my lap:

  1. Ticket was sitting backlogged for sometime with mention of circuits installed. Slow upload. So, had the provider test. Nothing. Some sites fine.
  2. Started probing/comparing one of the sites. I started looking at all the configs at the site (router, encryption router, switch). Nothing stood out.
  3. Started trial and error at the shaper and QoS settings - nothing, though looking back, probably had no reason to check QoS as, utilization wasn't being hit. /shrug
  4. Learned that customer at the site has a symantec web filter proxy running for multiple sites. Was awfully curious, but this seemed to turn out as a red herring (tested user without it enabled for the site).
  5. Did iperf. Along with gobs of speed tests with an access port. Fortunately we found that a direct connection to the wall jack showed symmetrical speeds! Yey! So I isolated the issue to the phone.
  6. After trial and error I tested the automatic port synchronization setting from Call Manager due to some weird negotiation issues I noticed. Fixed the issue. Mass deployed the fix to the phones.....
  7. Except issues still persisted at the other sites
  8. Eventually 'fixed' the rest of the sites. Most of the remaining sites I patched up were at the access switch, with some manner of hard coding speed/duplex on ports - the trunks and/or access switches, along with the APS setting in call manager.
  9. Last site remained. NO IDEA. There is nothing that stands out different. Now there are apparently more sites reporting asymmetrical speeds.

Equipment is 29xx for routers, and switches are 37xx. Customer is vrf'ed to our DC.I don't have any good explanation for why the negotiation behavior even was a problem. I also don't think the new sites, or even all the sites had necessarily all the same exact problem, but all the ones i fixed were speed/duplex issues i fixed.

If anyone has experience with this kind of weirdness I'd love to hear it. Or maybe some ideas on where to go next. Thankfully TAC is on board and kind of helping (with one of our routers still in contract). Packet captures on encryption router are next.



No comments:

Post a Comment