Tuesday, June 1, 2021

Cisco SG250 "Drop Events" on uplink port

Greetings

Sorry in advance if this isn't "enterprisey" enough but it's what we have in this office. If I should ask this somewhere else, let me know and I'll do that.

I have a Fortigate 60F (6.0.12) feeding into a Cisco SG250-50 (2.5.0.83), both of which are brand-new in the last 90 days. The Fortigate is plugged directly into an ISP-provided Hitron cable modem, attached to a 1G/50M service.

We are seeing inconsistence in our service. Speed tests can range anywhere from 50/5 to 950+/55. Days can go by without performance issues, and then we'll have days where the voip will get choppy/drop-happy/one-sided etc, and/or teams video will be laggy and choppy and freezy.

Speedtest issues do not correlate to the other issues.

Vendor support has been, so far, hopeless.

The only outlier that I can put my finger on, and that only inconsistently, is that when the speedtest results are bad, the switchport connected to the firewall LAN interface will sometimes accumulate RX Discards (called Drop Events in the web GUI) during the speedtest. As in, 10K to 15K packets per test run.

These results happen only when the speedtest is run through a web browser, connected to either speedtest.net or "whatever it is google uses when you search for speed test". If I use the win10 Speedtest.net App, it does not accumulate drop events, bad result or good.

Also, speedtest results are much less likely to be bad when run through the app.

Also, drop events do very slowly accumulate during non-speedtest use, but only in the order of a couple dozen per day.

My research suggests that drop events (InDiscards) indicate that the switch received a packet and did not forward the packet on -- due to ACLs (none active), QoS (default setting but the port shouldn't be triggering it) or a lack of resources on the switch -- ie buffer space. I can't rule out the last one, becuse I have been unable to find a guide to debug-mode on this switch (if debug-mode on the switch would even help diagnose something like this).

If I move the uplink port, the discards follow the move. So there's something about the way that this firewall talks to this switch.

Except, as I mentioned, both these devices are new in the last 90 days. The previous firewall, a FortiWifi 60D (6.0.8) was connected to an HP 1810G-24, and we saw the same kind of performance issues. I can't tell you if the "discard" symptom was showing because those HP switches don't export crap through SNMP. The intermediate combination, the FortiWiFi 60D connected to this Cisco SG250, also exhibited the same performance problems.

Every cable I can lay my hands on has been replaced. Both the switch and the firewall have been replaced. I can't see any evidence of ip conflicts or mac stealing. The only pre-existing "neworking" equipment still here are a pair of Aruba Instant Network things, and to simplify things I've turned them off while we work on fixing the wired network issues. And still.

Further upstream, the ISP has been in and replaced an open splitter on the input cable with a straight coupler. When they (or we) plug directly into the ISP device, the performance is always good.

I'm losing my mind here. At this point I'd welcome someone rolling up and saying the equivalent of "You idiot, have you set the [obvious parameter] from [broken] to [working]?" because it would just get this issue off my back.

What should I look at next?

Guidance gratefully appreciated. Thank you.



No comments:

Post a Comment