Tuesday, November 17, 2020

Dell N2200 Port Issue / Dead Port / No Data

Hey Reddit,

Just wanted to send out an S.O.S. regarding Dell N2200/3200 series switches. I can easily complain about my woes, but here's the breakdown of some facts(from some guy on the Internet, Me)

The Dell N2200(and N3200, read on please) series switches (not to be confused with N2000) which were released in 2020 have a fatal bug that exists even on the original Firmware version of 6.6.1.0. As of 11/17/2020, with version 6.6.3.3, the issue still exists.

Location of support and downloads --> https://www.dell.com/support/home/en-us/product-support/product/networking-n2200-on/drivers

For some context of the symptom. Occasionally, a random port on the Dell N2200 will simply stop working. With every debugging feature enabled, and Wireshark directly connected to said port in question(again, random), the N2200 series switch simply does not return any network traffic. Regardless of what configuration is on the interface, regardless of how spanning-tree is configured, regardless if it is a single switch with no uplink, the issue is random and simply shows no data. Interface counters stay at 0.

With Wireshark listening in, any PC, IP Phone, or network switch(dumb and Managed with appropriate Vlan config and routing) you'll notice that devices will send its typical DHCP request (even if you set it Statically with or without Vlan tagging), the N2200 just does not reply whatsoever.

If using IP phones and a PoE enabled N2200, you'll see that the port supplies PoE, but no data. Switch will not respond. Interface counters still 0, phone sending data to the port, but nothing coming back (used SharkTap + Wireshark to verify this)

And, yes, a shut and no shut was issued many times and extensive careful consideration of the config, with a plethora of combinations to the config to solve this issue resulted in 0 success(adjusted speed, duplex, even the inline for PoE, removing all config, just making it Access port, ect).

After escalation after escalation, and 9 Dell Engineers later, we received admission and confirmation from Dell that the issue was reproducible internally at Dell, received word that another customer of Dell also has the same issue we do and that this problem exists only on N2200 and N3200 (which we don't have) Series switches. The ETA to supply a fix is scheduled for the first week of December 2020 (fingers crossed).

The only fix currently is by rebooting the switch, and the port magically works for some time. Even logs before the issue occurs are void of any helpful information. We're able to determine when it stopped since a user was actively using it up until it went dead. It just stops working. Performing a Show Interface of the dead port has in some cases caused the switch's management Plane (control plane appears to route just fine for a little bit) to spike to 100% CPU and eventually, with enough time, the Control plane locks up, resulting in an immediate reboot to bring things back to normal. Console cable shows a frozen/usable session during this scenario.

Hope nobody else has to deal with this. Cheers.

tl;dr - Dell N2200 and N3300 Series switches as of 11/17/2020 with Firmware Version 6.6.1.0 to 6.6.3.3 have a fatal issue of ports not working. Issue was admitted and is currently being worked on by Dell. ETA of fix is the first week of December 2020.



No comments:

Post a Comment