Friday, January 18, 2019

Complete Network Failure When Cisco 3800 Series AP Joins Cisco 3500 Series WLC.

I have 2 WLC's on my network, a 3504, and a 2504 (for older AP's slowly being upgraded). Recently we purchased two new AIR-AP3802I-B-K9 AP's, and life has been swell. I unbox them, plug them into a switch in the proper VLAN and look at that... wireless. It's beautiful when it works. That said, we also ordered some spare AP's just in case (same model). I made sure these spare AP's could join the controller, and registered properly with the correct software version, they did. They were named SPARE-AP-01, and SPARE-AP-02 and placed back in their original box until they need to be called into service.

Fast forward, I go to install 2 new AP's as 2 new lines were recently pulled, tested, and certified, I install the first AP, and it joins as expected. I name it, etc... and move on to the next AP. I see the light's start flashing and then the blue beacon of happiness saying it's downloading the controller software, awesome. The light turns green, it associates... and I have no LAN communication. I can't ping out, I can't hit internal servers, nothing. I unplug the AP that just joined the network, and everything IMMEDIATELY comes back. I can ping out, hit servers, etc..

I proceeded to unplug the POE to other new AP just installed, and plug in the second one, it joins without problem, now I can configure it, name it etc.. cool. I plug in the AP that was previously JUST JOINED to the controller, and the same thing happens.. all LAN traffic ceases. I unplug the AP and traffic starts flowing again..

OK.. did we get 2 bad AP's? I plugged in SPARE-AP-01, it joined as expected, and then I plugged in SPARE-AP-02 and again.. all LAN traffic completely halts. I'm thoroughly at a loss for what could be causing this. I've tried different switch ports, made sure all connections are identical to functioning AP's, and everything matches.

I can provide some basic details below and provide more as needed (I don't want to dump out too much irrelevant info):

Hardware In Play:

3504 Controller: 10.0.1.249 (SW: 8.5.131.0)

  • Currently supporting 6 3802 AP's, and 6 2602 AP's

  • New AP's are configured to connect to this WLC first, and then join the older controller if this one is unavailable

  • Hands out 10.30.0.0-10.35.0.255 IP via internal DHCP server

  • Existing AP's DHCP leases converted to static

2504 Controller: 10.0.1.248 (SW: 8.3.143.0)

  • Currently Support 13 1261 AP's

  • Has supported the 3800 series AP's in the past

  • Hands out 10.30.0.0-10.35.0.255 IP via internal DHCP server

  • Existing AP's DHCP leases converted to static

Switches Powering AP's via PoE:

  • Juniper EX 3300-48p - powers all of our AP's without prior issues

  • All switches carrying AP's are configured identical, working AP's are configured the same way as the ones not working causing the LAN drop

Troubleshooting:

  • Replace 2 new AP's causing issues with 2 known good spares

  • Verify all physical connections

  • Verify switch configurations

  • Verify controller configurations

  • Verify DHCP/address leases

  • Try 1 new AP - Works (independent of one vs. the other, it's the same result)

  • Try both new AP's - Network Fail (if 2 new AP's are on the network fails, does not matter which one is up and which one is down)

Myself, as well as my escalation support are completely at a loss as none of this makes any sense (at least to us), and neither one of us have ever even heard of something like this. I'm hoping maybe one of you more seasons guys or gals can help out and think of something we might be missing.

Thanks to all!

Edit 0: Formatting.



No comments:

Post a Comment