Classified to De-Classified a New Bug.

Classified

 

Classified to De-Classified a New Bug.

A couple of months ago I chose an additional site to install the new WLC code 7.6.110.0. (Prior to the now 7.6.120.0 release) and encountered a new bug.

Now enter a pair of revision v0- Cisco 5508 WLC’s with lots of AP’s. {Why is the hardware revision important? I will circle back on that later}

After a standard software download to the controllers and a reboot I started to notice something odd while monitoring the console port.  At first glance I thought, did I just enable a debug by mistake? Well the controller decided to have a Crash Dump and judging by the size of the crash files, a big one it was.  The controller would only stay online for less than five minutes, even with no AP’s connected.

“What the heck could possibly be happening?”
“At this point you’re asking yourself what did I do?”
“Did I miss something during the FUS or software upgrade that didn’t apply properly perhaps?{review console logs}”

This was happening very fast and my downtime window still had plenty of time to spare. So I stopped and called Cisco TAC.  You mean you couldn’t figure it out yourself?  My first rule of thumb when working on production systems and you encounter something that you just can’t put your finger on it and you don’t always have Sam Clements  @samuel_clements (WiFi Rock star) sitting beside you.  CALL Cisco TAC that’s what they are there for….think of it as an insurance policy when things go sideways.

Enter Cisco TAC.
After a closer examination of the crash files it seemed to be CDP / SNMP related issue.  What CDP / SNMP? No way, really?  Yes really.  At first glance you wouldn’t think it, as each crash dump was showing CDP or SNMP then both.

On TAC’s advice we turned off SNMP, and it starts to be stable. Then we add SNMP back in and then crash is back.  Ok SNMP seems to be the issue.  Not so fast…  enter crash logs… {Yes make sure you configure crash log output on your controllers to a FTP server this will come in handy for troubleshooting…}  The crash file indicates CDP.  So is it a combination and SNMP? Apparently yes.

Here is a quote from the bug details. https://tools.cisco.com/bugsearch/bug/CSCuo12321

The vulnerability is due to a failure to properly check for certain NULL values present in a CDP packet. An attacker could exploit this vulnerability by by submitting crafted CDP requests to an affected device. The attacker would then need to convince a user to perform an SNMP poll of the affected MIB.. An attacker that can convince a user to perform an SNMP poll of the affected MIB could trigger a NULL Pointer error that results in a restart of the device.”   

“An attacker would need direct Layer-2 access to an affected device. This may also be triggered in circumstances where a non-Cisco switch that does not parse CDP is utilized with a Cisco WLC”

For some strange reason once the pair of controllers started sharing information via the mobility group the controllers became overwhelmed with CDP and SNMP information that it just crashed. Now an interesting twist is the revision number of the controller.  In separate tests with physically different 5508’s we had mixed results.  For example, revision 2 did not exhibit this behaviour even though this bug was easily reproduced on revision 0.

If you come across this behaviour with software revisions  7.5.x to 7.6.110.0 keep in mind that this issue has been fixed and made it into 7.6.120.0 and above. My advice is move away from those older versions now.

We end the blog at the title that got your attention. Only until just the other day “July 11th 2014” this bug had been previously Classified (eyes only) by Cisco.  Cisco has Classified bugs?  Yes this is true and for good reason.  If you look at the details of the bug you will see a PSIRT identification.  Because of the nature of this bug it needs to be identified, validated, tested and fixed promptly.  Denial of Service attacks are not a laughing matter and Cisco takes them very seriously.  (Your SMARTnet dollars at work)

I have personally had three such Classified bugs in my lifetime and sometimes Cisco TAC can’t release the Bug ID just yet.  So the next time you happen to come across one, be patient with the Cisco TAC Engineer as they will eventually get it to you as long as you keep your case open.

Remember its ok if you don’t know the answer sometimes, so reach out to your friendly neighbourhood Cisco TAC Engineer.  If its non critical..  reach out to your friendly mobility folks on either on the https://supportforums.cisco.com or for Cisco Partners https://communities.cisco.com  and of course Twitter

 

Stew

@WirelessStew

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s