Using Cisco Spectrum Expert and Backtrack to identify wireless anomolies

Cisco wireless controllers are great products, they allow the administrator to manage the entire wireless farm from a single console, making bulk changes and problem solving as required. Cisco recently released the clean air access points that take troubleshooting to a new level without the need for expensive spectrum analyser cards.

Recently I began receiving e-mail alerts from the wireless controller complaining of an "WiFi Invalid Channel", the exact error message was.

WCS has detected one or more alarms of category Security and severity Critical in Virtual Domain root for the following items:

Security-risk Interferer 'WiFi Invalid Channel' is detected. (2 times)

E-mail will be suppressed up to 30 minutes for these alarms.

Every 30 minutes the error would repeat, over and over again. The strange thing for this particular error was it would still alert in the middle of the night, excluding most business devices and devices like microwaves. While this wasn't affecting the clean air rating of the AP dramatically, it was continually triggering alerts and being flagged as a security issue.

After sending out an e-mail to key staff asking if any new wireless based equipment was installed recently and receiving no response I broke out the Cisco Spectrum Expert.

Detecting the problem

Fortunately the Cisco 3500i series clean air access points can be used in conjunction with Cisco Spectrum Expert software to troubleshoot issues such as this.

To do this you need to head over to Cisco.com and grab a copy of Cisco Spectrum Expert, it wasn't available in my download portal, but a quick email to Cisco resolved that.

Your AP can't service clients for the duration of the Spectrum Expert usage, so plan to do this after hours when your users won't be impacted.

After setting your AP to se-connect mode, either from the wireless controller or by connecting directly to the AP console, you can point Spectrum Expert at the AP and start analysing the results.

As soon as I fired up Spectrum Expert I was presented with the "WiFi Invalid Channel". While there is a great deal of detail, nothing definitively helped me identify what the problem device is. I tried searching in google for the exact frequency of the device but wasn't able to dig up any results.

One useful piece of information is the dBm (signal strength), that at -90.7 suggested the problem device in question was some distance from AP performing the analysis.

Where is it?

One question leads to another, I don't know what this device is, but can I find it? For this I fall back to a trusty laptop, my Alfa 500mw USB wireless adapter (RTL8187 chipset) and of course Backtrack 5 R1.

I decided to use a tool I have rarely used in the past, ssidsniff, which as its name suggests is normally used for uncovering hidden SSID's. Ssidsniff was chosen purely because I found it easier to view the BSSID and signal strength than in airodump (where BSSID's were jumping all over the screen based on the AP's current signal strength).

I quickly identified 00:00:00:00:00:00 as the problem BSSID, ssidsniff flagged it as "no identifiable channel" and "network only contains hosts" indicated by the H flag. While the valid AP's in my environment displayed as being "WPA/WPA2 capable". It may be totally different based on the device causing the problem, but it was extremely easy to identify this device as an anomaly compared to the rest of the devices.

My tracking process went as follows,

1) Starting right below the AP that originally detected the wifi invalid signal, I started ssidsniff and measured the dBm of  00:00:00:00:00:00.

2) I moved 5-10 metres in any direction then remeasure. I ctrl+c to kill ssidsniff and re-launch it every time I move to get the latest dBm. If the signal is getting stronger (which would be indicated by the dBm getting closer to 0, for example -25 is a stronger signal than -70) I keep moving in that direction, otherwise I change direction.

3) Repeat the above process until you find the highest signal strength you can, then look around.

Within about 5 minutes I had a dBm of -20, I found myself right next to a wireless microphone receiver, which funnily enough was turned on. After switching the receiver off and checking spectrum expert the invalid WiFi channel was gone, problem solved! You can then either suppress the error or replace the at fault equipment.

I am sure there are more technically amazing ways to accomplish this task but an inexpensive WiFi adapter and Backtrack was able to solve this problem perfectly.