Windows 7 WMI validation hotfix improves slow network logins

Like many other system administrators of Windows 7 systems, I have been experiencing some very slow logins on domain joined Windows 7 systems.

Some of the common factors of these slow network logins include:

• Computers that multiple users log into, for example Education computers where 100s of users log in
• Computers with a large number of group policies
• First time logons
• Windows 7 machines
• Machines hanging on "Applying user settings" or more commonly "Preparing your desktop"

The problem normally develops over time and as more and more users have logged in at least once, the login process slows even more. Some users on Microsoft forums are reporting login times as long as 30 minutes, but in my case the maximum times are 2-3 minutes. Fortunately for users that have previously logged in, login times are significantly quicker, but in Education using the same computer twice can be a rare occurrence.

After noticing some WMI and group policy based event log entries I decided to do some debug logging of the group policy service. This can potentially give me some in depth information as to what is happening during the 2-3 minute wait during login. 


Enabling the Group Policy debug logging

1. Open regedit and navigate to
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Diagnostics

2. Set the following value
"GPSvcDebugLevel"=dword:00030002

3. It is best to reboot and then login with an account that has previously never logged into that specific system.

4. After the user has logged in you will find the log located at %WINDIR%\debug\usermode\gpsvc.log.

Make sure you note the time of when you start the login and when you successfully make it to the desktop, this should help your identify the events in the log.

5. Make sure you remove the registry value when your done or else you may fill your hard drive with GPSVC logs.

I recommend using a tool like the SMS Trace32.exe to view your logs, it is much easier to search and will highlight warning and error events.

There is also a neat tool by Syspro called Policy Reporter that is able to enable, log and report on Group Policy Service logs in a more "user friendly" manner, its worth checking out.



Attempting to interpret the logs

Interpreting the gpsvc.log is easier said than done and unfortunately there doesn't seem to be much official documentation to help you do so. I found the easiest thing to do is look for significant time gaps in the logs and drill down on them more intensely.

I found the below section of the log repeated across a number of affected machines I tested. It references S-1-5-18 which is the SID for "Local System" and mentions "Caller requesting for user notification/lock is from session 0" and then hangs for around 1m 30s before the next log entry.

This time of 18:24:47 is during my "hang" period where I should already be at the desktop. 

GPSVC(388.b68) 18:24:47:795 Target = S-1-5-18
GPSVC(388.b68) 18:24:47:795 Could not find user by sid, finding user by session id
GPSVC(388.b68) 18:24:47:795 Caller requesting for user notification/lock is from session 0
GPSVC(c64.c78) 18:24:47:795 Client_RegisterForNotification: User = S-1-5-18, changenumber = 0
GPSVC(c64.c78) 18:24:47:796 CGPNotify::RegisterForNotification: Exiting with status = 0
GPSVC(388.b68) 18:24:47:796 Could not find user by sid, finding user by session id
GPSVC(388.b68) 18:24:47:796 Caller requesting for user notification/lock is from session 0
GPSVC(36c.420) 18:26:10:537 CGPNotify::UnregisterNotification: Entering with event 0x4ec
GPSVC(36c.420) 18:26:10:537 CGPNotify::AbortAsyncRegistration: No asyn registration is pending
GPSVC(36c.420) 18:26:10:537 CGPNotify::UnregisterNotification: Canceling pending calls
GPSVC(36c.420) 18:26:10:538 Client_CompleteNotificationCall: failed with 0x71a
GPSVC(36c.420) 18:26:10:538 CGPNotify::UnregisterNotification: Cancelled pending calls
GPSVC(36c.420) 18:26:10:538 CGPNotify::UnregisterNotification: Exiting with dwStatus = 0x0
GPSVC(388.f0c) 18:26:22:944 CGPNotify::RegisterForNotification: Entering with target Machine and event 0xdb0
GPSVC(388.f0c) 18:26:22:988 Client_InitialRegisterForNotification: User = machine, changenumber = 0
GPSVC(388.b68) 18:26:23:000 Target = Machine
GPSVC(388.f0c) 18:26:23:000 Client_RegisterForNotification: User = machine, changenumber = 0
GPSVC(388.f0c) 18:26:23:026 CGPNotify::RegisterForNotification: Exiting with status = 0

After trying a number of experimental techniques to improve the login speed I only managed to partially destroy an OS, so I gave up for a few weeks, that is until I saw Microsoft hotfix KB 2617858. 
 

Hotfix KB 2617858 to the rescue

This hotfix is described as fixing "Unexpectedly slow startup or logon process in Windows Server 2008 R2 or in Windows 7". This is down a slightly different path than the group policy logs I was examining above but ties in with group policy WMI filtering and the WMI event log errors I previously found.

The error is caused by Windows Management Instrumentation (WMI) performing a full validation of the WMI respository when it is not required. This operation takes a long time and delays the login process. This hotfix prevents this full validation from occurring and can alleviate the login delays.

I found this hotfix to significantly improve login times for "first time" logins. While the login is not nearly as fast as it is for users that have previously logged onto the system, it only takes around 1 minutes as opposed to the 1-4 minutes it took in the past.

I have started deploying this hotfix to one heavily affected area in my environment, as soon as I can confirm there are no caveats I will roll it out to all affected systems.

If your problem still persists after applying this hotfix then another debugging solution might be to use Sysinternals Process Monitor in "boot logging" mode. I tried this method but the resultant log was some 8 million lines and extremely hard to examine. This method didn't work for me but depending on your problem it may be another troubleshooting alternative.

Citrix Xenserver 6.0 fails update pre-check when applying updates

A while back I upgraded my Xenserver 5.6 FP1 servers to Xenserver 6.0, the upgrade was very smooth, well so I thought. Today I had scheduled some time to apply the 4 Xenserver 6.0 hotfixes currently available but hit a few snags along the way.

When I went to upload the new hotfixes, 1 of my Xenserver complained that pre-check 5 of 5 had failed, the exact error message was as below:

SERVERNAME: The update precheck stage failed: prerequisite updates are missing.

I also noticed the server was in a permanent update pending state (as can be seen from the arrow icon below), even though I hadn't initiated an update.

 

Upon further investigation I found the server was still trying to install old Xenserver 5.6 updates that I never installed before upgrading the server, which was in turn blocking the new updates from installing. This can be caused by not installed updates on the pool master first, but in my case the servers are not in a pool.




Resolving the problem

Fortunately the problem is (or was in my case) quite easy to fix.

1. Open Xencenter, select the server in question and click the console tab.

2. Issue the "xe patch-list" command, this command will give you a list of pending or installed patches.
xe patch-list

3. For every patch with a "name-description" containing 5.6, obtain the UUID (for example in the above image the UUID is "17fde43e-0a5e-48ac-8b85-cf6ed9c6344d"), then issue a "xe patch-destroy" command.

xe patch-destroy uuid=<UUID>

for example:
xe patch-destroy uuid=17fde43e-0a5e-48ac-8b85-cf6ed9c6344d

4. You may or may not need to restart the server but if you have been successful the "down arrow" icon next to the server name will disappear and return to a "green circle" (as per the image below).

This simple fix should resolve your problems, although some users on the Citrix forums have reported similar problems that are more serious. In most cases these users have migrated the data to another server and rebuilt the server.

I still don't know how this problem was caused, but I suspect I simply had pending updates on this particular server before I upgraded it to Xenserver 6.0 that became "stuck".

Exploiting WPA2 WPS vulnerabilities with Reaver and Backtrack 5

On December 27th, 2011, a new WPS vulnerability was disclosed by Stefan Viehböck. This severe vulnerability has the potential to expose pass phrases of any WPA/WPA2 networks running WPS via a brute force attack. US-Cert have acknowledged this attack with Note VU#723755.

WiFi Protected Setup (or WPS) is designed to make life easier for the average user. In most cases the user will press a button on their wireless access point, this will initiate an easy pairing sequence between the router and the client. If you have recently purchased a wireless router chances are it has WPS and it has probably been enabled by default.

Stefan worked out that the PIN used between the router/client is only 8 characters in length and that the last digit is a checksum, making the brute force password length 7 characters. To make matters worse, the router splits the 7 characters into 2 PINs, one of 4 characters and one of 3 and worse again it will confirm both pins independently. This means that there are 10⁴ + 10³ possible combinations or 11,000 in total.

Some routers have built in protection that will only allow a specific amount of PIN attempts per allotted period of time, this will only slow the attack process though, if someone is willing to wait long enough they will recover the pass phrase.

While Stefan was exploring this vulnerability, so were the good folks at Tactical Network Solutions with their release of a brute force attack software named reaver-wps. Reaver is able to brute force attack WPS PINs with the bold intention of recovering WPA/WPA2 pass phrases, lets take a look.



Using Reaver

This tools is extremely easy to use, I recommend a RTL8187 based wireless adapter (you already have one of these for your WEP pen testing right?).

1. Boot Backtrack 5, establish a network connection and install reaver.
apt-get install reaver

2. First lets take a look at the available networks, I still use airodump-ng for this.

airodump-ng wlan0

3. After finding the BSSID we are interested in (the one you just set up for your proof of concept), issue the following reaver command, replacing "00:11:22:33:44:55" with your target BSSID and "wlan0" with your adapter.

reaver -i wlan0 -b 00:11:22:33:44:55 -c 1 -vv

The above command is attacking BSSID "00:11:22:33:44:55" on interface "wlan0" and channel 1, it is also using a high verbose level.

You can see from the image (click to enlarge) that Reaver begins to brute force combinations of pins. This process can take hours, the Reaver website suggests on average it will take between 4-10 hours to recover a pass phrase. The particular AP I tested the attack against had some PIN rate limiting protection  (as reported by Reaver) that significantly delays but doesn't stop the attack. I didn't leave the attack going more than a few minutes but you get the idea.

Fortunately there is a simple fix, disable WPS or even better, move to WPA2-Enterprise with a RADIUS back end. With any luck the WiFi Alliance and associated manufactures will release firmware updates quickly to resolve this issue, but for the time being millions of wireless access points remain vulnerable to this simple attack.

PC and Tech Authority cover the Team Australia OC Workshops

Check out this article published in the latest PC and Tech Authority and Atomic magazines. It covers a recent overclocking workshop that the Team Australia boys and myself recently ran in Brisbane.

The concept was to introduce PC enthusiasts to extreme overclocking with Liquid Nitrogen. We covered everything from insulation to safe use of liquid nitrogen and how to achieve big benchmark results. We had a large turn out and made some great friends.

Click on the image below to access the article written by John Gillooly.

Sophos Update Manager stops updating, downloading binaries hangs forever

I was recently doing some maintenance on my Sophos Enterprise Console 4.7 when I noticed that updates had not been downloading for nearly a full month. Nothing in my environment had changed, so this was something I didn't expect.

After rebooting, re-entering my upstream update server credentials and deleting the relivent SUM directories I still had no luck.



The Problem

I first noticed there was something wrong as the "Last Updated" field was nearly a month old, which would indicate something was blocking the updates. Checking further I found the following logs/error messages.

In the Sophos Update Manager (SUM), the "Download status" was "Downloading binaries". Even 2 hours later, this status didn't change. I was not able to find any file system activity with Process Monitor so I presumed this was a hung process.

In the SUM I found the following error:

CODE: 80040406
Description: Delivery failed for software subscription 'Recommended'. Access to the source update location is denied or the location is otherwise available.

In the Windows Application Event Log I found the following error:

Log Name: Application
Source: SophosUpdateManager
EventID: 16443
Level: Error
Product release 'A845A8B5-6532-4EF1-B19E-1DB2B3CB73D1' could not be updated because the synchronize operation has failed due to an earlier error.

The SUM trace logs, which was located in "C:\ProgramData\Sophos\Update Manager\Logs", displayed the following error:

2011-12-29 19:46:31 : EventLog: 3758112769 1 Inserts:> "F26F7EC0-1302-4DA7-8B6B-A5383051D41A" "EXCEPTION_ACCESS_VIOLATION at 0x74BB4500" "RECOMMENDED" "http://contoso.com/databank/Warehouse"

There are a number of error messages associated with "access violation" or "source denied", all pointing to a permissions issue with the upstream update server. So I opened up a web browser, put in the upstream update server address and my username and password with no problems. The SUM is reporting a permissions error, yet my web browser can access the update source with no problems. Then whats going on?



The Resolution

Unfortunately these error messages are extremely misleading. If you were to purely troubleshoot SUM on its error messages you would never fix your problems.

After some google detective work by my mate Simon (thanks for your help with this one), he was able to uncover this Sophos forum thread. This thread gives some good tips on how to fix the problem, but I will break it down into easy to follow steps.

1. Stop the Sophos Agent Service, Sophos Message Router Service and the Sophos Update Manager Service.

2. Open Task Manager and kill the "SophosUpdateMgr.exe" and "SUMService.exe" processes if they are running. This will kill SUM if it has hung.

3. Open regedit and navigate to
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\{2C7A82DB-69BC-4198-AC26-BB862F1BE4D0}]

or if your on 64bit
[HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Windows\CurrentVersion\Uninstall\{2C7A82DB-69BC-4198-AC26-BB862F1BE4D0}]

4. Check the "UserData" value.

"UserData"="YourServer;YourDomain;SophosUpdateMgr;0;0"

The first part of this value (to the left of the first semi-colon) should be that of your SUM server. On my server (and it seems by the above linked Sophos forum the same applies to a number of other users) this "YourServer" value reads as a domain controller and not the Sophos server.

If the "YourServer" value does not reflect your SUM server then change it to reflect the correct server name.

5. Start the Sophos Update Manager service and wait 5 minutes. (The wait is important)

6. Start the Sophos Agent service and the Sophos Message Router service.

Voila, your SUM should be working again, at least it will be if you had the same error I did.

The "UserData" registry value is very interesting. Most of the time it doesn't matter if it reads as the domain controller and not the SUM server itself, in fact as soon as I updated again it changed back to the domain controller, yet I had no problems in the update process.

I believe a problem may arise when the "UserData" value is incorrect and there is a pending update to Sophos Update Manager itself, either way the above short process should resolve your issue.