DPM 2012 R2 only allows offline backups of VM with updated integration tools

All DPM admins that backup Hyper-V VMs have seen the dreaded offline backup state before. Admins of earlier versions of DPM might know this as "Saved State", either way its annoying and disruptive to backup VMs in an offline/saved state.

Recently when we did a migration of VMs from one Hyper-V cluster to another with SCVMM, we came across an issue where most of the VMs appeared as "Offline backup" in the DPM console.

This can be caused by a multitude of issues, including not having a SCSI adapter connected to the VM and running an old version of the integration tools.

However after checking all the requirements (listed below), we were still only able to capture offline backups on some VMs.

• The Backup (Volume Snapshot) Integration Service is disabled or not installed.
• A virtual machine has one or more dynamic disks.
• A virtual machine has one or more volumes that are based on non-NTFS file systems.
• In a cluster configuration, the virtual machine Cluster Resource Group is offline.
• A virtual machine is not in a running state.
• A Shadow Storage assignment of a volume inside the virtual machine is explicitly set to a different volume other than itself.

In testing we removed a VHD from a VM that had multiple VHD connected and then tested it, this worked! Funnily enough, when we re-added the VHD, the VM still could be backed up online. This led us to a fix that worked on most of our problematic VMs.

The Resolution

1. Open up Hyper-V or SCVMM.

2. Go into the hard drive settings of the problematic VM and remove all VHD (not delete, just remove for now - do I even need to say that?).

3. Save the VM.

4. Re-add the VHD and save the VM again.

5. Start the VM

6. Attempt to backup with DPM again, you may need to clear the state cache in DPM

Essentially we are doing nothing, removing and re-adding a configuration. However this seems to resolve the issue on most VM. This may or may not be similar in your environment, as these VM had all been migrated between Hyper-V servers with VMM.

Lync 2013 CMS error: Instance points to a CMS store that is being moved to a different pool

We were doing some routine checks of a newly provisioned Lync 2013 front end fail-over pool when we discovered a CMS error on the backup pool in the event logs. Our deployment is configured with two standard edition front end pools with resiliency.

The error is as follows:

Event ID 4082
LS Backup Service

Microsoft Lync Server 2013, Backup Service central management backup module failed to complete import operation.
Configurations:
Backup Module Identity:CentralMgmt.CMSMaster
Working Directory path:\\lync002.contoso.info\lyncstore\1-BackupService-20\BackupStore\Temp
Local File Store Unc path:\\lync002.contoso.info\lyncstore\1-BackupService-20\BackupStore
Remote File Store Unc path:\\lync002.contoso.info\lyncstore\1-BackupService-21\BackupStore
 Additional Message:
 Exception: Microsoft.Rtc.BackupService.ModuleUnavailableException: Backup module is temporarily unavailable at this point. Reason: Instance (local)\rtc points to a CMS store that is being moved to a different pool.
   at Microsoft.Rtc.BackupService.BackupModules.CentralMgmtBackupModule.CheckModuleAvailability(Nullable`1 primaryPool)
   at Microsoft.Rtc.BackupService.BackupModules.CentralMgmtBackupModule.GetBackupCookie()
   at Microsoft.Rtc.BackupService.BackupModuleHandler.ReceiveBackupDataTask.GetBackupCookie(Boolean& isModuleInitialized)
   at Microsoft.Rtc.BackupService.BackupModuleHandler.ReceiveBackupDataTask.InternalExecute()
   at Microsoft.Rtc.Common.TaskManager`1.ExecuteTask(Object state)
Cause: Either network or permission issues. Please look through the exception details for more information.
Resolution:
Resolution

This error message was repeated every 15 minutes as the backup jobs were run. A separate error was created for the import and export process.

During our testing phase we did some fail-over testing which included moving the CMS database to the fail-over node. We believe this may have caused this error to occur. While it was not causing any known issues in the environment, it may come to a head should we ever need to fail over to our resiliency node.



The Resolution

The resolution was fairly simple but did require a few attempts before we got the process order correct.


1. Open the Lync Topology Builder and remove the associated backup pool from the resiliency tab of the primary front end pool.

2. Publish the topology

3. Use the Get-CsManagementStoreReplicationStatus powershell applet to verify the topology has replicated to the associated backup pool.

4. Open a command prompt to "%ProgramFiles%\Microsoft Lync Server 2013\Deployment" and run the bootstrapper.exe on the backup pool. This will remove the management and backup service from the backup pool.

5. Remove the CMS database from the backup pool using the following powershell command.
Uninstall-CsDatabase -CentralManagementDatabase -SqlServerFqdn lync002.contoso.info

6. Go back to the Topology Builder and re-add the associated backup pool to the primary front end pool.

7. Publish the topology

8. Wait for the topology to replicate

9. Re-run the bootstrapper on the backup pool node

This 5 minute process fixed the error messages for us and hopefully saves us in the unfortunate even of a fail-over.

SCOM 2012 OleDB Module 0x80004005 errors after SQL database move

After migrating our SCOM 2012 R2 DB and Data Warehouse DB to a new SQL server we were receiving SCOM alerts that there was a problem with the OleDB module.


The Problem

The initial alert indicated that there was a login problem. This prompted us to check, re-check and triple-check all the SQL logins and permissions between the old and new SQL servers.

Alert description: OleDb Module encountered a failure 0x80004005 during execution and will post it as output data item. Unspecified error: Cannot open database "OperationsManager2012" requested by the login. The login failed.
Workflow name: Microsoft.SystemCenter.SqlBrokerAvailabilityMonitorForPool

After not having much luck we eventually decommissioned the old SQL server. Once the old SQL server was turned off, the alert changed from a login failed to a "SQL server does not exist". This error got us thinking, maybe its not a permission problem but some parts of SCOM may have been still pointing at the old SQL server.

Alert description: OleDb Module encountered a failure 0x80004005 during execution and will post it as output data item. Unspecified error: [DBNETLIB][ConnectionOpen (Connect()).]SQL Server does not exist or access denied.
Workflow name: Microsoft.SystemCenter.SqlBrokerAvailabilityMonitorForPool

The Solution 

A search of the registry found a few keys that were undocumented in the SQL migration document I was reading. These keys are:

HKLM\SOFTWARE\Microsoft\Microsoft Operations Manager\3.0\Setup\DatabaseServerName
HKLM\SOFTWARE\Microsoft\Microsoft Operations Manager\3.0\Setup\DataWarehouseDBServerName
HKLM\Software\Microsoft\System Center\SetupBackup\Blue\Database\DatabaseServerName

After changing these keys and restarting the SCOM server all was well!

Citrix, Windows 7 Thin PC, Thin kiosk and Wireless limitations

We have decided to go with a Xenapp model with predominately wireless devices. The challenge is how to get Windows 7 Embedded, write filtered, non-domain joined devices to connect to a wireless network and work smoothly. Our solution was to provision an open wireless network and present only a pair of DNS servers and a Citrix VPX cluster into the wireless network. This is effectively an internet style DMZ network that allows our internal wireless and BYOD machines to all connect in the same manner.

Thinkiosk is a great free product from Andrew Morgan that provides a locked down environment for launching VDI. As part of our adoption of Xenapp we wanted to modify our existing Windows 7 Thin PC image to support Thinkiosk in wired and wireless configurations.

Thinkiosk works great in wired deployments, but there are some limitations with wireless. If you are using Thinkiosk in an auto login, non-domain joined scenario, Thinkiosk launches very quickly, often before the wireless connection has initialized.



Making Thinkiosk work smoothly with wireless

To get around the existing limitations we have created a vbscript that waits for the Thinkiosk URL to become available before launching Thinkiosk. After 24 seconds if a connection isn't established, the wireless control panel applet is launched. After 60 seconds if a connection to the URL still can't be established then Thinkiosk is launched regardless.

You can modify these thresholds and URLs very easily in the below.

The script is available from my pastebin here

To launch the script on logon, simply replace your windows shell with cscript and the script path as per below

reg add "HKEY_local_machine\Software\Microsoft\Windows NT\CurrentVersion\Winlogon" /v shell /t reg_sz /d "cscript c:\windows\thinkiosklauncher.vbs" /f

Supporting wireless persistence with write filters

The other major limitation with Wireless and Windows 7 Embedded is the ability to retain wireless configurations when write filters are enabled. Write filters do a great job of keeping a consistent device and lower support, but they can also be restricted if local changes are required.

We want users to take the devices home and connect in the same manner as they would if they on site. We have configured split-brain DNS so the Citrix URL is accessible both internally and externally. For this to work smoothly our users need to be able to connect to their home wireless networks and the settings need to be remembered. Could you imaging typing in your uber secure 64 character WPA key every time you turn the device on? Yuck, no thanks.

We have determined the following exclusions need to be added to support wireless persistence on write filter enabled machines.

File: c:\programdata\microsoft\wlansvc\profiles
Registry: HKLM\software\microsoft\windows nt\currentversion\networklist\profiles
Registry: HKLM\software\microsoft\wlansvc\interfaces

Adding the above exclusions in conjunction with Thinkiosk wireless support gives your users the ability to connect and remember wireless networks on their thin client. Depending on the write filter method you are using you will may a different command, but for file based write filters you can add the file exclusion as per below.

fbwfmgr /addexclusion c: "\programdata\microsoft\wlansvc\profiles"

The registry settings are a little more involved, we suggest reading the following blog to get some more insight on how that's achieved http://geekswithblogs.net/WallabyFan/archive/2008/12/24/everything-you-wanted-to-know-about-fbwf-but-were-afraid.aspx

Thinkiosk wireless client support can be enabled with the following registry key.

reg add "HKEY_local_machine\Software\THINKIOSK" /v ShowWifi /t reg_dword /d "1" /f

We have also added a line to the above Thinkiosk wireless launcher script to re-import our internal wireless network after each boot. This is just to ensure our users don't accidentally (or intentionally) delete our wireless network.

objShell.exec("Netsh wlan add profile filename=c:\windows\wireless-open.xml user=all")

The above wireless-open.xml configuration can be exported with the netsh wlan export tool and then re-imported on each boot or login to ensure your network is never permanently removed.

Publishing Server 2012 R2 Work Folders with UAG 2010 SP3 Reverse Proxy

We already knew before 2012 R2 came out that we were going to use Work Folders in a trial later in the year. As soon as the ISO dropped we began organizing our server environment to handle Work Folders synchronization. Of course the biggest part of this is allowing synchronization from home, so reverse proxying a remote access solution was a must.

We use UAG 2010 and while Microsoft may have a wizard in the upcoming SP4, unfortunately is not available yet for early adopters. Here are the steps we used to make it work.



Prerequisites

Before you start it is best to use split-brain DNS for a smooth and speed workfolders experience. Create a new DNS record on your internal DNS servers using the external DNS FQDN with a low time-to-live, maybe 15 minutes, point this to your internal workfolders server IP. Then create the same DNS record on DNS servers authoritive for your external records and point this to your UAG trunk.

The records will resolve something like:
Internal: workfolders.consoto.info 192.168.1.1 (Internal Work Folders server)
External: workfolders.consoto.info 180.0.0.1 (External UAG Trunk)



Step by Step

1. Open UAG and take an existing trunk (or create a new trunk) that has Trunk authentication disabled. The trunk we used had a wildcard certificate and it worked perfectly. 

2. Add a new application. 

3. Select "Other Web Application (application specific hostname)", click OK. 

4. Name the application "workspaces" and application type "workspaces", click OK. 

5. Select "Configure an application server", click OK. 

6. In "addresses" enter your workspaces URL, we are using split-brain DNS so the internal and external address will be the same. 

For paths enter "/sync/1.0/" as this is the only part of the Work Spaces server that the reverse proxy needs to forward. 

In public hostname enter workspaces, you will need to create the corresponding external A record (or CNAME record for existing trunks), click OK. 

7. Leave "use SSO" unticked, click OK. 

8. Un-tick "add a portal and toolbar link", click OK. 

9. Leave "Authorize all users" ticked, click OK. 

10. Click finish to create the application. 

11. Now we must make some modifications to the URL set, to do this click "Configure trunk settings" under the trunk. 

12. Click the URL set tab and find the "workfolders_Rule1" rule. The rule already will show the URL of "/sync/1.0/.*", we need to modify the methods. 

The default methods are POST and GET, add in DELETE, PUT and HEAD. 

Click OK to save the settings. 

13. Click the "workfolders" application and select edit. 

14. Select the Web Settings tab and click "Allow POST requests without a content-type header", then click OK to accept the changes. 

15. Save your changes and activate your configuration

Aastra 6725ip POE problems with HP Procurve switching

We recently received a number of Aastra 6725ip handsets, the handsets immediately powered up via POE and connected to our Lync 2013 deployment. After I returned to work on Monday morning none of the phones were still connected to the Lync server, after rebooting them they no longer booted from POE.

This problem was introduced when they updated to newer OC Phone firmware editions 4.0.7577.4397 (CU9) and above. Although we can't confirm if the issue was introduced in firmware versions CU8 and below.

We tried a number of different Procurve switch models including other 5400 series switches (all fully populated with 900w POE+ power supplies) and 2900 series models, all experienced the same no power problems. The switches complained of not detecting an MPS signature and occasionally the switch also gave over current detection errors. If we used a POE injector or DC power supply the phone started without an issue. We also verified this was not an issue with POE power limits, pre-std-detection or LLDP.

After contacting Aastra they asked us to take a photo of the label underneath the phone. After the technician viewed the label they immediately offered to RMA all my phones but didn't give any reasons as to why they needed to be RMA'd.

After receiving the new phone I compared the labels and to no surprises the replacement unit is a Revision B compared to the previous Revision A unit.

While this is not in any way confirmed by Aastra, we theorize that in later editions of Lync 2013 phone edition some changes were made to the POE signature that caused incompatibilities with the Revision A Aastra 6725ip handsets and HP Procurve swit ches. If you are experiencing similar issues Aastra and very efficient at dealing with issues and promptly shipped me new handsets without any hassle.

The old handset

The new handset

Lync 2013 client popup credentials are required for Outlook

When we started migrating users to Exchange 2013 and Lync 2013, users began complaining of being prompted to enter their credentials for Outlook when starting Lync.

The exact error message is:
Credentials are required
Lync needs your username and password to connect for retrieving calendar data from Outlook

We did some Wiresharking and found this to occur when Lync was connecting to the EWS and Autodiscover Exchange URLs.



The Solution

1. Ensure your internal and external EWS/Autodiscover URLs are in the Internet Explorer local intranet zone. This will be extremely helpful in assisting you to troubleshoot the problem.

2. Logon to your Exchange server and open IIS.

3. Click the EWS directory under "Default Web Site".

4. Open Authentication, click Windows Authentication and then Providers.

5. Remove all entries except for NTLM.

6. Repeat the same for the Autodiscover directory.

7. Restart IIS.

Now restart your client and you should no longer be prompted for Outlook credentials.



Additional troubleshooting tips

Internet Explorer debug tools are great for troubleshooting authentication issues, but first ensure your EWS/Autodiscover URLs are in the Internet Explorer local intranet zone.

A simple test is to go to the EWS URL (http://yourdomain/EWS/exchange.asmx) in Internet Explorer, it should automatically negotiate authentication and show you the service page. If you don't receive the "You have created a service" page then your single sign on is not correctly configured.

1. Open Internet Explorer and press F12 to launch the debug tools.

2. Click Network and "Start Capturing".

3. Now open your EWS URL: http://yourdomain/EWS/exchange.asmx

4. Click "Stop Capturing" and click "Go to detailed view"

You can see the order of the authentication providers and any issues that might have occured during authentication. You can then repeat the same for Autodiscover, remember if Autodiscover can't negotiate pass through authentication then it will prompt the Lync user for credentials.