DPM 2012 R2 failing to backup SQL 2012 SP1 Always On Cluster VM

After we created a new Server 2012 R2 cluster and migrated our VMs across, the next job was to get backup in place with DPM 2012 R2.

Most of the machines backed up with Online backup without issue and our SQL cluster was no exception, the online backup initiated without fault. However after some time one of the Always On cluster came back with the following error.

The VSS application writer or the VSS provider is in a bad state. Either it was already in a bad state or it entered a bad state during the current operation. (ID 30111 Details: VssError:The writer experienced a non-transient error.  If the backup process is retried,
the error is likely to reoccur.
 (0x800423F4))

Digging further on the SQL machine with the "vssadmin list writers" command, we found that the SQL VSS Writer service was in an error state.



The Solution

The solution is very simple but introduces some other problems.

1. Jump on the erroneous SQL server.
2. Open a command prompt and type services.msc
3. Go the SQL VSS Writer service and stop the service.
4. Re-run the DPM job.

This successfully completes the VM snapshot backup, but it introduces an issue with SQL backup if you are also using DPM for SQL backup on the same Always On cluster.

In a SQL Always On cluster, the DPM backups are by normally taken from the secondary node. The primary node is left alone to do SQL, while the inactive secondary takes the heavy backup load.

This is why we only had the problem on our secondary node, it is also why it introduces an SQL backup problem. When the SQL VSS Writer service is stopped, DPM can't perform SQL backups on the server.

Two possible options are:

1. Don't perform VM snapshot on your backup node, it may be overkill anyway. In the event of a failure you can spin up a new VM and make it the new secondary.

2. Run a schedule that disables the SQL VSS Writer service some time before the VM snapshot and re-enable it again after the snapshot.

We are using option 2 and so far it is working well. You need to ensure that your SQL backup isn't occurring during your SQL VSS writer disabled period, but your a good admin and wouldn't have your VM snapshot and SQL backup scheduled at the same time anyway!

DPM 2012 R2 only allows offline backups of VM with updated integration tools

All DPM admins that backup Hyper-V VMs have seen the dreaded offline backup state before. Admins of earlier versions of DPM might know this as "Saved State", either way its annoying and disruptive to backup VMs in an offline/saved state.

Recently when we did a migration of VMs from one Hyper-V cluster to another with SCVMM, we came across an issue where most of the VMs appeared as "Offline backup" in the DPM console.

This can be caused by a multitude of issues, including not having a SCSI adapter connected to the VM and running an old version of the integration tools.

However after checking all the requirements (listed below), we were still only able to capture offline backups on some VMs.

• The Backup (Volume Snapshot) Integration Service is disabled or not installed.
• A virtual machine has one or more dynamic disks.
• A virtual machine has one or more volumes that are based on non-NTFS file systems.
• In a cluster configuration, the virtual machine Cluster Resource Group is offline.
• A virtual machine is not in a running state.
• A Shadow Storage assignment of a volume inside the virtual machine is explicitly set to a different volume other than itself.

In testing we removed a VHD from a VM that had multiple VHD connected and then tested it, this worked! Funnily enough, when we re-added the VHD, the VM still could be backed up online. This led us to a fix that worked on most of our problematic VMs.

The Resolution

1. Open up Hyper-V or SCVMM.

2. Go into the hard drive settings of the problematic VM and remove all VHD (not delete, just remove for now - do I even need to say that?).

3. Save the VM.

4. Re-add the VHD and save the VM again.

5. Start the VM

6. Attempt to backup with DPM again, you may need to clear the state cache in DPM

Essentially we are doing nothing, removing and re-adding a configuration. However this seems to resolve the issue on most VM. This may or may not be similar in your environment, as these VM had all been migrated between Hyper-V servers with VMM.

DPM 2012 SP1 replica inconsistent - datasource is owned by a different DPM server

Recently we took the leap of faith to DPM 2012 SP1 + Update Rollup 1. SP1 offers proper compatibility with SQL 2012 and Server 2012, products we have begun using within our organization  The initial install and management went without a hitch, in fact it went eerily too well.

The follow month however wasn't such smooth sailing, within a few days a number of SQL data sources belonging to two different protection groups began failing with "DPM could not run the backup/recovery job for the data source because it is owned by a different DPM server.". The error description went on to say the "Owner DPM Server: ." claiming "." owned the DPM job.

This was an unusual error to receive as there has only ever been a single DPM server within the organization  so the possibility of another DPM server owning the job was highly unlikely.

The problem in detail

The 5 data sources that were failing were all SharePoint data sources. We are using a Sharepoint 2010 Farm protection group (PG) and backing up any SQL resources that arn't covered in this PG with a simple SQL PG. "Sharepoint_Config" was one of the failing resources, as well as 4 SQL jobs "Application_Registry_Service", "Bdc_Service_DB", "Managed Metadata Service" and "PerformancePoint Service Application".

DPM complained that the "Replica is inconsistent" and attached the following detailed error description:

"The replica of SQL Server 2008 database SERVER\Application_Registry_Service on server.contoso.internal is inconsistent with the protected data source. All protection activies for data source will fail until the replica is synchronized with consistency check. You can recover data from existing recovery points, but new recovery points cannot be created until the replica is consistent.
For SharePoint farm, recovery points will continue getting created with the databases that are consistent. To backup inconsistent databases, run a consistency check on the farm. (ID 3106)
DPM could not run the backup/recovery job for the data source because it is owned by a different DPM server.
Data Source: SERVER\Application_Registry_Service
Owner DPM Server: . (ID 3184 Details: The file or directory is corrupted and unreadable (0x80070570))"

DPM suggests to take ownership, I attempted this, re-ran the consistency check and within 5 minutes received the errors messages again.

Our logs were also complaining of communication problems, which we initially put down to network issues, but this theory was quickly debunked as other data sources on the same server were successfully backing up.

FsmBlock.cs(178)        2DE6593E-B086-4002-9205-0A57B65BDC8E    WARNING    Backup.DeltaDataTransferLoop.CommonLoop : RAReadDatasetDelta, StatusReason = Error (StatusCode = -2147023671, ErrorCode = CommunicationProblem, workitem = 70aeaa93-b090-4a3c-bea0-c6fd1a1b4625)

01    TaskExecutor.cs(843)        2DE6593E-B086-4002-9205-0A57B65BDC8E    FATAL    Task stopped (state=Failed, error=RmAgentCommunicationError; -2147023671; WindowsHResult),

We also tried removing the Protect Group and re-adding it, checking SQL permissions, repairing the DPM agent, un-installing and re-installing the Agent, all of which failed.



The Solution

This was a very difficult one to troubleshoot, as SP1 was so new, no one had published details of experiencing similar problems.

Eventually we tracked down the issue to be ActiveOwner problems on the SQL server. The ActiveOwner files are located in "c:\program files\Microsoft Data Protection Manage\dpm\activeowner" on the server hosting the databases (SQL server being backed up). These ActiveOwner are used to manage ownership of databases, important for ensuring multiple DPM servers aren't attempting to backup/restore resources contemporaneously.

After opening the directory and locating the ActiveOwner files for the failing databases, we noticed they were all 0 KB, while healthy ActiveOwner files were 1 KB and contained the name of the owner DPM server.

1. Open  "c:\program files\Microsoft Data Protection Manage\dpm\activeowner" on the database server. 

2. Rename any 0 KB files to <name>.old 

3. Run SetDpmServer from ""c:\program files\Microsoft Data Protection Manage\dpm\bin"Syntax: SetDpmServer.exe -DpmServerName <SERVER>
Replace server with the computer name of your DPM server. 

4. Re-run your synchronization

This fix literally takes 3 minutes, yet it took us an entire week of investigation to come to this conclusion.

IBM AutoLoader TS2900 DPM 2012 setup tips

At the end of 2012 I received a new IBM TS2900 autoloader. I didn't have time to set it up, so for a couple of months it was being used as a glorified tape drive.

Last month I got around to configuring DPM 2012 to use the autoloader and hit a few speed bumps along the way. Hopefully the below information can help you setup your device quicker.

Just for reference my platform is Server 2008 R2 running DPM 2012 SP1 with Update Rollup 1.



I can't see the Autoloader in DPM or Device Manager

Hold on didn't I just buy an autoloader, so why can I only see the tape drive in device manager and DPM? This took me a while to work out, but its really easy to resolve.

The tape drive must be set to random mode, if you are using sequential mode your autoloader is essentially a manually controlled tape drive. As soon as you set random mode, the autoloader is presented to windows. You can set random mode by logging into your TS2900 front end and following the below steps.

1. Click on "Logical", located under the "Configure Library" menu.
2. Select "Random" from the "Library Mode:" drop down box.
3. Click submit.

If you are running sequential mode, you might see the below error messages when attempting to install the IBM driver. They are caused because random mode has not been selected and the autoloader is not being presented to windows, hence no device for the driver to install.

DBG:         install_exclusive.c, 1239: InstallVirtualBus: UpdateDriverForPlugAndPlayDevices failed - Update.
EXT: 0 -> -1: install_exclusive.c, 1249: InstallVirtualBus: status 0xe0000235.
DBG:         install_exclusive.c, 885: InstallVirtualBusByType: InstallVirtualBus failed

Program stopped prematurely due to error(s).If the debug flag was set, check debug.txt for details.

Driver Installion

Next you need to install the IBM driver from the IBM download centre. The one I am using at the time of writing this article is named "IBMTape.x64_w08_6233_WHQL_Cert.zip", but I have had success with non-WHQL versions aswell. There is a small tweak required to get the driver working with DPM, follow the below steps to install the driver correctly.

1. Extract the zip and install the driver by clicking on "install_exclusive.exe"
2. Wait for the driver installation to complete, then open up Control Panel > Device Manager
3. You should see the "IBM TotalStorage 3572 Tape Library" under "Medium Changer devices", this is your autoloader. If you don't see it, try uninstalling the driver, rebooting and re-installing the driver. Also ensure you followed the above steps to enable Random library mode on your autoloader.
4. Right click the "IBM ULTRIUM 5 HH 3580 TAPE DRIVE" (or your equivalent Tape driver), select "Update Driver Software".
5. Select "Browse my computer for driver software"
6. Select "Let me pick from a list of device drivers on my computer"
7. Select "LTO Tape Drive"
8. Install the driver and close device manager.

This process replaces the recently installed IBM LTO Tape driver with the default Microsoft driver. This is required as DPM (as of release 2012 SP1 Update Pack 1) doesn't work with the IBM provided tape driver. You still must install the IBM driver however, as the autoloader "IBM TotalStorage 3572 Tape Library" does require the IBM driver package to work.

Failure to replace the IBM drivers with the Microsoft drivers will result in the error message below.

The operation failed because of a protection agent failure. (ID 998 Details: The parameter is incorrect (0x80070057))

There you have it, a working TS2900 autoloader, be it with a few quirky work arounds, but since following the above steps mine hasn't missed a beat.

Spanning tapes with DPM 2010

Regardless of how great disk back is, most administrators will want to do some regular off-site bound backups to tape and often the data can be larger than a single tape. Enter tape spanning, the ability to span a single back job over multiple tapes. This isn't best practice, but if you have a single 2TB resource and only a maximum capacity of 1.6TB per tape (LTO4) you don't have many options. Either you can purchase an expensive new tape drive with higher capacity or enable spanning.

Initially I was a little taken back by the absense of any tape spanning options in the DPM 2010 GUI, in fact even the official manuals have very little reference to how tape spanning can be done.

After some digging I found that spanning will in fact kick in by default if the protection group is larger than a single tape, but DPM will only wait 1 hour for a replacement tape until the job fails. This is more than enough time for a tape loader, but if your using a manual LTO drive this could be a problem.



The Solution

The solution lies in a simple registry key.

“HKLM\Software\Microsoft\Microsoft Data Protection Manager\1.0\Prompting”

Under this key is a REG_DWORD value named "PromptingTimeOut"

The value is in milliseconds, so for each hour you want to wait, you need to multiply your value by 3600000.

For example, 4 hours * 3600000 = 14400000

Be sure you enter the value as a decimal or your wait time might be totally different than what you were hoping.