DPM 2012 SP1 replica inconsistent - datasource is owned by a different DPM server

Recently we took the leap of faith to DPM 2012 SP1 + Update Rollup 1. SP1 offers proper compatibility with SQL 2012 and Server 2012, products we have begun using within our organization  The initial install and management went without a hitch, in fact it went eerily too well.

The follow month however wasn't such smooth sailing, within a few days a number of SQL data sources belonging to two different protection groups began failing with "DPM could not run the backup/recovery job for the data source because it is owned by a different DPM server.". The error description went on to say the "Owner DPM Server: ." claiming "." owned the DPM job.

This was an unusual error to receive as there has only ever been a single DPM server within the organization  so the possibility of another DPM server owning the job was highly unlikely.

The problem in detail

The 5 data sources that were failing were all SharePoint data sources. We are using a Sharepoint 2010 Farm protection group (PG) and backing up any SQL resources that arn't covered in this PG with a simple SQL PG. "Sharepoint_Config" was one of the failing resources, as well as 4 SQL jobs "Application_Registry_Service", "Bdc_Service_DB", "Managed Metadata Service" and "PerformancePoint Service Application".

DPM complained that the "Replica is inconsistent" and attached the following detailed error description:

"The replica of SQL Server 2008 database SERVER\Application_Registry_Service on server.contoso.internal is inconsistent with the protected data source. All protection activies for data source will fail until the replica is synchronized with consistency check. You can recover data from existing recovery points, but new recovery points cannot be created until the replica is consistent.
For SharePoint farm, recovery points will continue getting created with the databases that are consistent. To backup inconsistent databases, run a consistency check on the farm. (ID 3106)
DPM could not run the backup/recovery job for the data source because it is owned by a different DPM server.
Data Source: SERVER\Application_Registry_Service
Owner DPM Server: . (ID 3184 Details: The file or directory is corrupted and unreadable (0x80070570))"

DPM suggests to take ownership, I attempted this, re-ran the consistency check and within 5 minutes received the errors messages again.

Our logs were also complaining of communication problems, which we initially put down to network issues, but this theory was quickly debunked as other data sources on the same server were successfully backing up.

FsmBlock.cs(178)        2DE6593E-B086-4002-9205-0A57B65BDC8E    WARNING    Backup.DeltaDataTransferLoop.CommonLoop : RAReadDatasetDelta, StatusReason = Error (StatusCode = -2147023671, ErrorCode = CommunicationProblem, workitem = 70aeaa93-b090-4a3c-bea0-c6fd1a1b4625)

01    TaskExecutor.cs(843)        2DE6593E-B086-4002-9205-0A57B65BDC8E    FATAL    Task stopped (state=Failed, error=RmAgentCommunicationError; -2147023671; WindowsHResult),

We also tried removing the Protect Group and re-adding it, checking SQL permissions, repairing the DPM agent, un-installing and re-installing the Agent, all of which failed.



The Solution

This was a very difficult one to troubleshoot, as SP1 was so new, no one had published details of experiencing similar problems.

Eventually we tracked down the issue to be ActiveOwner problems on the SQL server. The ActiveOwner files are located in "c:\program files\Microsoft Data Protection Manage\dpm\activeowner" on the server hosting the databases (SQL server being backed up). These ActiveOwner are used to manage ownership of databases, important for ensuring multiple DPM servers aren't attempting to backup/restore resources contemporaneously.

After opening the directory and locating the ActiveOwner files for the failing databases, we noticed they were all 0 KB, while healthy ActiveOwner files were 1 KB and contained the name of the owner DPM server.

1. Open  "c:\program files\Microsoft Data Protection Manage\dpm\activeowner" on the database server. 

2. Rename any 0 KB files to <name>.old 

3. Run SetDpmServer from ""c:\program files\Microsoft Data Protection Manage\dpm\bin"Syntax: SetDpmServer.exe -DpmServerName <SERVER>
Replace server with the computer name of your DPM server. 

4. Re-run your synchronization

This fix literally takes 3 minutes, yet it took us an entire week of investigation to come to this conclusion.

IBM AutoLoader TS2900 DPM 2012 setup tips

At the end of 2012 I received a new IBM TS2900 autoloader. I didn't have time to set it up, so for a couple of months it was being used as a glorified tape drive.

Last month I got around to configuring DPM 2012 to use the autoloader and hit a few speed bumps along the way. Hopefully the below information can help you setup your device quicker.

Just for reference my platform is Server 2008 R2 running DPM 2012 SP1 with Update Rollup 1.



I can't see the Autoloader in DPM or Device Manager

Hold on didn't I just buy an autoloader, so why can I only see the tape drive in device manager and DPM? This took me a while to work out, but its really easy to resolve.

The tape drive must be set to random mode, if you are using sequential mode your autoloader is essentially a manually controlled tape drive. As soon as you set random mode, the autoloader is presented to windows. You can set random mode by logging into your TS2900 front end and following the below steps.

1. Click on "Logical", located under the "Configure Library" menu.
2. Select "Random" from the "Library Mode:" drop down box.
3. Click submit.

If you are running sequential mode, you might see the below error messages when attempting to install the IBM driver. They are caused because random mode has not been selected and the autoloader is not being presented to windows, hence no device for the driver to install.

DBG:         install_exclusive.c, 1239: InstallVirtualBus: UpdateDriverForPlugAndPlayDevices failed - Update.
EXT: 0 -> -1: install_exclusive.c, 1249: InstallVirtualBus: status 0xe0000235.
DBG:         install_exclusive.c, 885: InstallVirtualBusByType: InstallVirtualBus failed

Program stopped prematurely due to error(s).If the debug flag was set, check debug.txt for details.

Driver Installion

Next you need to install the IBM driver from the IBM download centre. The one I am using at the time of writing this article is named "IBMTape.x64_w08_6233_WHQL_Cert.zip", but I have had success with non-WHQL versions aswell. There is a small tweak required to get the driver working with DPM, follow the below steps to install the driver correctly.

1. Extract the zip and install the driver by clicking on "install_exclusive.exe"
2. Wait for the driver installation to complete, then open up Control Panel > Device Manager
3. You should see the "IBM TotalStorage 3572 Tape Library" under "Medium Changer devices", this is your autoloader. If you don't see it, try uninstalling the driver, rebooting and re-installing the driver. Also ensure you followed the above steps to enable Random library mode on your autoloader.
4. Right click the "IBM ULTRIUM 5 HH 3580 TAPE DRIVE" (or your equivalent Tape driver), select "Update Driver Software".
5. Select "Browse my computer for driver software"
6. Select "Let me pick from a list of device drivers on my computer"
7. Select "LTO Tape Drive"
8. Install the driver and close device manager.

This process replaces the recently installed IBM LTO Tape driver with the default Microsoft driver. This is required as DPM (as of release 2012 SP1 Update Pack 1) doesn't work with the IBM provided tape driver. You still must install the IBM driver however, as the autoloader "IBM TotalStorage 3572 Tape Library" does require the IBM driver package to work.

Failure to replace the IBM drivers with the Microsoft drivers will result in the error message below.

The operation failed because of a protection agent failure. (ID 998 Details: The parameter is incorrect (0x80070057))

There you have it, a working TS2900 autoloader, be it with a few quirky work arounds, but since following the above steps mine hasn't missed a beat.

Spanning tapes with DPM 2010

Regardless of how great disk back is, most administrators will want to do some regular off-site bound backups to tape and often the data can be larger than a single tape. Enter tape spanning, the ability to span a single back job over multiple tapes. This isn't best practice, but if you have a single 2TB resource and only a maximum capacity of 1.6TB per tape (LTO4) you don't have many options. Either you can purchase an expensive new tape drive with higher capacity or enable spanning.

Initially I was a little taken back by the absense of any tape spanning options in the DPM 2010 GUI, in fact even the official manuals have very little reference to how tape spanning can be done.

After some digging I found that spanning will in fact kick in by default if the protection group is larger than a single tape, but DPM will only wait 1 hour for a replacement tape until the job fails. This is more than enough time for a tape loader, but if your using a manual LTO drive this could be a problem.



The Solution

The solution lies in a simple registry key.

“HKLM\Software\Microsoft\Microsoft Data Protection Manager\1.0\Prompting”

Under this key is a REG_DWORD value named "PromptingTimeOut"

The value is in milliseconds, so for each hour you want to wait, you need to multiply your value by 3600000.

For example, 4 hours * 3600000 = 14400000

Be sure you enter the value as a decimal or your wait time might be totally different than what you were hoping.