DPM 2012 R2 failing to backup SQL 2012 SP1 Always On Cluster VM

After we created a new Server 2012 R2 cluster and migrated our VMs across, the next job was to get backup in place with DPM 2012 R2.

Most of the machines backed up with Online backup without issue and our SQL cluster was no exception, the online backup initiated without fault. However after some time one of the Always On cluster came back with the following error.

The VSS application writer or the VSS provider is in a bad state. Either it was already in a bad state or it entered a bad state during the current operation. (ID 30111 Details: VssError:The writer experienced a non-transient error.  If the backup process is retried,
the error is likely to reoccur.
 (0x800423F4))

Digging further on the SQL machine with the "vssadmin list writers" command, we found that the SQL VSS Writer service was in an error state.



The Solution

The solution is very simple but introduces some other problems.

1. Jump on the erroneous SQL server.
2. Open a command prompt and type services.msc
3. Go the SQL VSS Writer service and stop the service.
4. Re-run the DPM job.

This successfully completes the VM snapshot backup, but it introduces an issue with SQL backup if you are also using DPM for SQL backup on the same Always On cluster.

In a SQL Always On cluster, the DPM backups are by normally taken from the secondary node. The primary node is left alone to do SQL, while the inactive secondary takes the heavy backup load.

This is why we only had the problem on our secondary node, it is also why it introduces an SQL backup problem. When the SQL VSS Writer service is stopped, DPM can't perform SQL backups on the server.

Two possible options are:

1. Don't perform VM snapshot on your backup node, it may be overkill anyway. In the event of a failure you can spin up a new VM and make it the new secondary.

2. Run a schedule that disables the SQL VSS Writer service some time before the VM snapshot and re-enable it again after the snapshot.

We are using option 2 and so far it is working well. You need to ensure that your SQL backup isn't occurring during your SQL VSS writer disabled period, but your a good admin and wouldn't have your VM snapshot and SQL backup scheduled at the same time anyway!

SCOM 2012 OleDB Module 0x80004005 errors after SQL database move

After migrating our SCOM 2012 R2 DB and Data Warehouse DB to a new SQL server we were receiving SCOM alerts that there was a problem with the OleDB module.


The Problem

The initial alert indicated that there was a login problem. This prompted us to check, re-check and triple-check all the SQL logins and permissions between the old and new SQL servers.

Alert description: OleDb Module encountered a failure 0x80004005 during execution and will post it as output data item. Unspecified error: Cannot open database "OperationsManager2012" requested by the login. The login failed.
Workflow name: Microsoft.SystemCenter.SqlBrokerAvailabilityMonitorForPool

After not having much luck we eventually decommissioned the old SQL server. Once the old SQL server was turned off, the alert changed from a login failed to a "SQL server does not exist". This error got us thinking, maybe its not a permission problem but some parts of SCOM may have been still pointing at the old SQL server.

Alert description: OleDb Module encountered a failure 0x80004005 during execution and will post it as output data item. Unspecified error: [DBNETLIB][ConnectionOpen (Connect()).]SQL Server does not exist or access denied.
Workflow name: Microsoft.SystemCenter.SqlBrokerAvailabilityMonitorForPool

The Solution 

A search of the registry found a few keys that were undocumented in the SQL migration document I was reading. These keys are:

HKLM\SOFTWARE\Microsoft\Microsoft Operations Manager\3.0\Setup\DatabaseServerName
HKLM\SOFTWARE\Microsoft\Microsoft Operations Manager\3.0\Setup\DataWarehouseDBServerName
HKLM\Software\Microsoft\System Center\SetupBackup\Blue\Database\DatabaseServerName

After changing these keys and restarting the SCOM server all was well!

Sharepoint Config SQL logs growing out of control

Today I woke up, bags packed, ready to go to Taiwan for Computex, but I also found a notification from SCOM telling me the SQL box had nearly filled up its harddrive.

Upon investigating I found the Sharepoint Configuration Log has grown to over 6GB in only a few short months, HUH? Why was this happening? I only have a couple of Sharepoint sites and they arn't even heavily used and I do daily SQL backups via Microsoft DPM 2010, shouldn't the logs be truncated at this point?

Actually no, DPM 2010 won't truncated the logs because the Sharepoint databases are set to full recovery mode by default, which is critical for log shipping and mirroring funtionality that you may need with a large Sharepoint farm deployment. I only have a very small Sharepoint deployment but I still don't want to disable the full recovery functionality at this stage.

Instead I have scripted a weekly log truncation via the following SQL Query script.


SHAREPOINT CONFIGURATION LOG TRUNCATION

USE Sharepoint_Config

GO

            DBCC SHRINKFILE(Sharepoint_Config_log, 1)

    BACKUP LOG Sharepoint_Config WITH TRUNCATE_ONLY

            DBCC SHRINKFILE(Sharepoint_Config_log, 1)

This took a LONG time for my 6GB log, but the result was beautiful, huge reduction on my logs and SCOM notification dismissing themselves. Of course you may want or be required to keep your logs in your organiziation. If you have this requirement you can simply export the log before you truncate it.


SHAREPOINT CONFIGURATION LOG BACKUP

BACKUP LOG [Sharepoint_Config] TO DISK=’D:configLogBackup.bak’



Hopefully one is all resolved now!