DPM 2012 R2 only allows offline backups of VM with updated integration tools

All DPM admins that backup Hyper-V VMs have seen the dreaded offline backup state before. Admins of earlier versions of DPM might know this as "Saved State", either way its annoying and disruptive to backup VMs in an offline/saved state.

Recently when we did a migration of VMs from one Hyper-V cluster to another with SCVMM, we came across an issue where most of the VMs appeared as "Offline backup" in the DPM console.

This can be caused by a multitude of issues, including not having a SCSI adapter connected to the VM and running an old version of the integration tools.

However after checking all the requirements (listed below), we were still only able to capture offline backups on some VMs.

• The Backup (Volume Snapshot) Integration Service is disabled or not installed.
• A virtual machine has one or more dynamic disks.
• A virtual machine has one or more volumes that are based on non-NTFS file systems.
• In a cluster configuration, the virtual machine Cluster Resource Group is offline.
• A virtual machine is not in a running state.
• A Shadow Storage assignment of a volume inside the virtual machine is explicitly set to a different volume other than itself.

In testing we removed a VHD from a VM that had multiple VHD connected and then tested it, this worked! Funnily enough, when we re-added the VHD, the VM still could be backed up online. This led us to a fix that worked on most of our problematic VMs.

The Resolution

1. Open up Hyper-V or SCVMM.

2. Go into the hard drive settings of the problematic VM and remove all VHD (not delete, just remove for now - do I even need to say that?).

3. Save the VM.

4. Re-add the VHD and save the VM again.

5. Start the VM

6. Attempt to backup with DPM again, you may need to clear the state cache in DPM

Essentially we are doing nothing, removing and re-adding a configuration. However this seems to resolve the issue on most VM. This may or may not be similar in your environment, as these VM had all been migrated between Hyper-V servers with VMM.

Microsoft Failover Cluster CSV Volume Disappear

We recently began experiencing some problems with the 3rd member of our Windows Failover Cluster. Our cluster consists of 3 servers running 2008 R2 SP1, running Failover cluster manager with a SAN backend. This SAN presents a number of Cluster Shared Volumes (CSV) to the servers, all of our data sits on these CSV's.

One afternoon our primary CSV went into redirected mode, this is a normal occurrence during backup operations, but no backup was scheduled and we were not able to turn off redirected mode. We had to schedule a short outage, fully power off the Hyper-V hosts and power back on. After a full investigation turned up nothing we put this down to an anomaly. Another 3 weeks later and the problem happened again, this time we scheduled a longer outage so we could investigate the problem more thoroughly.

During testing we discovered that one of the 3 hosts was causing the issue, when it was removed from the cluster, no problems, when it was in the cluster the CSV in question would randomly go into redirected mode. Logs of the SAN and Hyper-V hosts turned up nothing and all the cluster tests passed perfectly.

Unfortunately during our testing, we encountered a bigger problem. When bringing the faulty host back online for the 3rd time, the CSV itself disappeared on the 2 healthy hosts, the CSV was still visible on the 3rd host. We promptly removed the 3rd host from the cluster but the CSV did not reappear on the 2 healthy hosts.

What didn't work

We tried a number of processes to get the volume to re-appear.

• Rescanning/refreshing in Disk Manager
• Deleting and re-adding the CSV
• Repairing the CSV
• Restarting the Hyper-V hosts
• Removing the faulty host from the SAN LUN zones

At this point we were a little worried, our primary CSV was displayed in windows as an empty disk (as above)  With the Failover Cluster tools we checked out the DiskSignature and we were greeted with a grim 0. 

Command: cluster resource VMs /priv

D  VMs                  DiskSignature                  0 (0x0)

Scanning the FailoverClustering event logs we turned up the following events:

Event ID: 1568

Source: FailoverClustering

Cluster physical disk resource 'VMs' cannot be brought online because the associated disk could not be found. The expected signature of the disk was 'F62FC592'. If the disk was replaced or restored, in the Failover Cluster Manager snap-in, you can use the Repair function (in the properties sheet for the disk) to repair the new or restored disk. If the disk will not be replaced, delete the associated disk resource.

and

Event ID: 1568

Source: FailoverClustering

Cluster disk resource 'VMs' found the disk identifier to be stale. This may be expected if a restore operation was just performed or if this cluster uses replicated storage. The DiskSignature or DiskUniqueIds property for the disk resource has been corrected.

This was repeated over and over, the Cluster was trying to repair the problem but not having any success.

The Solution

After reading this thread we noticed in the last post a user mentioned "a Microsoft tech fixed the problem, the disk first sector was corrupted" We decided a partition table scan and re-write were worth a shot.

Using testdisk I was able to successfully recover the volume by first analyzing the disk for partitions then writing the changes.

I then re-wrote the disk signature (which I found in the FailoverClustering logs, as per above) to the volume using the below command.

CLUSTER RESOURCE VMs DiskSignature F62FC592

The volume then successfully came online, phew and all within my outage window!

Server 2008 R2 Standard and Hyper-V dynamic memory management not working

This isn't exactly a blog post but just a note to those having problems with dynamic memory management with Server 2008 R2 standard hosts.

Microsoft were kind enough to include the great new dynamic memory management in Hyper-V, which is capable of balancing VM (virtual machine) memory allocation dynamically, and it does so quite well. Unfortunately they failed to mention one specific requirement for this to work properly, all Server 2008 R2 standard hosts require SP1 to be installed on the VM itself.

So for 2008 R2 Standard hosts you must:

1. Upgrade the hypervisor (server 2008 r2 Hyper-V) server to service pack 1, reboot.
2. Upgrade the integration services in the VM, reboot.
3. Upgrade the VM to service pack 1, reboot.

This is particularly unusual because Windows Vista, Server 2003 R2 with SP2 and Server 2008 only need the integration services upgraded, think you might have dropped the ball briefly on that one Microsoft.

Hyper-V 2008 R2 SP1 and Clickview incompatibility

With the introduction of the dynamic memory management features in the new SP1 release for Server 2008 R2 it is an exciting platform for desktop virtualization. I have been waiting for its release to deploy my virtual desktops in our organization, another key reason for my Hyper-V choice is the amazing pricing that Microsoft offers to education facitilies.

With this is mind I built up my IBM x3650 servers, 110GB of ram each, Xendesktop 5 setup in conjunction with PVS 5.6 FP1 and set to deliver my desktops on Hyper-V.

At the time there was no citrix tools available for building multiple desktops on Hyper-V so I wrote a small powershell script to facilitate this.

I provisioned my two servers, setup 30 test virtual desktops and begun my testing. As per best pratice I used an emulated network adapter to PXE boot my PVS image, then a synthetic network adapter for high network performance once in windows (yes the synthetic network adapter automatically takes preference over the emulated adapter in Windows 7).

My testing was going brilliant and users were saying their desktop experience was faster on this VDI than even the fastest machines in our environment. That makes sense as every modern systems bottleneck is the disk speed and I was using 10 disk RAID-10 arrays on my virtual hosts to provide disk cache for the VDI, not to mention considerable caching on the raid controllers of my SAN that hosts the PVS image.

Things went pair shape when users started using Clickview to view videos, the video was slightly choppy at full screen, but fine in a window. This was only a side issue, as users started getting disconnected from their VDI sessions, users described this as the system freezing. Upon investigating I found that clickview was causing the systems to BSOD, thats strange, why is that occuring? The BSOD messaged were relating to the synthetic Hyper-V NIC driver.

I went back to the drawing board and test the emulated network driver and after a few days testing never had any BSOD, but the performance was horrible. Video was very choppy and even in a small window it the framerate was extremely slow and unacceptable.

I tried a number of different Hyper-V drivers, the most recent, some older versions, I tried without 2008 R2 SP1, no luck. Getting desperate, I reprovisioned one of my servers as Citrix Xenserver 5.6, and retested. Not only did the BSOD go away, but I could now view video at full screen with perfect FPS, no choppy video or audio, no desyncronization, perfect.

For now, it looks like if you want to deliver Clickview on your environment, Hyper-V SP1 is not an option for you...