DPM 2010 fails to backup virtual guests on a Hyper-V Cluster Shared Volume using hardware VSS provider

Question

We have been battling with this problem for 2 years now and have raised calls with Microsoft and EMC both of which have not resulted in a resolution.
We have an 8 node CSV with about 40 virtual servers on it.  The Hyper-V hosts are Windows 2008 R2 SP1 servers with the Hyper-V role installed and the SAN is a Clarion CX3-10c and we are using the EMC 4.7.1 hardware provider for snaphots. The problem
we have is that snapshots are not 'always' getting created on the SAN and the recovery points in DPM fail.  It often takes several attempts to re-run the job before it successfully works.
The MaxParallelBackups registry key is set to 1 on the DPM server so we are running jobs serially on each node and we have aligned all the VMs so the Hyper-V owner of each VM is the same owner of the CSV that those VMs have their resources on. We have done
this to avoid any ownership changes to reduce the risk of failures.  This has had some success and it is not always the same servers that fail which gives this problem an unfortunate degree on intermittency!
There is no DataSourceGroups.xml anymore after I have tried to implement previously (it's effectiveness unproven though). We get a few days grace sometimes where backups are running with no problems but the longest run of success we have had is about 2-3
weeks which is our most recent run and it's cessation has led me to post this on the forum. Since Wednesday last week we have had regular failures with between 9 - 12 failures each night.  The annoying part of this is that we hadn't made any changes to
either DPM or the Cluster which makes no real sense.
One thing I could do is split the Protection Groups so all the VMs on Hyper-V host1 are in one group, all the ones on Hyper-V host2 in another and so forth, however if anyone could advise me if this will be any help or not before I do it that would be much
appreciated as I don't want to do this unless I have to in case I hit capacity problems (most VMs are in one large protection group at present)
Any help would be much appreciated (I have logs I can attach if anyone would like to see a sample of when the failures occur),
Thanks
Chris

Answer 1

Hi,
I'm not on the Windows team, so don't know about the inner workings of the fix and what to expect.  However, you can try using diskshadow.exe to delete all the snapshots for the given volume and see if that clears them up.
DISKSHADOW> delete shadows /?
DELETE SHADOWS { ALL | VOLUME <volume> | OLDEST <volume> | SET <setID> | ID <shadowID> | EXPOSED <drive letter, mountPoint or share> }
        Delete shadow copies, both persistent and non-persistent.
        ALL                     All shadow copies.
        VOLUME <volume>         Delete all shadow copies of the given volume.
        OLDEST <volume>         Delete the oldest shadow copy of the given volume.
        SET <setID>             Delete the shadow copies in the shadow copy set specified by the setId parameter.
        ID <shadowID>           Delete the shadow copy specified by the shadowId parameter.
        EXPOSED <exposeName>    Delete the shadow copy that is exposed at the specified drive letter, mount point or share.
        Examples: DELETE SHADOWS ALL
                  DELETE SHADOWS EXPOSED p:
                  DELETE SHADOWS EXPOSED ShareName
So something like:  DELETE SHADOWS VOLUME E:   or  DELETE SHADOWS ALL
Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT]
This posting is provided "AS IS" with no warranties, and confers no rights.