Hyper-V cluster Backup causes virtual machine reboots for common Cluster Shared Volumes members.

Question

I am having a problem where my VMs are rebooting while other VMs that share the same CSV are being backed up. I have provided all the information that I have gather to this point below. If I have missed anything, please let me know.
My HyperV Cluster configuration:
5 Node Cluster running 2008R2 Core DataCenter w/SP1. All updates as released by WSUS that will install on a Core installation
Each Node has 8 NICs configured as follows:
 NIC1 - Management/Campus access (26.x VLAN)
 NIC2 - iSCSI dedicated (22.x VLAN)
 NIC3 - Live Migration (28.x VLAN)
 NIC4 - Heartbeat (20.x VLAN)
 NIC5 - VSwitch (26.x VLAN)
 NIC6 - VSwitch (18.x VLAN)
 NIC7 - VSwitch (27.x VLAN)
 NIC8 - VSwitch (22.x VLAN)
Following hotfixes additional installed by MS guidance (either while build or when troubleshooting stability issue in Jan 2013)
 KB2531907 - Was installed during original building of cluster
 KB2705759 - Installed during troubleshooting in early Jan2013
 KB2684681 - Installed during troubleshooting in early Jan2013
 KB2685891 - Installed during troubleshooting in early Jan2013
 KB2639032 - Installed during troubleshooting in early Jan2013
Original cluster build was two hosts with quorum drive. Initial two hosts were HST1 and HST5
Next host added was HST3, then HST6 and finally HST2.
NOTE: HST4 hardware was used in different project and HST6 will eventually become HST4
Validation of cluster comes with warning for following things:
 Updates inconsistent across hosts
  I have tried to manually install "missing" updates and they were not applicable
  Most likely cause is different build times for each machine in cluster
   HST1 and HST5 are both the same level because they were built at same time
   HST3 was not rebuilt from scratch due to time constraints and it actually goes back to Pre-SP1 and has a larger list of updates that others are lacking and hence the inconsistency
   HST6 was built from scratch but has more updates missing than 1 or 5 (10 missing instead of 7)
   HST2 was most recently built and it has the most missing updates (15)
 Storage - List Potential Cluster Disks
  It says there are Persistent Reservations on all 14 of my CSV volumes and thinks they are from another cluster.
  They are removed from the validation set for this reason. These iSCSI volumes/disks were all created new for
  this cluster and have never been a part of any other cluster.
 When I run the Cluster Validation wizard, I get a slew of Event ID 5120 from FailoverClustering. Wording of error:
  Cluster Shared Volume 'Volume12' ('Cluster Disk 13') is no longer available on this node because of
  'STATUS_MEDIA_WRITE_PROTECTED(c00000a2)'. All I/O will temporarily be queued until a path to the
  volume is reestablished.
 Under Storage and Cluster Shared VOlumes in Failover Cluster Manager, all disks show online and there is no negative effect of the errors.
Cluster Shared Volumes
 We have 14 CSVs that are all iSCSI attached to all 5 hosts. They are housed on an HP P4500G2 (LeftHand) SAN.
 I have limited the number of VMs to no more than 7 per CSV as per best practices documentation from HP/Lefthand
 VMs in each CSV are spread out amonst all 5 hosts (as you would expect)
Backup software we use is BackupChain from BackupChain.com.
Problem we are having:
 When backup kicks off for a VM, all VMs on same CSV reboot without warning. This normally happens within seconds of the backup starting
What have to done to troubleshoot this:
 We have tried rebalancing our backups
  Originally, I had backup jobs scheduled to kick off on Friday or Saturday evening after 9pm
  2 or 3 hosts would be backing up VMs (Serially; one VM per host at a time) each night.
  I changed my backup scheduled so that of my 90 VMs, only one per CSV is backing up at the same time
   I mapped out my Hosts and CSVs and scheduled my backups to run on week nights where each night, there
   is only one VM backed up per CSV. All VMs can be backed up over 5 nights (there are some VMs that don't
   get backed up). I also staggered the start times for each Host so that only one Host would be starting
   in the same timeframe. There was some overlap for Hosts that had backups that ran longer than 1 hour.
  Testing this new schedule did not fix my problem. It only made it more clear. As each backup timeframe
  started, whichever CSV the first VM to start was on would have all of their VMs reboot and come back up.
 I then thought maybe I was overloading the network still so I decided to disable all of the scheduled backup
 and run it manually. Kicking off a backup on a single VM, in most cases, will cause the reboot of common
 CSV members.
 Ok, maybe there is something wrong with my backup software.
  Downloaded a Demo of Veeam and installed it onto my cluster.
  Did a test backup of one VM and I had not problems.
  Did a test backup of a second VM and I had the same problem. All VMs on same CSV rebooted
 Ok, it is not my backup software. Apparently it is VSS. I have looked through various websites. The best troubleshooting
 site I have found for VSS in one place it on BackupChain.com (http://backupchain.com/hyper-v-backup/Troubleshooting.html)
 I have tested almost every process on there list and I will lay out results below:
  1. I have rebooted HST6 and problems still persist
  2. When I run VSSADMIN delete shadows /all, I have no shadows to delete on any of my 5 nodes
   When I run VSSADMIN list writers, I have no error messages on any writers on any node...
  3. When I check the listed registry key, I only have the build in MS VSS writer listed (I am using software VSS)
  4. When I run VSSADMIN Resize ShadowStorge command, there is no shadow storage on any node
  5. I have completed the registration and service cycling on HST6 as laid out here and most of the stuff "errors"
   Only a few of the DLL's actually register.
  6. HyperV Integration Services were reconciled when I worked with MS in early January and I have no indication of
   further issue here.
  7. I did not complete the step to delete the Subscriptions because, again, I have no error messages when I list writers
  8. I removed the Veeam software that I had installed to test (it hadn't added any VSS Writer anyway though)
  9. I can't realistically uninstall my HyperV and test VSS
  10. Already have latest SPs and Updates
  11. This is part of step 5 so I already did this. This seems to be a rehash of various other stratgies
 I have used the VSS Troubleshooter that is part of BackupChain (Ctrl-T) and I get the following error:
  ERROR: Selected writer 'Microsoft Hyper-V VSS Writer' is in failed state!
  - Status: 8 (VSS_WS_FAILED_AT_PREPARE_SNAPSHOT)
  - Writer Failure code: 0x800423f0 (<Unknown error code>)
  - Writer ID: {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}
  - Instance ID: {d55b6934-1c8d-46ab-a43f-4f997f18dc71}
  VSS snapshot creation failed with result: 8000FFFF
VSS errors in event viewer. Below are representative errors I have received from various Nodes of my cluster:
I have various of the below spread out over all hosts except for HST6
Source: VolSnap, Event ID 10, The shadow copy of volume took too long to install
Source: VolSnap, Event ID 16, The shadow copies of volume x were aborted because volume y, which contains shadow copy storage for this shadow copy, wa force dismounted.
Source: VolSnap, Event ID 27, The shadow copies of volume x were aborted during detection because a critical control file could not be opened.
I only have one instance of each of these and both of the below are from HST3
Source: VSS, Event ID 12293, Volume Shadow Copy Service error: Error calling a routine on a Shadow Copy Provider {b5946137-7b9f-4925-af80-51abd60b20d5}. Routine details RevertToSnashot [hr = 0x80042302, A Volume Shadow Copy Service component encountered an
unexpected error.
Source: VSS, Event ID 8193, Volume Shadow Copy Service error: Unexpected error calling routine GetOverlappedResult.  hr = 0x80070057, The parameter is incorrect.
So, basically, everything I have tried has resulted in no success towards solving this problem.
I would appreciate anything assistance that can be provided.
Thanks,
Charles J. Palmer
Wright Flood

Answer 1

Tim,
Thanks for the reply. I ran the first two commands and got this:
Name                                                            
Role Metric
Cluster Network 1                                              
3  10000
Cluster Network 2 - HeartBeat                              1   1300
Cluster Network 3 - iSCSI                                    0  10100
Cluster Network 4 - LiveMigration                         1   1200
When you look at the properties of each network, this is how I have it configured:
Cluster Network 1 - Allow cluster network communications on this network and Allow clients to connect through this network (26.x subnet)
Cluster Network 2 - Allow cluster network communications on this network. New network added while working with Microsoft support last month. (28.x subnet)
Cluster Network 3 - Do not allow cluster network communications on this network. (22.x subnet)
Cluster Network 4 - Allow cluster network communications on this network. Existing but not configured to be used by VMs for Live Migration until MS corrected. (20.x subnet)
Should I modify my metrics further or are the current values sufficient.
I worked with an MS support rep because my cluster (once I added the 5th host) stopped being able to live migrate VMs and I had VMs host jumping on startup. It was a mess for a couple of days. They had me add the Heartbeat network as part of the solution
to my problem. There doesn't seem to be anywhere to configure a network specifically for CSV so I would assume it would use (based on my metrics above) Cluster Network 4 and then Cluster Network 2 for CSV communications and would fail back to the Cluster Network
1 if both 2 and 4 were down/inaccessible.
As to the iSCSI getting a second NIC, I would love to but management wants separation of our VMs by subnet and role and hence why I need the 4 VSwitch NICs. I would have to look at adding an additional quad port NIC to my servers and I would be having to
use half height cards for 2 of my 5 servers for that to work.
But, on that note, it doesn't appear to actually be a bandwidth issue. I can run a backup for a single VM and get nothing on the network card (It caused the reboots before any real data has even started to pass apparently) and still the problem occurs.
As to Backup Chain, I have been working with the vendor and they are telling my the issue is with VSS. They also say they support CSV as well. If you go to this page (http://backupchain.com/Hyper-V-Backup-Software.html)
they say they support CSVs. Their tech support has been very helpful but unfortunately, nothing has fixed the problem.
What is annoying is that every backup doesn't cause a problem. I have a daily backup of one of our machines that runs fine without initiating any additional reboots. But most every other backup job will trigger the VMs on the common CSV to reboot.
I understood about the updates but I had to "prove" it to the MS tech I was on the phone with and hence I brought it up. I understand on the storage as well. Why give a warning for something that is working though... I think that is just a poor indicator
that it doesn't explain that in the report.
At a loss for what else I can do,
Charles J. Palmer