Cluster shared volume disappear... STATUS_MEDIA_WRITE_PROTECTED(c00000a2)

Hi all, I am having an issue hopefully someone can help me with. I have recently inherited a 2 node cluster,
both nodes are one half of an ASUS
RS702D-E6/PS8 so both nodes should be near identical. They are both running Hyper-V Server 2008 R2 hosting some 14 VM's.
Each node is hooked up via cat5e to a Promise
VessRAID 1830i via iSCSI using one of the servers onboard NICs each, whose cluster network is setup as Disabled for cluster use (the way I think it is supposed to be not the way I had originally inherited it) on it's own Class A Subnet and on it's own
private physical switch...
The SAN hosts a 30GB CSV Witness Disk and 2 2TB CSV Volumes, one for each node labeled Volume1 and Volume2. Some VHD's on each.
The Cluster Clients connect to the rest of the company via the Virtual ExternalNIC adapters created in Hyper-V manager but physically are off of Intel ET Dual Gigabit adapters
wired into our main core switch which is set up with class c subnets.
I also have a crossover cable wired up running to the other ports on the Intel ET Dual Port NICs using yet a third Class B Subnet and is configured in the Failover Cluster
Manger as internal so there are 3 ipv4 Cluster networks total.
Even though the cluster passes the validation tests with flying colors I am not convinced all is well. With Hyperv1 or node 1,
I can move the CSV's and machines over to hyperv2 or node 2, stop the cluster service on 1 and perform maintenance such as a reboot or install patches if needed. When it reboots or I restart the cluster service to bring it back online,
it is well behaved leaving hyperv2 the owner of all 3 CSV's Witness, Volume 1 and 2. I can then pass them back or split them up any which way and at no point is cluster service interrupted or noticed by users, duh I know this is how it is SUPPOSED to work
but...
if I try the same thing with Node 2, that is move the witness and volumes to node 1 as owner and migrate all VM's over, stop cluster service on node 2, do whatever I have
to do and reboot, as soon as node 2 tries to go back online, it tries to snatch volume 2 back, but it never succeeds and then the following error is logged in cluster event log:
Hyperv1
Event ID: 5120
Source: Microsoft-Windows-FailoverClustering
Task Category: Cluster Shared Volume
The listed message is:
Cluster Shared Volume 'Volume2' ('HyperV1 Disk') is no longer available on this node because of 'STATUS_MEDIA_WRITE_PROTECTED(c00000a2)'. All I/O will temporarily be queued until
a path to the volume is reestablished.
Followed 4 seconds later by:
Hyperv1
event ID: 1069
Source: Microsoft-Windows-FailoverClustering
Task Catagory: Resource Control Manager
Message: Cluster Resource 'Hyperv1 Disk in clustered service or application '75d88aa3-8ecf-47c7-98e7-6099e56a097d'
failed.
- AND -
2 of the following:
Hyperv1
event ID: 1038
Source: Microsoft-Windows-FailoverClustering
Task Catagory: Physical Disk Resource
Message: Ownership of cluster disk 'HyperV1 Disk' has been unexpectedly lost by this node. Run the Validate
a Configuration wizard to check your storage configuration.
Followed 1 second later by another 1069 and then various machines are failing messages.
If you browse to
\\hyperv-1\c$\clusterstorage\ or
\\hyperv-2\c$\Clusterstorage\, Volume 2 is indeed missing!!
This has caused me to panic a few times as the first time I saw this I thought everything was lost but I can get it back by stopping the service on node 1 or shutting it
down, restarting node 2 or the service on node 2 and waiting forever for the disk to list as failed and then shortly thereafter it comes back online. I can then boot node 1 back up and let it start servicing the cluster again. It doesn’t pull the same
craziness node 2 does when it comes online; it leaves all ownership with 2 unless I tell I to move.
I am very new to clusters and all I know at this point is this is pretty cool stuff but basically if it is running don’t mess with it is the attitude I have taken
with it but there is a significant amount of money tied up in this hardware and we should be able to leverage this as needed, not wonder if it is going to act up again.
To me it seems for a ‘failover’ cluster it should be way more robust than this...
I can go into way more detail if needed but I didn’t see any other posts on this specific issue no matter what forum I scoured. I’m obviously looking for advice
on how to get this resolved as well as advice on whether or not I wired the cluster networks correctly. I am also not sure about what protocols are bound to what nics anymore and what the binding order should be, could this be what is causing my issue?
I have NVSPBIND and NVSPSCRUB on both boxes if needed.
Thanks!
-LW

Hello Ravikumar,
Thanks about your attention!
All disks are Online, see below the status of disks, but the problem continues, any ideas?
PS.: For your information, all disks are delivered to hosts by SAN/HBA and all tests from Cluster Validation are passed.
PS C:\Users\hyperv_admin> Get-ClusterSharedVolume
Name State Node
hyperv-04_vol1_fc
Online vmserver27
hyperv-04_vol2_fc
Online vmserver26
hyperv-04_vol3_sata
Online vmserver25
hyperv-04_vol4_sata
Online vmserver27
See below the patches applied on my hosts:
KB2263829 http://support.microsoft.com/?kbid=2263829
KB2425227 http://support.microsoft.com/?kbid=2425227
KB2484033 http://support.microsoft.com/?kbid=2484033
KB2488113 http://support.microsoft.com/?kbid=2488113
KB2492386 http://support.microsoft.com/?kbid=2492386
KB2494016 http://support.microsoft.com/?kbid=2494016
KB2494162 http://support.microsoft.com/?kbid=2494162
KB2505438 http://support.microsoft.com/?kbid=2505438
KB2506014 http://support.microsoft.com/?kbid=2506014
KB2506212 http://support.microsoft.com/?kbid=2506212
KB2506928 http://support.microsoft.com/?kbid=2506928
KB2507618 http://support.microsoft.com/?kbid=2507618
KB2509553 http://support.microsoft.com/?kbid=2509553
KB2510531 http://support.microsoft.com/?kbid=2510531
KB2511250 http://support.microsoft.com/?kbid=2511250
KB2511455 http://support.microsoft.com/?kbid=2511455
KB2512715 http://support.microsoft.com/?kbid=2512715
KB2515325 http://support.microsoft.com/?kbid=2515325
KB2518869 http://support.microsoft.com/?kbid=2518869
KB2520235 http://support.microsoft.com/?kbid=2520235
KB2522422 http://support.microsoft.com/?kbid=2522422
KB2525835 http://support.microsoft.com/?kbid=2525835
KB2529073 http://support.microsoft.com/?kbid=2529073
KB2531907 http://support.microsoft.com/?kbid=2531907
KB2533552 http://support.microsoft.com/?kbid=2533552
KB2536275 http://support.microsoft.com/?kbid=2536275
KB2536276 http://support.microsoft.com/?kbid=2536276
KB2541014 http://support.microsoft.com/?kbid=2541014
KB2544521 http://support.microsoft.com/?kbid=2544521
KB2544893 http://support.microsoft.com/?kbid=2544893
KB2545698 http://support.microsoft.com/?kbid=2545698
KB2547666 http://support.microsoft.com/?kbid=2547666
KB2550886 http://support.microsoft.com/?kbid=2550886
KB2552040 http://support.microsoft.com/?kbid=2552040
KB2552343 http://support.microsoft.com/?kbid=2552343
KB2556532 http://support.microsoft.com/?kbid=2556532
KB2560656 http://support.microsoft.com/?kbid=2560656
KB2563227 http://support.microsoft.com/?kbid=2563227
KB2564958 http://support.microsoft.com/?kbid=2564958
KB2567680 http://support.microsoft.com/?kbid=2567680
KB2570947 http://support.microsoft.com/?kbid=2570947
KB2572077 http://support.microsoft.com/?kbid=2572077
KB2584146 http://support.microsoft.com/?kbid=2584146
KB2585542    http://support.microsoft.com/?kbid=2585542
KB2588516 http://support.microsoft.com/?kbid=2588516
KB2598845 http://support.microsoft.com/?kbid=2598845
KB2603229 http://support.microsoft.com/?kbid=2603229
KB2607047 http://support.microsoft.com/?kbid=2607047
KB2608658 http://support.microsoft.com/?kbid=2608658
KB2618451 http://support.microsoft.com/?kbid=2618451
KB2620704 http://support.microsoft.com/?kbid=2620704
KB2620712 http://support.microsoft.com/?kbid=2620712
KB2621440    http://support.microsoft.com/?kbid=2621440
KB2631813 http://support.microsoft.com/?kbid=2631813
KB2632503 http://support.microsoft.com/?kbid=2632503
KB2633873 http://support.microsoft.com/?kbid=2633873
KB2633952 http://support.microsoft.com/?kbid=2633952
KB2636573 http://support.microsoft.com/?kbid=2636573
KB2639308 http://support.microsoft.com/?kbid=2639308
KB2640148 http://support.microsoft.com/?kbid=2640148
KB2641653 http://support.microsoft.com/?kbid=2641653
KB2641690 http://support.microsoft.com/?kbid=2641690
KB2643719 http://support.microsoft.com/?kbid=2643719
KB2644615    http://support.microsoft.com/?kbid=2644615
KB2645640 http://support.microsoft.com/?kbid=2645640
KB2647516 http://support.microsoft.com/?kbid=2647516
KB2647518 http://support.microsoft.com/?kbid=2647518
KB2654428
http://support.microsoft.com/?kbid=2654428
KB2656356 http://support.microsoft.com/?kbid=2656356
KB2660075 http://support.microsoft.com/?kbid=2660075
KB2665364 http://support.microsoft.com/?kbid=2665364
KB2667402 http://support.microsoft.com/?kbid=2667402
KB976902 http://support.microsoft.com/?kbid=976902
KB982018
http://support.microsoft.com/?kbid=982018
Thanks
Ricardo

Similar Messages

Cluster Shared Volume disappeared after taking the volume offline for Validation Tests.

Hi,
After an unknown issue with one of our Hyper-V 4 Node cluster running on Server 2008 R2 SP1 with fibre channel NEC D3-10 SAN Storage all our cluster shared volumes were in redirecting mode and I was unable to get them back online. Only after rebooting all
the nodes one by one the disks came back online. Eventlog messages indicated that I had to test my cluster validation. After shutting down all the virtual machines I set all the cluster shared volumes offline and started the complete validation test. The following
warnings/errors appeared during the test.
An error occurred while executing the test.
An error occurred retrieving the
disk information for the resource 'VSC2_DATA_H'.
Element not found (Validate Volume Consistency Test)
Cluster disk 4 is a Microsoft MPIO based disk
Cluster disk 4 from node has 4 usable path(s) to storage target
Cluster disk 4 from node has 4 usable path(s) to storage target
Cluster disk 4 is not managed by Microsoft MPIO from node
Cluster disk 4 is not managed by Microsoft MPIO from node (Validate Microsoft MPIO-based disks test)
SCSI page 83h VPD descriptors for cluster disk 4 and 5 match (Validate SCSI device Vital Product Data (VPD) test)
After the test the cluster shared volume was disappeared (the resource is online).
Cluster events that are logged
Cluster physical disk resource 'DATA_H' cannot be brought online because the associated disk could not be found. The expected signature of the disk was '{d6e6a1e0-161e-4fe2-9ca0-998dc89a6f25}'. If the disk was replaced or restored, in the Failover Cluster
Manager snap-in, you can use the Repair function (in the properties sheet for the disk) to repair the new or restored disk. If the disk will not be replaced, delete the associated disk resource. (Event 1034)
Cluster disk resource found the disk identifier to be stale. This may be expected if a restore operation was just performed or if this cluster uses replicated storage. The DiskSignature or DiskUniqueIds property for the disk resource has been corrected.
(Event 1568)
In disk management the disk is unallocated, unknown, Reserved. When the resource is on one node and i open disk management i get the warning that i have to initialize the disk. I did not do this yet.
Reading from other posts i think that the partition table got corrupted but i have no idea how to get it back. I found the following information but it's not enough for me to go ahead with: Using a tool like TestDisk to rewrite the partition table. then
rewriting the uniqueID to the disk brought everything back. But still no explaination as to why we had our "High Availability" Fail Over cluster down for nearly 2 Days. This happened to us twice within the past week.
Anybody that an idea how to solve this? I think my data is still intact.
Thanx for taking the time to read this.
DJITS.

Hi,
Error information you provided indicate disk connection failure issue, please confirm shared disk 4 is available:
To review hardware, connections, and configuration of a disk in cluster storage:
On each node in the cluster, open Disk Management (which is in Server Manager under Storage) and see if the disk is visible from one of the nodes (it should be visible from one node but not multiple nodes). If it is visible to
a node, continue to the next step. If it is not visible from any node, still in Disk Management on a node, right-click any volume, click Properties, and then click the Hardware tab. Click the listed disks or LUNs to see if all expected disks or LUNs appear.
If they do not, check cables, multi-path software, and the storage device, and correct any issues that are preventing one or more disks or LUNs from appearing. If this corrects the overall problem, skip all the remaining steps and procedures.
Review the event log for any events that indicate problems with the disk. If an event provides information about the disk signature expected by the cluster, save this information and skip to the last step in this procedure.
To open the failover cluster snap-in, click Start, click Administrative Tools, and then click Failover Cluster Management. If the User Account Control dialog box appears, confirm that the action it displays is what you want, and
then click Continue.
In the Failover Cluster Management snap-in, if the cluster you want to manage is not displayed, in the console tree, right-click Failover Cluster Management, click Manage a Cluster, and then select or specify the cluster that
you want.
If the console tree is collapsed, expand the tree under the cluster you want to manage, and then click Storage.
In the center pane, find the disk resource whose configuration you want to check, and record the exact name of the resource for use in a later step.
Click Start, point to All Programs, click Accessories, right-click Command Prompt, and then click Run as administrator.
Type:
CLUSTER RESOURCE DiskResourceName /PRIV >path\filename.TXT
For DiskResourceName, type the name of the disk resource, and for path\filename, type a path and a new filename of your choosing.
Locate the file you created in the previous step and open it. For a master boot record (MBR) disk, look in the file for DiskSignature. For a GPT disk, look in the file for DiskIdGuid.
Use the software for your storage to determine whether the signature of the disk matches either the DiskSignature or DiskIdGuid for the disk resource. If it does not, use the following procedure to repair the disk configuration.
For more information please refer to following MS articles:
Event ID 1034 — Cluster Storage Functionality
http://technet.microsoft.com/en-us/library/cc756229(v=WS.10).aspx
Hope this helps!
TechNet Subscriber Support
If you are
TechNet Subscription user and have any feedback on our support quality, please send your feedback
here.
Lawrence
TechNet Community Support

Hyper-V cluster Backup causes virtual machine reboots for common Cluster Shared Volumes members.

I am having a problem where my VMs are rebooting while other VMs that share the same CSV are being backed up. I have provided all the information that I have gather to this point below. If I have missed anything, please let me know.
My HyperV Cluster configuration:
5 Node Cluster running 2008R2 Core DataCenter w/SP1. All updates as released by WSUS that will install on a Core installation
Each Node has 8 NICs configured as follows:
NIC1 - Management/Campus access (26.x VLAN)
NIC2 - iSCSI dedicated (22.x VLAN)
NIC3 - Live Migration (28.x VLAN)
NIC4 - Heartbeat (20.x VLAN)
NIC5 - VSwitch (26.x VLAN)
NIC6 - VSwitch (18.x VLAN)
NIC7 - VSwitch (27.x VLAN)
NIC8 - VSwitch (22.x VLAN)
Following hotfixes additional installed by MS guidance (either while build or when troubleshooting stability issue in Jan 2013)
KB2531907 - Was installed during original building of cluster
KB2705759 - Installed during troubleshooting in early Jan2013
KB2684681 - Installed during troubleshooting in early Jan2013
KB2685891 - Installed during troubleshooting in early Jan2013
KB2639032 - Installed during troubleshooting in early Jan2013
Original cluster build was two hosts with quorum drive. Initial two hosts were HST1 and HST5
Next host added was HST3, then HST6 and finally HST2.
NOTE: HST4 hardware was used in different project and HST6 will eventually become HST4
Validation of cluster comes with warning for following things:
Updates inconsistent across hosts
  I have tried to manually install "missing" updates and they were not applicable
  Most likely cause is different build times for each machine in cluster
   HST1 and HST5 are both the same level because they were built at same time
   HST3 was not rebuilt from scratch due to time constraints and it actually goes back to Pre-SP1 and has a larger list of updates that others are lacking and hence the inconsistency
   HST6 was built from scratch but has more updates missing than 1 or 5 (10 missing instead of 7)
   HST2 was most recently built and it has the most missing updates (15)
Storage - List Potential Cluster Disks
  It says there are Persistent Reservations on all 14 of my CSV volumes and thinks they are from another cluster.
  They are removed from the validation set for this reason. These iSCSI volumes/disks were all created new for
  this cluster and have never been a part of any other cluster.
When I run the Cluster Validation wizard, I get a slew of Event ID 5120 from FailoverClustering. Wording of error:
  Cluster Shared Volume 'Volume12' ('Cluster Disk 13') is no longer available on this node because of
  'STATUS_MEDIA_WRITE_PROTECTED(c00000a2)'. All I/O will temporarily be queued until a path to the
  volume is reestablished.
Under Storage and Cluster Shared VOlumes in Failover Cluster Manager, all disks show online and there is no negative effect of the errors.
Cluster Shared Volumes
We have 14 CSVs that are all iSCSI attached to all 5 hosts. They are housed on an HP P4500G2 (LeftHand) SAN.
I have limited the number of VMs to no more than 7 per CSV as per best practices documentation from HP/Lefthand
VMs in each CSV are spread out amonst all 5 hosts (as you would expect)
Backup software we use is BackupChain from BackupChain.com.
Problem we are having:
When backup kicks off for a VM, all VMs on same CSV reboot without warning. This normally happens within seconds of the backup starting
What have to done to troubleshoot this:
We have tried rebalancing our backups
  Originally, I had backup jobs scheduled to kick off on Friday or Saturday evening after 9pm
  2 or 3 hosts would be backing up VMs (Serially; one VM per host at a time) each night.
  I changed my backup scheduled so that of my 90 VMs, only one per CSV is backing up at the same time
   I mapped out my Hosts and CSVs and scheduled my backups to run on week nights where each night, there
   is only one VM backed up per CSV. All VMs can be backed up over 5 nights (there are some VMs that don't
   get backed up). I also staggered the start times for each Host so that only one Host would be starting
   in the same timeframe. There was some overlap for Hosts that had backups that ran longer than 1 hour.
  Testing this new schedule did not fix my problem. It only made it more clear. As each backup timeframe
  started, whichever CSV the first VM to start was on would have all of their VMs reboot and come back up.
I then thought maybe I was overloading the network still so I decided to disable all of the scheduled backup
and run it manually. Kicking off a backup on a single VM, in most cases, will cause the reboot of common
CSV members.
Ok, maybe there is something wrong with my backup software.
  Downloaded a Demo of Veeam and installed it onto my cluster.
  Did a test backup of one VM and I had not problems.
  Did a test backup of a second VM and I had the same problem. All VMs on same CSV rebooted
Ok, it is not my backup software. Apparently it is VSS. I have looked through various websites. The best troubleshooting
site I have found for VSS in one place it on BackupChain.com (http://backupchain.com/hyper-v-backup/Troubleshooting.html)
I have tested almost every process on there list and I will lay out results below:
  1. I have rebooted HST6 and problems still persist
  2. When I run VSSADMIN delete shadows /all, I have no shadows to delete on any of my 5 nodes
   When I run VSSADMIN list writers, I have no error messages on any writers on any node...
  3. When I check the listed registry key, I only have the build in MS VSS writer listed (I am using software VSS)
  4. When I run VSSADMIN Resize ShadowStorge command, there is no shadow storage on any node
  5. I have completed the registration and service cycling on HST6 as laid out here and most of the stuff "errors"
   Only a few of the DLL's actually register.
  6. HyperV Integration Services were reconciled when I worked with MS in early January and I have no indication of
   further issue here.
  7. I did not complete the step to delete the Subscriptions because, again, I have no error messages when I list writers
  8. I removed the Veeam software that I had installed to test (it hadn't added any VSS Writer anyway though)
  9. I can't realistically uninstall my HyperV and test VSS
  10. Already have latest SPs and Updates
  11. This is part of step 5 so I already did this. This seems to be a rehash of various other stratgies
I have used the VSS Troubleshooter that is part of BackupChain (Ctrl-T) and I get the following error:
  ERROR: Selected writer 'Microsoft Hyper-V VSS Writer' is in failed state!
  - Status: 8 (VSS_WS_FAILED_AT_PREPARE_SNAPSHOT)
  - Writer Failure code: 0x800423f0 (<Unknown error code>)
  - Writer ID: {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}
  - Instance ID: {d55b6934-1c8d-46ab-a43f-4f997f18dc71}
  VSS snapshot creation failed with result: 8000FFFF
VSS errors in event viewer. Below are representative errors I have received from various Nodes of my cluster:
I have various of the below spread out over all hosts except for HST6
Source: VolSnap, Event ID 10, The shadow copy of volume took too long to install
Source: VolSnap, Event ID 16, The shadow copies of volume x were aborted because volume y, which contains shadow copy storage for this shadow copy, wa force dismounted.
Source: VolSnap, Event ID 27, The shadow copies of volume x were aborted during detection because a critical control file could not be opened.
I only have one instance of each of these and both of the below are from HST3
Source: VSS, Event ID 12293, Volume Shadow Copy Service error: Error calling a routine on a Shadow Copy Provider {b5946137-7b9f-4925-af80-51abd60b20d5}. Routine details RevertToSnashot [hr = 0x80042302, A Volume Shadow Copy Service component encountered an
unexpected error.
Source: VSS, Event ID 8193, Volume Shadow Copy Service error: Unexpected error calling routine GetOverlappedResult. hr = 0x80070057, The parameter is incorrect.
So, basically, everything I have tried has resulted in no success towards solving this problem.
I would appreciate anything assistance that can be provided.
Thanks,
Charles J. Palmer
Wright Flood

Tim,
Thanks for the reply. I ran the first two commands and got this:
Name
Role Metric
Cluster Network 1
3  10000
Cluster Network 2 - HeartBeat                              1   1300
Cluster Network 3 - iSCSI                                    0  10100
Cluster Network 4 - LiveMigration                         1   1200
When you look at the properties of each network, this is how I have it configured:
Cluster Network 1 - Allow cluster network communications on this network and Allow clients to connect through this network (26.x subnet)
Cluster Network 2 - Allow cluster network communications on this network. New network added while working with Microsoft support last month. (28.x subnet)
Cluster Network 3 - Do not allow cluster network communications on this network. (22.x subnet)
Cluster Network 4 - Allow cluster network communications on this network. Existing but not configured to be used by VMs for Live Migration until MS corrected. (20.x subnet)
Should I modify my metrics further or are the current values sufficient.
I worked with an MS support rep because my cluster (once I added the 5th host) stopped being able to live migrate VMs and I had VMs host jumping on startup. It was a mess for a couple of days. They had me add the Heartbeat network as part of the solution
to my problem. There doesn't seem to be anywhere to configure a network specifically for CSV so I would assume it would use (based on my metrics above) Cluster Network 4 and then Cluster Network 2 for CSV communications and would fail back to the Cluster Network
1 if both 2 and 4 were down/inaccessible.
As to the iSCSI getting a second NIC, I would love to but management wants separation of our VMs by subnet and role and hence why I need the 4 VSwitch NICs. I would have to look at adding an additional quad port NIC to my servers and I would be having to
use half height cards for 2 of my 5 servers for that to work.
But, on that note, it doesn't appear to actually be a bandwidth issue. I can run a backup for a single VM and get nothing on the network card (It caused the reboots before any real data has even started to pass apparently) and still the problem occurs.
As to Backup Chain, I have been working with the vendor and they are telling my the issue is with VSS. They also say they support CSV as well. If you go to this page (http://backupchain.com/Hyper-V-Backup-Software.html)
they say they support CSVs. Their tech support has been very helpful but unfortunately, nothing has fixed the problem.
What is annoying is that every backup doesn't cause a problem. I have a daily backup of one of our machines that runs fine without initiating any additional reboots. But most every other backup job will trigger the VMs on the common CSV to reboot.
I understood about the updates but I had to "prove" it to the MS tech I was on the phone with and hence I brought it up. I understand on the storage as well. Why give a warning for something that is working though... I think that is just a poor indicator
that it doesn't explain that in the report.
At a loss for what else I can do,
Charles J. Palmer

Error when adding a disk to Cluster Shared Volumes

When adding a disk to Cluster Shared Volumes via Failover Cluster Manager, I get a couple of errors.
Event ID 5145 in System Log:
While adding the disk ('Cluster Disk 1') to Cluster Shared Volumes, setting explicit snapshot diff area association for volume ('\\?\Volume{420e2cc4-4fb4-41be-afb1-65f2ee62457a}\') failed with error 'HrError(0x8004230d)'. The only supported software snapshot
diff area association for Cluster Shared Volumes is to self.
Cluster disk resource 'Cluster Disk 1' failed to delete a software snapshot. The diff area on volume '\\?\Volume{420e2cc4-4fb4-41be-afb1-65f2ee62457a}\' could not be dissociated from volume '\\?\Volume{420e2cc4-4fb4-41be-afb1-65f2ee62457a}\'. This
may be caused by active snapshots. Cluster Shared Volumes requires that the software snapshot be located on the same disk.
Any ideas why I'm getting this error? This disk was previously added as a CSV to different Windows failover cluster, if that matters. Thanks.

Hi,
As the disk was previously used as a CSV, I assume there is still data on it.
Please check if any VSS snapshot created on that disk. If so, delete them and re-add it as a CSV to see the result - a quick way is to backup important files and perform a re-format if you cannot confirm.
If you have any feedback on our support, please send to [email protected]

Hyper V hosts with Cluster Shared Volume do not provide Shared VHDX ability

Hi there, i've configured a pair of Hyper V hosts (UCS B230 M2) with SAN-Boot and access to a Shared Fiber Channel LUN.
No matter what I try, when I add a SCSI Adapter and one VHDX virtual disk to a VM, the option for "Advanced" never shows up therefore I can not share the VHDX file between another VM, which prevents any VM based clustering from working (IE: Clustered
SQL server)
Here are the settings I have configured:
The LUN has been formatted with NTFS
The Hyper V hosts have been configured as a Windows Cluster with an IP address and the Fiber Channel LUN configured as a Cluster Shared Volume
I follows the guides below
Create a CSV
https://technet.microsoft.com/en-us/library/dn265980.aspx
Create a shared VHDX
https://technet.microsoft.com/en-us/library/dn282283.aspx
Further details:
Two HyperV server core hosts built
Both joined to AD
Both joined to a Windows failover cluster
One 2TB Fibre Channel LUN presented from an EMC VNX array to both hosts
LUN formatted as NTFS
LUN created as a Cluster Shared Volume
Cluster Shared Volume present on both HyperV hosts
VM created within the Cluster Shared Volume
VM Booted up and Windows 2012 has been installed
VM gracefully shut down
Through the HyperV Manager, going in to VM settings,create a SCSI adapter, create a new VHDX file inside the CSV, the “Advanced” option for the VHDX is not present, therefore the option to make the VHDX file shared is not available
I have followed all steps in the Microsft Technet pages to create a CSV and a VM with Shared VHDX
I’ve cleared all settings and restarted multiple times, but this option is not present

Hi Sir,
>>No matter what I try, when I add a SCSI Adapter and one VHDX virtual disk to a VM, the option for "Advanced" never shows up therefore I can not share the VHDX file between another VM
>>Yes we are using Hyper-V Server 2012 R2
As we know Hyper-v server 2012R2 is a free version without GUI , I'm afraid you are not using Win 8.1 or other windows server 2012R2 to manage that free version hyper-v server .
Shared vhdx is a new feature came with 2012R2 , if use previous windows to manage 2012R2 you will not see "advanced" option (The blue circle belongs 2012 , the red circle belongs to 2012R2 ):
Best Regards,
Elton Ji
Please remember to mark the replies as answers if they help and unmark them if they provide no help. If you have feedback for TechNet Subscriber Support, contact [email protected] .

Logical Disk Performance counter for cluster shared volume on Hyper-V

Hello All,
I am trying to collect counters like latency, queuelength from Win32_PerfFormattedData_PerfDisk_LogicalDisk WMI class.
Output of "Name" attribute for logical disks in this class as below:
Name: _Total
Name: C:
Name: E:
Name: HarddiskVolume1
Name: Q:
Name here doesn't show the actual label so I queried Win32_Volume class and wanted to join with performance WMI class. Out put of Win32_Volume is as below:
Caption: E:\
Label: New Volume
Name: E:\
Caption: Q:\
Label: Quorum
Name: Q:\
Caption: C:\
Label: Voume C
Name: C:\
Caption: F:\
Label: SAN
Name: F:\
Please note that "Name" attribute matches for all except one with label "SAN". This is cluster shared volume and "Name" attribute value is "HardDiskVolume1" in Win32_PerfFormattedData_PerfDisk_LogicalDisk class.
Is this is a configuration issue or any other alternative to get volume label and corresponding performance counters.
Thanks in advance
Regards,
Udupa

Hi Udupa,
I haven't found a better way, if you want to combine the two script, please refer to the script below:
$output=@()
$volumes = gwmi Win32_Volume
foreach($volume in $volumes){
$match=($volume.name).TrimEnd("\")
$counter=gwmi Win32_PerfFormattedData_PerfDisk_LogicalDisk |where{$_.name -eq $match}
$Object = New-Object PSObject
$Object | add-member Noteproperty name $volume.name
$Object | add-member Noteproperty label $volume.label
$Object | add-member Noteproperty AvgDiskQueueLength $counter.AvgDiskQueueLength
$output += $Object
$output
I hope this helps.

Cluster Shared Volume is no longer accessible from cluster node

Hello,
We have a 3 nodes Hyper-v Cluster running Windows Server 2012. Recently we start having error below intermittently on a node, and the VMs running on this host and LUN will power off.
Alert: Cluster Shared Volume is no longer accessible from cluster node
Source: Cluster Service
Path: HV01.itl.local
Last modified by: System
Last modified time: 12/1/2013 12:27:18 AM
Alert description: Cluster Shared Volume 'Volume1' ('Cluster_Vol1_R6') is no longer accessible from this cluster node because of error 'ERROR_TIMEOUT(1460)'. Please troubleshoot this node's connectivity to the storage device and network connectivity.
The only changes made recently is we installed VEEAM on test basis for DR replication. We switched off the Veeam server and stop the Veeam Services on the Hyper-V Hosts but we are still having same issue.
We are using an EMC SAN connected via FC as Shared storage and Powerpath as Multi-Pathing. No errors were found on the SAN.
I don't think the issue is related to the number of IO as we also experienced the issue at midnight during the week-end where no one was working.
Any help would be very much appreciated.
Thanks.
Irfan
Irfan Goolab SALES ENGINEER (Microsoft UC) MCP, MCSA, MCTS, MCITP, MCT

Hi,
Also, try to install the following recommend KBs.
Recommended hotfixes and updates for Windows Server 2012-based Failover Clusters
http://support.microsoft.com/kb/2784261
Also, there please confirm your VSS provider have the correct version.
The third party article:
VSS Provider with 2012 HyperV and CSV
https://community.emc.com/thread/170636
Thanks.
We
are trying to better understand customer views on social support experience, so your participation in this
interview project would be greatly appreciated if you have time.
Thanks for helping make community forums a great place.

SQL 2014 cluster shared volume

Hi,
I have testing SQL 2014 cluster in my lab to see how it's performing. 2xserver 2012r2 (virtual machines). There are some weird issues that are putting me in dust.
1. If I try to backup/restore database, I cannot select data path from gui. (I cannot "browse")
2. I cannot connect applications to databases (sqlclustername\instance) - permissions denied on cluster shared volume.
Can you please give me a hand on this?
Thank you.

Please check Symantec Backup Exec 2014 it works with CSV.
remove the CSV from the failover cluster manager and then try to point the tool and see if it is able to see the volumes without CSV.
http://www.symantec.com/business/support/index?page=content&id=TECH205833
Hope this helps.
Please Mark As Answer if it is helpful. \\Aim To Inspire Rather to Teach

Moving a Cluster Shared Volume with a VM

Hi,
Is that possible to move a Cluster Shared Volume (dedicated to 1 VM) with its associated VM together if the VM live migrates to another node ?
I try to find a way on Internet but nothing ...
Thank you for the answer :-)

Actually, I have an exact block copy in the secondary site that is being replicated using the storage replication engine.......the CSV in site A is read/Write, and in a "paired" status with the exact block copy in site B. While in "Paired"
status, access to the copy in site B is restricted and cannot be written to. So the Coordinator Node has to be on a node in the same site as the Read/Write Primary CSV in order for the VMs to access the CSV. I have tested this and there is no way
around it, so I am looking for a way to control the Coordinator Node Resource so I can keep it with the VM's CSV when I failover my Geo-cluster to the B site. I have a way to reverse the replication and swap sites during the failover, but I can't make
the coordinator Node resource follow that failover and move with the CSV that has all my clustered VMs on?
I can make the Coordinator Node move to Site B if I Pause and Drain the Roles in Site A, but this is not a viable way to move between sites. so I am looking for a way to move the Coordinator Node resource not the CSV as the link you provide shows.

LUN can't be accessed after move it to another Hyper-V Failover Cluster without "remove from cluster shared volumes" on the original cluster

Hi all,
I have a old cluster, let's call it cluster01, and a new cluster, cluster02. There is a LUN attach to cluster01 as a CSV volume. I forgot to "remove from cluster shared volumes" in Failover Cluster console, then power off the cluster01 and
attach the LUN to the cluster02. Now the LUN can't be accessed in cluster02, it show as a RAW disk. I tried to attach the LUN to it's original cluster cluster01 but can't read it too.
Is there any way to get it back?

Hi Zephyrhu,
Can you run Clear-ClusterDiskReservation powershell cmdlet and see that helps?
http://technet.microsoft.com/en-us/library/ee461016(WS.10).aspx
Thanks,
Umesh.S.K

Losing Access to Cluster Shared Volumes: Cluster Shared Volume 'Volume1' ('CSV Disk1') has entered a paused state because of '(c0000435)'

Hi,
Just built a Server 2012 R2 Hyper-V failover cluster connected to Equallogic 4110 storage arrays with latest firmware and HIT kits.
When creating a clone or vm from a template we see that the cluster loses access to the storage csv volume that is hosted on the equallogic storage with the following errors:
Cluster Shared Volume 'Volume1' ('CSV Disk1') has entered a paused state because of '(c0000435)'. All I/O will temporarily be queued until a path to the volume is reestablished.
Can anyone shed any light onto this issue?
Full details below:
Log Name: System
Source: Microsoft-Windows-FailoverClustering
Date: 06/08/2014 09:31:17
Event ID: 5120
Task Category: Cluster Shared Volume
Level: Error
Keywords:
User: SYSTEM
Computer: SVR1
Description:
Cluster Shared Volume 'Volume1' ('CSV Disk1') has entered a paused state because of '(c0000435)'. All I/O will temporarily be queued until a path to the volume is reestablished.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Microsoft-Windows-FailoverClustering" Guid="{BAF908EA-3421-4CA9-9B84-6689B8C6F85F}" />
<EventID>5120</EventID>
<Version>0</Version>
<Level>2</Level>
<Task>38</Task>
<Opcode>0</Opcode>
<Keywords>0x8000000000000000</Keywords>
<TimeCreated SystemTime="2014-08-06T08:31:17.330643100Z" />
<EventRecordID>36230</EventRecordID>
<Correlation />
<Execution ProcessID="2336" ThreadID="3524" />
<Channel>System</Channel>
<Computer>SVR1</Computer>
<Security UserID="S-1-5-18" />
</System>
<EventData>
<Data Name="VolumeName">Volume1</Data>
<Data Name="ResourceName">CSV Disk1</Data>
<Data Name="ErrorCode">(c0000435)</Data>
</EventData>
</Event>
Microsoft Partner

Hi rEMOTE_eVENT,
Could you tell us how you clone a vm “When creating a clone or vm from a template” , did your cluster can pass the cluster validation test, the copied vm have the same
BIOSGUID information and etc. Please try to use the general installed system to install failover cluster.
More information:
How to use uniquely identify a virtual machine in Hyper-V
http://blogs.technet.com/b/jhoward/archive/2008/09/16/how-to-use-uniquely-identify-a-virtual-machine-in-hyper-v.aspx
The similar thread:
How to Clone VMs in Hyper-V
http://social.technet.microsoft.com/Forums/windowsserver/en-US/67c4c555-14fd-4164-bf5b-59ce883c8b18/how-to-clone-vms-in-hyperv?forum=winserverhyperv
I’m glad to be of help to you!
We
are trying to better understand customer views on social support experience, so your participation in this
interview project would be greatly appreciated if you have time.
Thanks for helping make community forums a great place.

Rename Cluster Shared Volume folder

I renamed my cluster shared volume from Volume1 to the same name as my SAN LUN to keep things straight. The name has a dash in it and looks something like this: LUN 3 - HyperV Virtual Desktop Storage 2 I am trying to deploy a new
desktop collection with the virtual desktop location set to C:\ClusterStorage\LUN 3 - HyperV Virtual Desktop Storage 2\Virtual Desktops but every time I enter that path in the desktop creation wizard Server Manager locks up.
Anyone seen this?

Thank you for updating the result!
If you have any feedback on our support, please send to [email protected]

DPM 2010 fails to backup virtual guests on a Hyper-V Cluster Shared Volume using hardware VSS provider

We have been battling with this problem for 2 years now and have raised calls with Microsoft and EMC both of which have not resulted in a resolution.
We have an 8 node CSV with about 40 virtual servers on it. The Hyper-V hosts are Windows 2008 R2 SP1 servers with the Hyper-V role installed and the SAN is a Clarion CX3-10c and we are using the EMC 4.7.1 hardware provider for snaphots. The problem
we have is that snapshots are not 'always' getting created on the SAN and the recovery points in DPM fail. It often takes several attempts to re-run the job before it successfully works.
The MaxParallelBackups registry key is set to 1 on the DPM server so we are running jobs serially on each node and we have aligned all the VMs so the Hyper-V owner of each VM is the same owner of the CSV that those VMs have their resources on. We have done
this to avoid any ownership changes to reduce the risk of failures. This has had some success and it is not always the same servers that fail which gives this problem an unfortunate degree on intermittency!
There is no DataSourceGroups.xml anymore after I have tried to implement previously (it's effectiveness unproven though). We get a few days grace sometimes where backups are running with no problems but the longest run of success we have had is about 2-3
weeks which is our most recent run and it's cessation has led me to post this on the forum. Since Wednesday last week we have had regular failures with between 9 - 12 failures each night. The annoying part of this is that we hadn't made any changes to
either DPM or the Cluster which makes no real sense.
One thing I could do is split the Protection Groups so all the VMs on Hyper-V host1 are in one group, all the ones on Hyper-V host2 in another and so forth, however if anyone could advise me if this will be any help or not before I do it that would be much
appreciated as I don't want to do this unless I have to in case I hit capacity problems (most VMs are in one large protection group at present)
Any help would be much appreciated (I have logs I can attach if anyone would like to see a sample of when the failures occur),
Thanks
Chris

Hi,
I'm not on the Windows team, so don't know about the inner workings of the fix and what to expect. However, you can try using diskshadow.exe to delete all the snapshots for the given volume and see if that clears them up.
DISKSHADOW> delete shadows /?
DELETE SHADOWS { ALL | VOLUME <volume> | OLDEST <volume> | SET <setID> | ID <shadowID> | EXPOSED <drive letter, mountPoint or share> }
        Delete shadow copies, both persistent and non-persistent.
        ALL                     All shadow copies.
        VOLUME <volume>         Delete all shadow copies of the given volume.
        OLDEST <volume>         Delete the oldest shadow copy of the given volume.
        SET <setID>             Delete the shadow copies in the shadow copy set specified by the setId parameter.
        ID <shadowID>           Delete the shadow copy specified by the shadowId parameter.
        EXPOSED <exposeName>    Delete the shadow copy that is exposed at the specified drive letter, mount point or share.
        Examples: DELETE SHADOWS ALL
                  DELETE SHADOWS EXPOSED p:
                  DELETE SHADOWS EXPOSED ShareName
So something like: DELETE SHADOWS VOLUME E:   or DELETE SHADOWS ALL
Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT]
This posting is provided "AS IS" with no warranties, and confers no rights.

2 Cluster Shared Volumes filling up with shadow copies and I can't delete them.

three node Server 2012 R2 cluster
2 CSVs
Veeam Backup software v7
We've been running out of space on our CSVs. I determined that the hidden System Volume Information directory is huge. It is huge because of large VSS files in the directory.
Path to CSVs
c:\ClusterStorage\Volume1
c:\ClusterStorage\Volume2
===========================================================================
List of VSS providers
vssadmin list providers
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2013 Microsoft Corp.
Provider name: 'SmVssProvider'
Provider type: Hardware
Provider Id: {24e4e8a3-69cb-4370-8b7b-c5276cd49765}
Version: 11.10.0001
Provider name: 'Microsoft CSV Shadow Copy Helper Provider'
Provider type: Software
Provider Id: {26d02d81-6aac-4275-8504-b9c6edc5261d}
Version: 1.0.0.1
Provider name: 'Microsoft CSV Shadow Copy Provider'
Provider type: Software
Provider Id: {400a2ff4-5eb1-44b0-8a05-1fcac0bcf9ff}
Version: 1.0.0.1
Provider name: 'Microsoft File Share Shadow Copy provider'
Provider type: Fileshare
Provider Id: {89300202-3cec-4981-9171-19f59559e0f2}
Version: 1.0.0.1
Provider name: 'Microsoft Software Shadow Copy provider 1.0'
Provider type: System
Provider Id: {b5946137-7b9f-4925-af80-51abd60b20d5}
Version: 1.0.0.7
===========================================================================
Vssadmin list shadowstorage doesn't reference the location of my CSVs...
PS C:\Windows\system32> vssadmin list shadowstorage
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2013 Microsoft Corp.
Shadow Copy Storage association
For volume: (C:)\\?\Volume{6bf32257-eccb-11e3-80b5-806e6f6e6963}\
Shadow Copy Storage volume: (C:)\\?\Volume{6bf32257-eccb-11e3-80b5-806e6f6e6963}\
Used Shadow Copy Storage space: 0 bytes (0%)
Allocated Shadow Copy Storage space: 0 bytes (0%)
Maximum Shadow Copy Storage space: 13.6 GB (10%)
============================================================================
PS C:\clusterstorage\volume1> mountvol
Creates, deletes, or lists a volume mount point.
MOUNTVOL [drive:]path VolumeName
MOUNTVOL [drive:]path /D
MOUNTVOL [drive:]path /L
MOUNTVOL [drive:]path /P
MOUNTVOL /R
MOUNTVOL /N
MOUNTVOL /E
path Specifies the existing NTFS directory where the mount
point will reside.
VolumeName Specifies the volume name that is the target of the mount
point.
/D Removes the volume mount point from the specified directory.
/L Lists the mounted volume name for the specified directory.
/P Removes the volume mount point from the specified directory,
dismounts the volume, and makes the volume not mountable.
You can make the volume mountable again by creating a volume
mount point.
/R Removes volume mount point directories and registry settings
for volumes that are no longer in the system.
/N Disables automatic mounting of new volumes.
/E Re-enables automatic mounting of new volumes.
Possible values for VolumeName along with current mount points are:
\\?\Volume{6bf32256-eccb-11e3-80b5-806e6f6e6963}\
*** NO MOUNT POINTS ***
\\?\Volume{6bf32257-eccb-11e3-80b5-806e6f6e6963}\
C:\
\\?\Volume{bd46bae7-015c-4115-88ca-e6e9067fdea1}\
C:\ClusterStorage\Volume2\
\\?\Volume{caf6938a-5faa-4f3b-8e2b-e27d1498dedf}\
C:\ClusterStorage\Volume1\
\\?\Volume{6bf3225c-eccb-11e3-80b5-806e6f6e6963}\
D:\
====================================================================

When veeam backup call VSS, it create cache files that at the end of backup should be purged. I suppose that in some case this process don't work properly and leave orphaned cache files.
it's possible that when you move CSV on another node these orphaned files will be purged.
from the disk management, right click on CSV, properties, can you see the shadow copies tab?

VM using cluster shared volume will not run if the CSV is owned by a different node.

If a VM is owned by node "A" and the VHD file is on a CSV that is owned by node "B", it will fail to start. Hyper-v manager popup says that the VHD file is not accessible. If you change the owner of the CSV file to
node "A", it works. This would imply that the node cannot access the files on the CSV it does not own. However, the you can logon to the nodes and all have access to the csv.
This is a Server 2008 R2 cluster. I would appreciate any help in solving this.

Hi Dana,
No you don't need to destroy the cluster.
If you run cluster validation from fail over cluster manager it will allow you to run all checks. It will mention that you have resources that are using some of the CSV but then give you the option to take those cluster resources offline during the test
to check all storage. Or to leave those CSV out of the test depending on which you choose. All other test should be possible with the cluster up and it will NOT need to be destroyed to run them.
Open validation wizard
Select "Run all test (recommended)
On the next window check all storage and then run the checks.
NOTE: This will take all resources offline so if checking CSV with VMs on it would be probably best to shut them down first.
Kind Regards
Michael Coutanche
Blog:Twitter:LinkedIn:
Note: Posts are provided “AS IS” without warranty of any kind, either expressed or implied, including but not limited to the implied warranties of merchantability and/or fitness for a particular purpose.

Cluster shared volume disappear... STATUS_MEDIA_WRITE_PROTECTED(c00000a2)

Similar Messages

Maybe you are looking for