Cluster Disk i/o Timeout

Hi ,
We are stuck in problem with our private cloud protection , when ever DPM trying to backup virtual machines the cluster shared volume i.o timeout and the disk disappeared for a moment and that cause my virtual machine rebooted unexpectedly and move to different
nodes of cluster . 
Follow are the configuration of my Infrastructure .
1. Windows server 2012 Cluster X 6 nodes
2. DPM 2012 SP1
Event Generated when backup initiate :
Cluster Shared Volume 'Volume2' ('CSV 7TB Cluster Disk Production') is no longer available on this node because of 'STATUS_CLUSTER_CSV_AUTO_PAUSE_ERROR(c0130021)'. All I/O will temporarily be queued until a path to the volume is reestablished.
I have applied hot fix as Microsoft recommended 
http://support.microsoft.com/kb/2813630/en-us
Disabled ODX as well , because my storage doesn't support this feature .
please help me out  to resolve this matter .
Best Regards,
Muzammil
Muzammil Ubaray

Hi,
As far as in know, the FC SAN and the iSCSI SAN just have the different interface in SAN architecture, please try the solution of Elden and Darren then review the issue status.
Thanks.
We
are trying to better understand customer views on social support experience, so your participation in this
interview project would be greatly appreciated if you have time.
Thanks for helping make community forums a great place.

Similar Messages

  • How to get information about cluster disk using powershell

    Hi
    How can I list all cluster disks in my Hyper-V cluster using Powershell? I see two cluster disk "Cluster Disk 1" and "Cluster Disk 1" in my cluster management console but I would like to see them using powershell.
    Kind Regards Tomasz

    http://tinyurl.com/mfro3q5
    Get-ClusterResource
    Get-ClusterQuorum
    Get-ClusterSharedVolume
    . : | : . : | : . tim

  • SQL 2008 R2 Cluster | change Cluster Disks

    Hello,
    We have a SQL cluster (for SharePoint 2010) consists of two nodes Windows server 2008 R2 & SQL 2008 r2.
    The SQL cluster have MSDTC (Clustered) & SQL services, with total four disks:
    Quorum Disk
    MSDTC Disk
    Databases disk
    Logs disk.
    Now the old SAN will be decommissioned and new LUNs have added to replace the our disks above. I managed to change Quorum & MSDTC. I used the below robocopy command to copy databases and logs with the same folder structure and permissions:
    robocopy t:\ l:\ /E /MIR /SEC /COPYALL /V
    I stopped SQL services then swapped drive letters , when I start SQL services it starts without problems (using the new Disks).
    But the issue is when I connect to SQL management studio, all databases are in suspect mode. I know there some SQL query to be run against each database , but this a production environment and I don't want to mess with it.
    Is there any other way to change cluster disks of SQL cluster? or use the above method without getting into suspect mode?
    Thanks, Shehatovich

    Hello, Shehatovich
    I you have copied the files while there were still online, that might have been the cause 'suspect' state of your databases.
    I would suggest you to follow the steps:
    Add the new disks to cluster, with validation in every single Cluster node (then cluster will be able to get the signature of the disks)
    Stop SQL Server service.
    Copy the MDF and LDF to the new disks,
    use the Detach and attach Method.  (http://msdn.microsoft.com/en-us/library/ms187858.aspx)
    After check tha your databases are online and consistent.
    Stop SQL Server service again and remove the disks
    Hope it may help you...
    Regards,
    Edvaldo Castro
    Edvaldo Castro http://edvaldocastro.com MCITP Database Administrator on SQL Server® 2008 MCITP Database Administrator on SQL Server® 2005 MCTS: SQL Server® 2008 MCTS: SQL Server® 2005 MCT: Microsoft Certified Trainer MTA: Microsoft Technology Associate
    MTAC – Microsoft Technical Audience Contributor CCSQLA – Cambridge Certified SQL Associate TOEIC – Test of English for International Communication

  • Cannot add cluster disk on windows 2012

    I have 2 servers connected to a shared storage array using Fibre Channel.  Currently there are 2 luns on the storage device.  
    From the Failover Cluster Manager, when I select “Add Disk” from Storage – Disks, both luns show up and I can add them without problems.  But if I choose to only
    add one of them first (it doesn't make a difference which one is added first), then it will not allow adding the second one later.  I get a message: "No disks suitable for cluster disks were found. For diagnostic information about disks available to the
    cluster, use Validate a Configuration Wizard to run Storage tests."
    When I do add one (or both disks at the same time), they work just fine for failover clustering.
    I can’t imagine this is by design. Is this a known/unknown issue, or is there something special that needs to be done?
    Thanks

    Ok, no problem. I run the validation tests (Validate Cluster - Storage only) and I received the details shown below. I can clearly see that Cluster Validation tool does not find any disk suitable for 2012 cluster, and even I can see why (it is plain English).
    Unfortunately this still does not explain how to fix the problem. Recommendations like "run Clear-ClusterDiskReservation PowerShell cmdlet to remove the Persistent Reservation from the disk" do not work - this discussion is available in another thread where
    people having the same issue and no luck so far. I guess that I'm using old SAN (and I know that) that simply does not support SCSI-3 persistent reservation so those disks cannot be used in 2012 cluster. Oh well, will have to wait fro new SAN and then I can
    play again...
    Validation test results:
    Physical disk with identifier {95059d50-a173-49bd-b029-fec326acd78c} has an existing Persistent Reservation placed on it, and will be removed from the validation set. This disk may already be clustered, on a different cluster or in use by another system.
    Please verify the storage configuration and LUN zoning. If you wish to cluster this disk, you can use the Clear-ClusterDiskReservation PowerShell cmdlet to remove the Persistent Reservation from the disk.
    Physical disk with identifier {0b301c72-c88f-47ee-9389-11efdeed6cdb} has an existing Persistent Reservation placed on it, and will be removed from the validation set. This disk may already be clustered, on a different cluster or in use by another system.
    Please verify the storage configuration and LUN zoning. If you wish to cluster this disk, you can use the Clear-ClusterDiskReservation PowerShell cmdlet to remove the Persistent Reservation from the disk.
    Physical disk with identifier {eec0be37-5dfe-47f7-a149-4f1dec18450b} has an existing Persistent Reservation placed on it, and will be removed from the validation set. This disk may already be clustered, on a different cluster or in use by another system.
    Please verify the storage configuration and LUN zoning. If you wish to cluster this disk, you can use the Clear-ClusterDiskReservation PowerShell cmdlet to remove the Persistent Reservation from the disk.
    Physical disk with identifier {dcb8b1fe-7671-41c9-b996-da26c0b93751} has an existing Persistent Reservation placed on it, and will be removed from the validation set. This disk may already be clustered, on a different cluster or in use by another system.
    Please verify the storage configuration and LUN zoning. If you wish to cluster this disk, you can use the Clear-ClusterDiskReservation PowerShell cmdlet to remove the Persistent Reservation from the disk.
    No disks were found on which to perform cluster validation tests. To correct this, review the following possible causes:
    * The disks are already clustered and currently Online in the cluster. When testing a working cluster, ensure that the disks that you want to test are Offline in the cluster.
    * The disks are unsuitable for clustering. Boot volumes, system volumes, disks used for paging or dump files, etc., are examples of disks unsuitable for clustering.
    * Review the "List Disks" test. Ensure that the disks you want to test are unmasked, that is, your masking or zoning does not prevent access to the disks. If the disks seem to be unmasked or zoned correctly but could not be tested, try restarting the servers
    before running the validation tests again.
    * The cluster does not use shared storage. A cluster must use a hardware solution based either on shared storage or on replication between nodes. If your solution is based on replication between nodes, you do not need to rerun Storage tests. Instead, work
    with the provider of your replication solution to ensure that replicated copies of the cluster configuration database can be maintained across the nodes.
    * The disks are Online in the cluster and are in maintenance mode. No disks were found on which to perform cluster validation tests.

  • Cluster Disk keeps going offline. Service or application failed

    We currently have 2 servers, both housing 3 VMs (6 nodes total).  2 Disk are used to house these VM's.  Cluster Disk 1 keeps going offline, which kicks 2 of our servers offline until we migrate the disk to another node (puts it into redirected
    mode).  We then have to restart the server to re-establish to connection to the cluster disk.  Digging around in event viewer, I saw the following:
    Cluster resource 'Cluster Disk 2' in clustered service or application 'b0a21897-e59b-46c9-a495-273dc5ad2aea' failed.
    Cluster physical disk resource 'Cluster Disk 2' cannot be brought online because the associated disk could not be found. The expected signature of the disk was '{92eb716c-6878-42ad-b7de-f3879d72e232}'. If the disk was replaced or restored, in
    the Failover Cluster Manager snap-in, you can use the Repair function (in the properties sheet for the disk) to repair the new or restored disk. If the disk will not be replaced, delete the associated disk resource.
    The Cluster service failed to bring clustered service or application 'b0a21897-e59b-46c9-a495-273dc5ad2aea' completely online or offline. One or more resources may be in a failed state. This may impact the availability of the clustered service
    or application.
    I ran the cluster validation test and all seemed to check out except for Cluster Disk 1 and the two servers associated with this server, as shown below:
    Validating cluster resource Cluster Disk 2.
    This resource is marked with a state of 'Offline'. The functionality that this resource provides is not available while it is in the offline state. The resource may be put in this state by an administrator or program. It may also be a newly created
    resource which has not been put in the online state or the resource may be dependent on a resource that is not online. Resources can be brought online by choosing the 'Bring this resource online' action in Failover Cluster Manager.
    This resource is marked with a state of 'Failed' instead of 'Online'. This failed state indicates that the resource had a problem either coming online or had a failure while it was online. The event logs and cluster logs may have information
    that is helpful in identifying the cause of the failure.
    All disks are housed on our SAN.  Any idea where I can start troubleshooting further?  Changing the signature of the disk or finding out which application is failing?  All help is greatly appreciated.

    Please, help is greatly appreciated.  I have been trying to find a solution for almost a week.  I have a feeling this has to do with the disk.  We have 3 CSV's but the only ones in question are CSV 2 & 3. Cluster Disk 2 is the disk that
    keeps going offline.  Cluster Disk 3 has no problems at all.  They both are GPT volumes, however, after running a "cluster res", I received the following output:
    Listing private properties for 'Cluster Disk 2':
    T  Resource             Name                           Value
    D  Cluster Disk 2       DiskIdType                     1 (0x1)
    D  Cluster Disk 2       DiskSignature                0 (0x0)
    S  Cluster Disk 2       DiskIdGuid                     {92eb716c-6878-42ad-b7de-f3879d72e232}
    ===============================================================
    Listing private properties for 'Cluster Disk 3':
    T  Resource             Name                           Value
    D  Cluster Disk 3       DiskIdType                     1 (0x1)
    D  Cluster Disk 3       DiskSignature                 994692308 (0x3b49ccd4)
    S  Cluster Disk 3       DiskIdGuid                     {4f27bc43-6964-466e-a77d-2321b2b0b718}
    =============================================================
    As shown, cluster disk 1 has a disk signature but NOT a diskID.  Cluster disk 2 as a DiskID but NOT a signature.  It may be beneficial to add that cluster disk 3 used to be an MBR disk.  Not sure if that makes any difference given the output
    above...Performing DISKPART on the same disks shows the following (Don't mind the naming scheme, I know it can be a bit confusing)
    Disk 1 (Cluster Disk 2):
    HP P2000G3 FC/iSCSI  Multi-Path Disk Device
    Disk ID: {92EB716C-6878-42AD-B7DE-F3879D72E232}
    Disk 3 (Cluster Disk 3):
    HP P2000G3 FC/iSCSI  Multi-Path Disk Device
    Disk ID: {4F27BC43-6964-466E-A77D-2321B2B0B718}
    ================================================================
    Comparing the two, it selects the correct DiskID that was taken from "cluster res" command.  
    The error in my initial post states a signature was expected.  If I am not mistaken, in win2008 it uses "DiskSignature". The lack of this signature being populated for Cluster Disk 2 was resulting in the resources failing to come online.  So
    how can i go about fixing this signature issue.  Again, all help is appreciated since I am fairly new to all of this.  

  • Windows Server 2012 Cluster Validation Failing On List All Potential Cluster Disks

    I'm in the process of setting up a new cluster using Windows Server 2012 with a view to migrate from our existing 2008 R2 cluster.  I am having problems validating the storage and the wizard keeps failing on the List All Potential Cluster Disks
    test (which causes all the other storage tests to be cancelled).
    I'm getting the following error:
    Failed while verifying removal of any Persistent Reservation on physical disk 19ba99a4 at node RC-HYPERV-3.riddlesdown.local.
    When I look in validation report, disk 19ba99a4 is the management LUN (Access LUN 31 as termed by the storage software).
    Looking on my 2008 R2 cluster this doesn't appear in Disk Management.
    Any ideas how I can get rid of it or the error that it is causing?
     

    Hi,
    Thank you for your question.
    I am trying to involve someone familiar with this topic to further look at this issue. There might be some time delay. Appreciate your patience.
    Thank you for your understanding and support.
    Best Regards,
    Aiden
    TechNet Subscriber Support
    If you are
    TechNet Subscription
    user and have any feedback on our support quality, please send your feedback
    here.
    Aiden Cao
    TechNet Community Support

  • Taking cluster disk offline made all other resources in the group to go offline.

    Hi,
    We have three node windows 2008 R2 failover cluster. When we tried to take one of the cluster disk offline from a application and services, it made all other resources in that application and services to go offline even though there was no dependency on
    this disk. I want to know why it made all other resources to go offline. Can some one clarify please? Is it a default behavior?
    Umesh

    "When we tried to take one of the cluster disk offline from a application and services, it made all other resources in that application and services to go offline even though there was no dependency on this disk."
    I'm a little unclear on your problem description.  You state you are taking a "disk offline from a application and services".  That implies the disk is necessary for the application and/or service, otherwise there is no way to take it
    offline from that application and/or service.  Then you state "there was no dependency on this disk".  If there was no dependency on the disk, how you could take it "offline from a application and services"?
    Using the Failover Cluster Manager consolde, select one of the resources that is going offline.  Click on the Resources tab at the bottom of the center window.  That resource does not show any dependency on the disk being taken offline? 
    Does it show dependency on any other resource that is dependent upon the disk you are taking offline?
    . : | : . : | : . tim

  • SCOM 2012 event id 10801 cluster disks don't discovered.

    Hello.
    I have errors in Operations Manager log:
    Discovery data couldn't be inserted to the database. This could have happened because  of one of the following reasons:
         - Discovery data is stale. The discovery data is generated by an MP recently deleted.
         - Database connectivity problems or database running out of space.
         - Discovery data received is not valid.
     The following details should help to further diagnose:
     DiscoveryId: 5a84ee62-20c2-46a2-10b9-3dedaff65df6
     HealthServiceId: 3aeaca7c-48de-c0fc-0441-ffd5ef7aa7c3
     Microsoft.EnterpriseManagement.Common.DiscoveryDataInvalidRelationshipTargetException,The relationship target specified in the discovery data item is not valid.
    Relationship target ID: 2478193e-1a5f-4087-1b5f-95459123321e
    Rule ID: 5a84ee62-20c2-46a2-10b9-3dedaff65df6
    Instance:
    <?xml version="1.0" encoding="utf-16"?><RelationshipInstance TypeId="{acfe2f40-0a73-6764-21a5-bf59c41b2844}" SourceTypeId="{00000000-0000-0000-0000-000000000000}" TargetTypeId="{00000000-0000-0000-0000-000000000000}"><Settings
    /><SourceRole><Settings><Setting><Name>5c324096-d928-76db-e9e7-e629dcc261b1</Name><Value>SQL-01</Value></Setting><Setting><Name>af13c36e-9197-95f7-393c-84aa6638fec9</Name><Value>\\.\PHYSICALDRIVE18</Value></Setting></Settings></SourceRole><TargetRole><Settings><Setting><Name>5c324096-d928-76db-e9e7-e629dcc261b1</Name><Value>PDC-S-SQL-01.sibgenco.local</Value></Setting><Setting><Name>af13c36e-9197-95f7-393c-84aa6638fec9</Name><Value>Disk
    #18, Partition #0</Value></Setting></Settings></TargetRole></RelationshipInstance>.
    SQL-01 is server with clusters disks, and cluster disks are don't discovered.

    Hi,
    Hope the below articles can be helpful:
    Cluster resource groups are not monitored! Is there anything I can do?
    http://blogs.msdn.com/b/mariussutara/archive/2008/05/03/cluster-resource-groups-are-not-monitored-is-there-anything-i-can-do.aspx
    Event ID 10801 and 33333 in Operations Manager log
    http://www.itbl0b.com/2014/02/event-id-10801-33333-operations-manager-log.html#.U-QwunmKBes
    Please Note: Since site is not hosted by Microsoft, the link may change without notice. Microsoft does not guarantee the accuracy of this information the web.
    Regards, Yan Li

  • Cluster Disk failover problem

    Hi Everyone
    I have built a Windows 2008 R2 Failover Cluster. It is scheduled to be a HyperV cluster, but I have removed HyperV from the equation as I am having problems.
    Its a 2 node cluster connecting to a PS4000 via iscsi.
    Network config is
    Nic 1 - Host and Cluster Management - 10.160.19.x/24
    Nic 2 & 3 - icsci 10.160.46..x/24, MPIO,
    Nic 4 - CSV - 10.160.47/24
    Nic 5 - LiveMigration - 10.160.49.x/24
    Nic6 - VMSwitch - HyperV Virtual Machine "External" nic.
    The cluster ip is on 10.160.19.x.
    The Cluster validation passes sucessfully and so does creating the cluster and performing post creating validations. When I simulate failures of the Quorum disk, it fails over to the second node after requisites retrys on the first node...all good.
    However when I simulate failures on the second cluster disk, it just fails on the primary node. I can bring it online on same node or if I stop the cluster service on that node, everything successfully fails over to the second node. But automatic/simulating
    failover isn't working.
    When I dump out the cluster log(below)this is all the entries pertaining to the failover attempt. Sorry for the log dump.
    My analysis of the log, leads me in two directions, Is my nic assignment and config correct? Can I confirm the Cluster disk guids are identical between nodes?
    I appreciate any assistance. Apologies on the large log.
    Thanks
    [NM] Received request from client address 10.160.19.22.
    00000b18.00000f50::2011/01/13-16:10:32.005 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
    00000730.00000d3c::2011/01/13-16:10:32.005 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
    00000730.00000d3c::2011/01/13-16:10:32.005 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node.  Attempting to find a good node...
    00000b18.00000f50::2011/01/13-16:10:32.005 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
    00000730.00000d40::2011/01/13-16:10:32.005 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
    00000b18.00000f50::2011/01/13-16:10:32.317 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
    00000730.00000d3c::2011/01/13-16:10:32.317 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
    00000730.00000d3c::2011/01/13-16:10:32.317 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node.  Attempting to find a good node...
    00000b18.00000f50::2011/01/13-16:10:32.317 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
    00000730.000005e0::2011/01/13-16:10:32.317 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
    00000b18.00000f50::2011/01/13-16:10:32.317 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
    00000730.000005e0::2011/01/13-16:10:32.317 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
    00000730.000005e0::2011/01/13-16:10:32.317 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node.  Attempting to find a good node...
    00000b18.00000c1c::2011/01/13-16:10:32.317 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
    00000730.00000d3c::2011/01/13-16:10:32.317 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
    00000b18.00000c1c::2011/01/13-16:10:32.332 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
    00000730.00000d3c::2011/01/13-16:10:32.332 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
    00000730.00000d3c::2011/01/13-16:10:32.332 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node.  Attempting to find a good node...
    00000b18.00000c1c::2011/01/13-16:10:32.332 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
    00000730.00000d40::2011/01/13-16:10:32.332 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
    00000b18.00000c1c::2011/01/13-16:10:32.332 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
    00000730.000005e0::2011/01/13-16:10:32.332 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
    00000730.000005e0::2011/01/13-16:10:32.332 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node.  Attempting to find a good node...
    00000b18.00000c1c::2011/01/13-16:10:32.332 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
    00000730.00000d40::2011/01/13-16:10:32.332 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
    00000b18.00000c1c::2011/01/13-16:10:32.332 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
    00000730.00000d3c::2011/01/13-16:10:32.332 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
    00000730.00000d3c::2011/01/13-16:10:32.332 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node.  Attempting to find a good node...
    00000b18.00000c1c::2011/01/13-16:10:32.332 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
    00000730.000005e0::2011/01/13-16:10:32.332 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
    00000730.00000d3c::2011/01/13-16:10:35.296 INFO  [RCM] rcm::RcmApi::FailResource: (VM1)
    00000730.00000d40::2011/01/13-16:10:35.296 INFO  [RCM] HandleMonitorReply: FAILURENOTIFICATION for 'VM1', gen(1) result 0.
    00000730.00000d40::2011/01/13-16:10:35.296 INFO  [RCM] TransitionToState(VM1) Online-->ProcessingFailure.
    00000730.00000d40::2011/01/13-16:10:35.296 INFO  [RCM] rcm::RcmGroup::UpdateStateIfChanged: (Available Storage, Online --> Failed)
    00000730.00000d40::2011/01/13-16:10:35.296 ERR   [RCM] rcm::RcmResource::HandleFailure: (VM1)
    00000730.00000d40::2011/01/13-16:10:35.296 INFO  [RCM] resource VM1: failure count: 1, restartAction: 2.
    00000730.00000d40::2011/01/13-16:10:35.296 INFO  [RCM] Will restart resource in 500 milliseconds.
    00000730.00000d40::2011/01/13-16:10:35.296 INFO  [RCM] TransitionToState(VM1) ProcessingFailure-->[WaitingToTerminate to DelayRestartingResource].
    00000730.00000d40::2011/01/13-16:10:35.296 INFO  [RCM] rcm::RcmGroup::UpdateStateIfChanged: (Available Storage, Failed --> Pending)
    00000730.00000d40::2011/01/13-16:10:35.296 INFO  [RCM] TransitionToState(VM1) [WaitingToTerminate to DelayRestartingResource]-->[Terminating to DelayRestartingResource].
    00000bd4.00000eb0::2011/01/13-16:10:35.296 INFO  [RES] Physical Disk <VM1>: Terminate request.
    00000bd4.00000eb0::2011/01/13-16:10:35.327 WARN  [RES] Physical Disk <VM1>: Terminate: Failed to offline volume \Device\Harddisk3\Partition1, Error 5
    00000b18.00000c1c::2011/01/13-16:10:35.343 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
    00000730.00000d3c::2011/01/13-16:10:35.343 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
    00000730.00000d3c::2011/01/13-16:10:35.343 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node.  Attempting to find a good node...
    00000b18.00000c1c::2011/01/13-16:10:35.343 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
    00000730.00000d30::2011/01/13-16:10:35.343 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
    00000bd4.00000eb0::2011/01/13-16:10:35.343 INFO  [RES] Physical Disk <VM1>: HardDiskpCloseSVIHandles: Exit
    00000b18.00000c1c::2011/01/13-16:10:35.343 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
    00000730.00000d3c::2011/01/13-16:10:35.343 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
    00000730.00000d3c::2011/01/13-16:10:35.343 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node.  Attempting to find a good node...
    00000b18.00000c1c::2011/01/13-16:10:35.343 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
    00000730.00000d30::2011/01/13-16:10:35.343 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
    00000b18.00000c1c::2011/01/13-16:10:35.359 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
    00000730.00000d3c::2011/01/13-16:10:35.359 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
    00000730.00000d3c::2011/01/13-16:10:35.359 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node.  Attempting to find a good node...
    00000b18.00000c1c::2011/01/13-16:10:35.359 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
    00000730.00000d30::2011/01/13-16:10:35.359 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
    00000b18.00000c1c::2011/01/13-16:10:35.359 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
    00000730.00000d3c::2011/01/13-16:10:35.359 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
    00000730.00000d3c::2011/01/13-16:10:35.359 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node.  Attempting to find a good node...
    00000b18.00000c1c::2011/01/13-16:10:35.359 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
    00000730.00000d30::2011/01/13-16:10:35.359 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
    00000b18.00000c1c::2011/01/13-16:10:35.437 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
    00000730.00000d3c::2011/01/13-16:10:35.437 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
    00000730.00000d3c::2011/01/13-16:10:35.437 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node.  Attempting to find a good node...
    00000b18.00000c1c::2011/01/13-16:10:35.437 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
    00000730.00000d30::2011/01/13-16:10:35.437 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
    00000b18.00000c1c::2011/01/13-16:10:35.452 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
    00000730.00000d3c::2011/01/13-16:10:35.452 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
    00000730.00000d3c::2011/01/13-16:10:35.452 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node.  Attempting to find a good node...
    00000b18.00000c1c::2011/01/13-16:10:35.452 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
    00000730.00000d30::2011/01/13-16:10:35.452 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
    00000b18.00000c1c::2011/01/13-16:10:35.452 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
    00000730.00000d3c::2011/01/13-16:10:35.452 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
    00000730.00000d3c::2011/01/13-16:10:35.452 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node.  Attempting to find a good node...
    00000b18.00000c1c::2011/01/13-16:10:35.452 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
    00000730.00000d30::2011/01/13-16:10:35.452 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
    00000b18.00000c1c::2011/01/13-16:10:35.452 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
    00000730.00000d3c::2011/01/13-16:10:35.452 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
    00000730.00000d3c::2011/01/13-16:10:35.452 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node.  Attempting to find a good node...
    00000b18.00000c1c::2011/01/13-16:10:35.452 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
    00000730.00000d30::2011/01/13-16:10:35.452 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
    00000bd4.00000eb0::2011/01/13-16:10:36.263 INFO  [RES] Physical Disk: ReleaseDisk: stop reserve succeeded on device 3 (sig 44423655)
    00000730.00000d30::2011/01/13-16:10:36.279 INFO  [RCM] HandleMonitorReply: TERMINATERESOURCE for 'VM1', gen(2) result 0.
    00000730.00000d30::2011/01/13-16:10:36.279 INFO  [RCM] TransitionToState(VM1) [Terminating to DelayRestartingResource]-->DelayRestartingResource.
    00000730.00000d3c::2011/01/13-16:10:36.794 INFO  [RCM] Delay-restarting VM1 and any waiting dependents.
    00000730.00000d3c::2011/01/13-16:10:36.794 INFO  [RCM] TransitionToState(VM1) DelayRestartingResource-->OnlineCallIssued.
    00000bd4.00000eb0::2011/01/13-16:10:36.794 INFO  [RES] Physical Disk <VM1>: Online request.
    00000730.00000298::2011/01/13-16:10:36.794 INFO  [RCM] HandleMonitorReply: ONLINERESOURCE for 'VM1', gen(2) result 997.
    00000730.00000298::2011/01/13-16:10:36.794 INFO  [RCM] TransitionToState(VM1) OnlineCallIssued-->OnlinePending.
    00000bd4.00000c04::2011/01/13-16:10:36.794 INFO  [RES] Physical Disk <VM1>: Arbitrate request: FastPath 0
    00000bd4.00000c04::2011/01/13-16:10:36.794 INFO  [RES] Physical Disk: Enter EnumerateDevices: EnumDevice 0
    00000bd4.00000c04::2011/01/13-16:10:36.809 INFO  [RES] Physical Disk: Exit EnumerateDevices: status 0
    00000bd4.00000c04::2011/01/13-16:10:36.841 INFO  [RES] Physical Disk: Successful reserve, disk is unowned, key 4daf7466734d
    00000bd4.00000c04::2011/01/13-16:10:36.841 INFO  [RES] Physical Disk: Update disk props returns 0
    00000bd4.00000c04::2011/01/13-16:10:36.841 INFO  [RES] Physical Disk <VM1>: Disk is offline
    00000730.00000298::2011/01/13-16:10:36.841 INFO  [GUM] Node 1: Processing RequestLock 1:42
    00000730.00000314::2011/01/13-16:10:36.841 INFO  [GUM] Node 1: Processing GrantLock to 1 (sent by 2 gumid: 566)
    00000bd4.00000c04::2011/01/13-16:10:36.841 WARN  [RES] Physical Disk <VM1>: OnlineThread: Failed to get volume guid for device
    \\?\GLOBALROOT\Device\Harddisk3\Partition1\. Error 3
    00000b18.00000c1c::2011/01/13-16:10:36.856 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
    00000730.00000d3c::2011/01/13-16:10:36.856 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
    00000730.00000d3c::2011/01/13-16:10:36.856 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node.  Attempting to find a good node...
    00000bd4.00000bf8::2011/01/13-16:10:36.856 INFO  [RES] Physical Disk: PNP: Adding volume
    \\?\STORAGE#Volume#{73432ff7-18ee-11e0-aded-0026b93b46a0}#0000000000100000#{53f5630d-b6bf-11d0-94f2-00a0c91efb8b}
    00000bd4.00000f30::2011/01/13-16:10:36.856 INFO  [RES] Physical Disk <VM1>: Completed HardDiskpUpdateRegistryWorker (start time 16:10:36.841)
    00000bd4.00000bf8::2011/01/13-16:10:36.856 INFO  [RES] Physical Disk: PNP: Add Volume exit, status 0
    00000b18.00000c1c::2011/01/13-16:10:36.856 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
    00000730.00000d40::2011/01/13-16:10:36.856 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
    00000b18.00000c1c::2011/01/13-16:10:36.856 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
    00000730.00000d3c::2011/01/13-16:10:36.856 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
    00000730.00000d3c::2011/01/13-16:10:36.856 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node.  Attempting to find a good node...
    00000b18.00000c1c::2011/01/13-16:10:36.856 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
    00000730.00000d40::2011/01/13-16:10:36.856 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
    00000b18.00000c1c::2011/01/13-16:10:36.872 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
    00000730.00000d3c::2011/01/13-16:10:36.872 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
    00000730.00000d3c::2011/01/13-16:10:36.872 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node.  Attempting to find a good node...
    00000b18.00000c1c::2011/01/13-16:10:36.872 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
    00000730.00000d40::2011/01/13-16:10:36.872 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
    00000b18.00000c1c::2011/01/13-16:10:36.872 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
    00000730.00000d3c::2011/01/13-16:10:36.872 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
    00000730.00000d3c::2011/01/13-16:10:36.872 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node.  Attempting to find a good node...
    00000b18.00000c1c::2011/01/13-16:10:36.872 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
    00000730.00000298::2011/01/13-16:10:36.872 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
    00000bd4.00000c04::2011/01/13-16:10:40.850 WARN  [RES] Physical Disk <VM1>: OnlineThread: Failed to set volguid \??\Volume{7343300a-18ee-11e0-aded-0026b93b46a0}. Error: 183.
    00000bd4.00000c04::2011/01/13-16:10:40.850 INFO  [RES] Physical Disk <VM1>: Found 2 mount points for device \Device\Harddisk3\Partition1
    00000bd4.00000c04::2011/01/13-16:10:40.850 INFO  [RES] Physical Disk <VM1>: VolumeIsNtfs: Volume
    \\?\GLOBALROOT\Device\Harddisk3\Partition1\ has FS type NTFS
    00000bd4.00000c04::2011/01/13-16:10:40.850 INFO  [RES] Physical Disk: Volume
    \\?\GLOBALROOT\Device\Harddisk3\Partition1\ has FS type NTFS
    00000bd4.00000c04::2011/01/13-16:10:40.850 INFO  [RES] Physical Disk: MountPoint V:\ points to volume
    \\?\Volume{7343300a-18ee-11e0-aded-0026b93b46a0}\
    00000bd4.00000c04::2011/01/13-16:10:40.850 DBG   [ClRtl] CaptureShareInfo: Share 0000000000000050, Server 0000000000000000, Path 0000000000000056 SD 0000000000000000 size 242
    00000bd4.00000c04::2011/01/13-16:10:40.850 WARN  [ClRtl] SsCoreShareAdd(): status = 2118 share = V$ server = (null)
    00000bd4.00000c04::2011/01/13-16:10:40.850 DBG   [ClRtl] CaptureShareInfo: Share 0000000000000050, Server 0000000000000000, Path 0000000000000056 SD 0000000000000000 size 242
    00000bd4.00000c04::2011/01/13-16:10:40.850 INFO  [RHS] Resource VM1 has come online. RHS is about to report status change to RCM
    00000730.00000d3c::2011/01/13-16:10:40.850 INFO  [RCM] HandleMonitorReply: ONLINERESOURCE for 'VM1', gen(2) result 0.
    00000730.00000d3c::2011/01/13-16:10:40.850 INFO  [RCM] TransitionToState(VM1) OnlinePending-->Online.
    00000730.00000d3c::2011/01/13-16:10:40.850 INFO  [RCM] rcm::RcmGroup::UpdateStateIfChanged: (Available Storage, Pending --> Online)
    00000bd4.00000c04::2011/01/13-16:10:40.850 INFO  [RES] Physical Disk: DriveLetter mask: 0x200000
    00000b18.00000c1c::2011/01/13-16:10:40.896 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
    00000730.00000d30::2011/01/13-16:10:40.896 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
    00000730.00000d30::2011/01/13-16:10:40.896 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node.  Attempting to find a good node...
    00000b18.00000c1c::2011/01/13-16:10:40.896 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
    00000730.000006d8::2011/01/13-16:10:40.896 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
    00000b18.00000c1c::2011/01/13-16:10:40.912 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
    00000730.00000d30::2011/01/13-16:10:40.912 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
    00000730.00000d30::2011/01/13-16:10:40.912 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node.  Attempting to find a good node...
    00000b18.00000c1c::2011/01/13-16:10:40.912 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
    00000730.000006d8::2011/01/13-16:10:40.912 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
    00000b18.00000c1c::2011/01/13-16:10:40.912 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
    00000730.00000d30::2011/01/13-16:10:40.912 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
    00000730.00000d30::2011/01/13-16:10:40.912 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node.  Attempting to find a good node...
    00000b18.00000c1c::2011/01/13-16:10:40.912 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
    00000730.000006d8::2011/01/13-16:10:40.912 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
    00000b18.00000c1c::2011/01/13-16:10:40.912 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
    00000730.00000d30::2011/01/13-16:10:40.912 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
    00000730.00000d30::2011/01/13-16:10:40.912 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node.  Attempting to find a good node...
    00000b18.00000c1c::2011/01/13-16:10:40.912 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
    00000730.000006d8::2011/01/13-16:10:40.912 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
    00000b18.00000c1c::2011/01/13-16:10:40.928 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
    00000730.00000d30::2011/01/13-16:10:40.928 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
    00000730.00000d30::2011/01/13-16:10:40.928 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node.  Attempting to find a good node...
    00000b18.00000c1c::2011/01/13-16:10:40.928 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
    00000730.000006d8::2011/01/13-16:10:40.928 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
    00000b18.00000c1c::2011/01/13-16:10:40.928 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
    00000730.00000d30::2011/01/13-16:10:40.928 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
    00000730.00000d30::2011/01/13-16:10:40.928 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node.  Attempting to find a good node...
    00000b18.00000c1c::2011/01/13-16:10:40.928 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
    00000730.000006d8::2011/01/13-16:10:40.928 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
    00000b18.00000c1c::2011/01/13-16:10:40.943 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
    00000730.00000d30::2011/01/13-16:10:40.943 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
    00000730.00000d30::2011/01/13-16:10:40.943 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node.  Attempting to find a good node...
    00000b18.00000c1c::2011/01/13-16:10:40.943 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
    00000730.000006d8::2011/01/13-16:10:40.943 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
    00000b18.00000c1c::2011/01/13-16:10:40.943 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
    00000730.00000d30::2011/01/13-16:10:40.943 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
    00000730.00000d30::2011/01/13-16:10:40.943 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node.  Attempting to find a good node...
    00000b18.00000c1c::2011/01/13-16:10:40.943 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
    00000730.000006d8::2011/01/13-16:10:40.943 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
    00000730.00000d30::2011/01/13-16:10:47.354 INFO  [RCM] rcm::RcmApi::FailResource: (VM1)
    00000730.00000d40::2011/01/13-16:10:47.354 INFO  [RCM] HandleMonitorReply: FAILURENOTIFICATION for 'VM1', gen(3) result 0.
    00000730.00000d40::2011/01/13-16:10:47.354 INFO  [RCM] TransitionToState(VM1) Online-->ProcessingFailure.
    00000730.00000d40::2011/01/13-16:10:47.354 INFO  [RCM] rcm::RcmGroup::UpdateStateIfChanged: (Available Storage, Online --> Failed)
    00000730.00000d40::2011/01/13-16:10:47.354 ERR   [RCM] rcm::RcmResource::HandleFailure: (VM1)
    00000730.00000d40::2011/01/13-16:10:47.354 INFO  [RCM] resource VM1: failure count: 2, restartAction: 2.
    00000730.00000d40::2011/01/13-16:10:47.354 INFO  [RCM] TransitionToState(VM1) ProcessingFailure-->[WaitingToTerminate to Failed].
    00000730.00000d40::2011/01/13-16:10:47.354 INFO  [RCM] rcm::RcmGroup::UpdateStateIfChanged: (Available Storage, Failed --> Pending)
    00000730.00000d40::2011/01/13-16:10:47.354 INFO  [RCM] TransitionToState(VM1) [WaitingToTerminate to Failed]-->[Terminating to Failed].
    00000bd4.00000eb0::2011/01/13-16:10:47.354 INFO  [RES] Physical Disk <VM1>: Terminate request.
    00000730.00000d40::2011/01/13-16:10:47.354 INFO  [RCM] Resource VM1 is causing group Available Storage to failover.  Posting worker thread.
    00000730.00000d40::2011/01/13-16:10:47.354 INFO  [RCM] rcm::RcmGroup::Failover: (Available Storage)
    00000bd4.00000eb0::2011/01/13-16:10:47.417 WARN  [RES] Physical Disk <VM1>: Terminate: Failed to offline volume \Device\Harddisk3\Partition1, Error 5
    00000b18.00000c1c::2011/01/13-16:10:47.432 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
    00000730.0000076c::2011/01/13-16:10:47.432 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
    00000730.0000076c::2011/01/13-16:10:47.432 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node.  Attempting to find a good node...
    00000b18.00000c1c::2011/01/13-16:10:47.432 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
    00000730.00000d30::2011/01/13-16:10:47.432 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
    00000b18.00000c1c::2011/01/13-16:10:47.432 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
    00000730.0000076c::2011/01/13-16:10:47.432 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
    00000730.0000076c::2011/01/13-16:10:47.432 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node.  Attempting to find a good node...
    00000b18.00000c1c::2011/01/13-16:10:47.432 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
    00000730.00000d30::2011/01/13-16:10:47.432 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
    00000b18.00000c1c::2011/01/13-16:10:47.432 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
    00000730.0000076c::2011/01/13-16:10:47.432 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
    00000730.0000076c::2011/01/13-16:10:47.432 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node.  Attempting to find a good node...
    00000b18.00000c1c::2011/01/13-16:10:47.448 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
    00000730.00000d30::2011/01/13-16:10:47.448 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
    00000b18.00000c1c::2011/01/13-16:10:47.448 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
    00000730.0000076c::2011/01/13-16:10:47.448 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
    00000730.0000076c::2011/01/13-16:10:47.448 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node.  Attempting to find a good node...
    00000b18.00000c1c::2011/01/13-16:10:47.448 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
    00000730.00000d30::2011/01/13-16:10:47.448 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
    00000bd4.00000eb0::2011/01/13-16:10:47.666 INFO  [RES] Physical Disk <VM1>: HardDiskpCloseSVIHandles: Exit
    00000bd4.00000eb0::2011/01/13-16:10:48.259 INFO  [RES] Physical Disk: ReleaseDisk: stop reserve succeeded on device 3 (sig 44423655)
    00000730.0000076c::2011/01/13-16:10:48.368 INFO  [RCM] HandleMonitorReply: TERMINATERESOURCE for 'VM1', gen(4) result 0.
    00000730.0000076c::2011/01/13-16:10:48.368 INFO  [RCM] TransitionToState(VM1) [Terminating to Failed]-->Failed.
    00000730.0000076c::2011/01/13-16:10:48.368 INFO  [RCM] rcm::RcmGroup::UpdateStateIfChanged: (Available Storage, Pending --> Failed)
    00000730.00000d40::2011/01/13-16:10:48.384 WARN  [RCM] Not failing over group Available Storage, failoverCount 1, failover threshold 0, nodeAvailCount 0.
    00000730.00000d40::2011/01/13-16:10:48.384 INFO  [RCM] Will retry online of VM1 in 3600000 milliseconds.
    00000b18.00000c1c::2011/01/13-16:10:48.384 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
    00000730.00000d40::2011/01/13-16:10:48.384 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
    00000730.00000d40::2011/01/13-16:10:48.384 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node.  Attempting to find a good node...
    00000b18.00000c1c::2011/01/13-16:10:48.384 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
    00000730.0000076c::2011/01/13-16:10:48.384 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
    00000b18.00000c1c::2011/01/13-16:10:48.400 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
    00000730.00000d40::2011/01/13-16:10:48.400 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
    00000730.00000d40::2011/01/13-16:10:48.400 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node.  Attempting to find a good node...
    00000b18.00000c1c::2011/01/13-16:10:48.400 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
    00000730.0000076c::2011/01/13-16:10:48.400 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
    00000b18.00000c1c::2011/01/13-16:10:48.400 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
    00000730.00000d40::2011/01/13-16:10:48.400 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
    00000730.00000d40::2011/01/13-16:10:48.400 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node.  Attempting to find a good node...
    00000b18.00000c1c::2011/01/13-16:10:48.400 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
    00000730.0000076c::2011/01/13-16:10:48.400 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
    00000b18.00000c1c::2011/01/13-16:10:48.400 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
    00000730.00000d40::2011/01/13-16:10:48.400 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
    00000730.00000d40::2011/01/13-16:10:48.400 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node.  Attempting to find a good node...
    00000b18.00000c1c::2011/01/13-16:10:48.400 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
    00000730.0000076c::2011/01/13-16:10:48.400 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.

    Thanks Ramzan and Elden for your reply. :) :)
    Issue Faced:
    we have 3 cluster server(2008 enterprise R2). The cluster server gets rebooted and failover happens frequently.As a workaround,we use to unistall Network threat component(symantec) when the issue happens.
    The Cluster validation passes sucessfully for both storage and network.
    I dont find any logs apart from the below.
    00000d38.00000698::2011/04/15-20:04:38.452 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
    00000984.00000d88::2011/04/15-20:04:38.452 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
    00000d38.00000698::2011/04/15-20:04:38.467 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
    00000984.000012f0::2011/04/15-20:04:38.467 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
    00000984.000012f0::2011/04/15-20:04:38.467 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node. 
    Attempting to find a good node...
    We collected the bugcheck analysis report after the reboot. The report as follows
    MM_INTERNAL_CODE:  0
    DEFAULT_BUCKET_ID:  VISTA_DRIVER_FAULT
    BUGCHECK_STR:  0x50
    PROCESS_NAME:  svchost.exe
    CURRENT_IRQL:  0
    STACK_COMMAND:  kb
    FOLLOWUP_IP:
    srvnet!FillSessionInfoBuffer+a2
    fffff880`05472ed2 448b9c2480000000 mov     r11d,dword ptr [rsp+80h]
    SYMBOL_STACK_INDEX:  4
    SYMBOL_NAME:  srvnet!FillSessionInfoBuffer+a2
    FOLLOWUP_NAME:  MachineOwner
    MODULE_NAME: srvnet
    IMAGE_NAME:  srvnet.sys
    DEBUG_FLR_IMAGE_TIMESTAMP:  4c7732f4
    FAILURE_BUCKET_ID:  X64_0x50_srvnet!FillSessionInfoBuffer+a2
    BUCKET_ID:  X64_0x50_srvnet!FillSessionInfoBuffer+a2
    Followup: MachineOwner
    As per microsoft technet, we are planning to deploy the hotfix provided in the article
    http://support.microsoft.com/kb/980135
    Please let us know if another troubleshooting steps needs to be performed. Any solutions please let us know
    Thanks
    Dinesh

  • Cluster Disk Logical Disk Performance Counters Problem

    I have some problem with Oracle cluster. It's a two node Windows server 2008 R2 SP1 cluster with MS Failover cluster. I have SCOM 2012 SP1 with latest OS MP (agent running with local system rights). All other cluster are showing fine performance counters
    but this cluster does only show one cluster disk free space counters. It has 5 cluster disks (MBR formatted). Logical disk performance counters are running fine. So my questions:
    What performance counters does the new "Cluster Disk % and MB free space rule" use? Is it the same old Logical Disk performance counter that collects data for example for drive C:?
    What should I check from this cluster because performance view from my cluster groups are not showing all disks, just one and there are 5 cluster disks?
    Only one Cluster Disk is showing under performance view!
    Health Explorer from Cluster group shows all Cluster disks!

    We have this issue with normal Win server 2008 SQL servers. I can see disks at the Health Explorer view but no collection rules at the performance view. Rule does not have any overrides and it's enabled by default. We use the latest Core OS pack and Cluster
    MP. IS this ANOTHER bug in the MS management pack?

  • Cluster Disk Space Report - DESPERATE

    Please can someone help the worldwide SCOM community and give us a Cluster Disk Report?
    Name your price - I will pay for this.

    Here's how to do it. We have a 2-node cluster, NODE1 and NODE2. The Cluster Name is WATER. We want a report for a file system that's on the cluster called Backups (the M: drive, but the drive letter doesn't matter, as SCOM uses the name of the file system).
    Start with a Performance Detail report. Change your date range to whatever you want. Create a new series. Click on "Add Group" in the "Series Details" section (make sure you pick "Add Group", this is important). Find your cluster name, WATER and choose the
    Windows Computer object.(it's odd, but trust me on this!)
    In the Rule section, in the Search by Name tab, find the "Windows Server Cluster Disks Monitoring" MP. Pick the rule you want to run the report for, click OK. Back in the Rules section, select "Specific Instances" and find the name of your disk you want
    the report for (in our case, "backups").
    Here's what the settings/parameters look like when you're done:
    Click OK and run the report. That's it!
    "Fear disturbs your concentration"

  • Cannot see cluster disk resources after updating Windows base MP

    Hello - we've recently updated the Windows base MP for SCOM 2007 R2.  Once we did that we lost monitoring of our clustered disk resources.  They're discovered but not monitored any more.  Can someone please help!!! TIA
    Mike Zamborini

    Hi,
    From the "Windows Server Operating System Management Pack Guide.docx" - page 42
    Known Issue: Cluster Disks Managed by Third-Party Software are Not Monitored
    Issue: If the Cluster disks are managed by third party software, and they change the resource type to anything other than “Physical Disk”, these disks will be discovered but we do not provide monitoring for these.
    Workaround: There is no workaround currently available. In future, we will be removing discovery for these too.
    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.

  • Cluster Disk Health Reports

    Hi,
    we have installed and configured SCOM to monitor all windows server including Windows cluster. we would like to get the details of the cluster disk health. The information is already available on the dashboard but not as a report.
    we would like to have the status in the reports.
    how to achieve this ?
    BR, Phaixan

    Hi,
    Have you imported the management pack for Windows cluster below?
    http://www.microsoft.com/en-hk/download/details.aspx?id=2268
    Within the management pack, we could find some builtin reports, and if there is no reports that can meet your goal, then we may create report by ourselves. Or we may create views for Cluster Shared Volume Health, and Cluster Shared Volume Disk Capacity.
    Please look into the below blog for more details:
    http://blogs.technet.com/b/kevinholman/archive/2011/09/30/opsmgr-new-base-os-mp-6-0-6956-0-adds-cluster-shared-volume-monitoring-bpa-and-many-changes.aspx
    Regards,
    Yan Li
    Regards, Yan Li

  • Cluster Disk not detect

    Hi Experts, I am installing EHP7 on windows server 2012. Hardware vendor provided the Cluster Disk 2 as I:\ drive But during installation its not showing Cluster disk (I:\) in drop down it is showing local disk c:\,f:\ and g:\. Find the screenshot attached. Please help. Regards, Shivraj

    Hi Experts, I am installing EHP7 on windows server 2012. Hardware vendor provided the Cluster Disk 2 as I:\ drive But during installation its not showing Cluster disk (I:\) in drop down it is showing local disk c:\,f:\ and g:\. Find the screenshot attached. Please help. Regards, Shivraj

  • SCOM cluster disk space monitor

    Hi Guys,
    Can we monitor the cluster disks thresholds by taking the samples as we do in the logical disks??
    Because i cannot see that option coming in the cluster disk space monitor when i am overriding the monitor.There is no option for the number of samples coming in that. I want the monitor to check the utilization three times and if it exceeds then generate
    an alert.
    Thanks
    Ashish

    Hi,
    As you mentioned, threre is no option for the number of samples, you may need to rewrite the MP.
    Please refer to the following links which should be helpful for our question.
    Management Pack Authoring Options
    http://www.code4ward.net/2012/07/06/system-center-2012-operations-manager-authoring-options/
    OpsMgr: MP Update: New Base OS MP 6.0.6972.0 Adds new cluster disks, changes free space monitoring, other fixes
    http://blogs.technet.com/b/kevinholman/archive/2012/06/14/opsmgr-mpupdate-new-base-os-mp-6-0-6972-0-adds-new-cluster-disks-changes-free-space-monitoring-other-fixes.aspx
    Please remember to mark the replies as answers if they help and unmark them if they provide no help. If you have feedback for TechNet Subscriber Support, contact [email protected]

Maybe you are looking for