One VM Cluster Resources Regularly Failing

Hi All,
We run hundreds of Windows and Linux VMs in clustered and non-clustered environments. However, we're having issues with one particular VM that regularly restarts itself. The environment the problem VM is running in is a Windows 2012 R2 cluster.
The event log within the VM provides no BSOD information, the only entry of any note:
The system has rebooted without cleanly shutting down first. This error could be caused if the system stopped responding, crashed, or lost power unexpectedly.
Therefore, I don't believe that the actual OS within the VM (Windows 2008 R2) is crashing.
The cluster log shows only a single entry:
Cluster resource 'Virtual Machine vps.xxxxxx.com' of type 'Virtual Machine' in clustered role 'vps.xxxxxx.com' failed.
Based on the failure policies for the resource and role, the cluster service may try to bring the resource online on this node or move the group to another node of the cluster and then restart it. Check the resource and group state using Failover Cluster
Manager or the Get-ClusterResource Windows PowerShell cmdlet.
How can I debug this? I've added extra RAM to it, and moved it to other nodes and other storage but the problem continues to occur.
Thanks
Will

Hi,
To such issue, it is not an efficient way to work in this community since we may need more resources, for example memory (an application) dump or ETL trace, which is not appropriate to handle in the community. I’d like to suggest that you submit a
service request to MS Professional tech support service so that a dedicated Support Professional can further assist with this request.
Please visit the below link to see the various paid support options that are available to better meet your needs.
http://support.microsoft.com/default.aspx?id=fh;en-us;offerprophone
Best regards,
We
are trying to better understand customer views on social support experience, so your participation in this
interview project would be greatly appreciated if you have time.
Thanks for helping make community forums a great place.

Similar Messages

TES v6.1 want to install a Win Tidal Agent on a Cluster Resource with Fail Over... help

I am working with my Window Admins and they want to know how to install the Win Tidal Agent on a Cluster Resource with Fail Over. Currently running Tidal Master (UNIX) v6.1.0.483
Thanks,
Rich

Please refer to the Agent Installation and Configuration guide.pdf from Cisco. The steps to configure the agents in a cluster have been explained in section Configuring the Agents for a Cluster

Constantly "Cluster resource 'Virtual Machine' in clustered service or application 'SERVER' failed

Hi...
I have an IBM BladeCenter S with 3 blades and an IBM System Storage DS3300 (ISCSI).
In each blade is running Windows Server 2008 R2 with
HYPER-V,
Failover Clustering with Cluster Shared Volumes.
I have observed that many errors occur constantly in "Failover Cluster Manager" and some VM´s are relocated to another blade automatically, however thoses VM´s sometimes no longer responds to network activity.
The errors I have observed in the "Failover Cluster Manager" are:
Source: Microsoft-Windows-FailoverClusting
Event ID: 1069
Description:
Cluster resource 'Virtual Machine' in clustered service or application 'SERVER' failed.
Source: Microsoft-Windows-FailoverClusting
Event ID: 1205
Description:
The Cluster service failed to bring clustered service or application 'SV-DBURAS' completely online or offline. One or more resources may be in a failed state. This may impact the availability of the clustered service or application.
Another error (warning type) that is constantly generated in the SYSTEM events (Mirage is the storage name):
Source:        ds4dsm
Event ID:      769
Description:
IO error being retried via alternate controller Mirage:1
Source:        ds4dsm
Event ID:      10
Description:
Mirage:0 Failover command issued.
Source:        ds4dsm
Event ID:      801
Description:
Failover succeeded to Mirage:0.
Thank you in advance any help!

Hi,
I suggest referring to the following articles:
http://technet.microsoft.com/en-us/library/cc756225(WS.10).aspx
http://technet.microsoft.com/en-us/library/cc773525(WS.10).aspx
Tim Quan - MSFT

Cluster resource 'SQL Server' in Resource Group 'MSSQL' failed.

Hi All,
Last week we face problem on SQL server 2005 Cluster server.
SQL cluster was down with below issue.
Event 1069 : Cluster resource 'SQL Server' in Resource Group 'MSSQL' failed.
Event 19019 : [sqsrvres] CheckServiceAlive: Service is dead
[sqsrvres] OnlineThread: service stopped while waiting for QP.
[sqsrvres] OnlineThread: Error 1 bringing resource online
Kindly any one provide resolution for my above issue.

I have checked in event viewer Application error side error:
Event 19019 : [sqsrvres]
CheckServiceAlive: Service is dead
[sqsrvres] OnlineThread: service stopped while waiting for QP.
[sqsrvres] OnlineThread: Error 1 bringing resource online
System error :
Event 1069 : Cluster resource 'SQL Server' in Resource
Group 'MSSQL' failed.
Before this no error is there in event viewer

Windows server 2012 failover cluster error: Cluster resource 'Virtual Machine Configuration ... of type 'Virtual machine configuration in clustered role ... failed.

I have two windows 2012 host server that are clustered using windows failover cluster feature. Each server is hosting four VMs. When migrating from Host2 to Host1, the migration failed with the following error:
Cluster resource 'Virtual Machine Configuration SCPCSQLSRV01' of type 'Virtual Machine Configuration' in clustered role 'SCPCSQLSRV01' failed. The error code was '0x569' ('Logon failure: the user has not been granted the requested logon type at this computer.').
When this happens, the VM that I was migrating can no longer be started even on the original host. The only remedy is to restart the host server.
Any suggestion on resolving this problem?
Thanks
Ikad

Thanks. The article referred to above gives the solution to my issue. There is a group policy that is applied to the OU where the host servers were placed. Doing gpupdate /force temporarily removes the problem. Unfortunately the NT Virtual Machine\Virtual
Machines account is a special account that cannot be added like other accounts and granted the log on as a service right. The thread
http://social.technet.microsoft.com/Forums/en-US/winserverhyperv/thread/d56f2eae-726e-409a-8813-670a406593e8 contains how it can be added which is by creating a group and running the command
Net localgroup VMTest “NT Virtual Machine\Virtual Machines” /add
to add it to a local group VMTest. VMTest is then assigned the right to log on as a service.
Ikad

Cluster resource 'Analysis Services' of type 'Generic Service' in clustered role 'SQL Server' failed.

Windows Server 2012 R2
SQL Server 2012
After a recent cluster failover from node 1 to node 2, the Analysis Services role is in a failed state, with the service stopped. When attempting to start the service, there are two error messages captured in Failover Cluster Manager:
Log Name:      System
Source:        Microsoft-Windows-FailoverClustering
Date:          4/10/2014 11:48:49 AM
Event ID:      1042
Task Category: Generic Service Resource
Level:         Error
Keywords:
User:          SYSTEM
Computer:      HQ-HASQL-1.sbgnet.int
Description:
Generic service 'Analysis Services (HASQL)' failed with error '1067'. Please examine the application event log.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
    <Provider Name="Microsoft-Windows-FailoverClustering" Guid="{BAF908EA-3421-4CA9-9B84-6689B8C6F85F}" />
    <EventID>1042</EventID>
    <Version>0</Version>
    <Level>2</Level>
    <Task>16</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8000000000000000</Keywords>
    <TimeCreated SystemTime="2014-04-10T15:48:49.752168200Z" />
    <EventRecordID>26212</EventRecordID>
    <Correlation />
    <Execution ProcessID="9036" ThreadID="14748" />
    <Channel>System</Channel>
    <Computer>HQ-HASQL-1.sbgnet.int</Computer>
    <Security UserID="S-1-5-18" />
</System>
<EventData>
    <Data Name="ResourceName">Analysis Services (HASQL)</Data>
    <Data Name="Status">1067</Data>
</EventData>
</Event>
Log Name:      System
Source:        Microsoft-Windows-FailoverClustering
Date:          4/10/2014 11:48:49 AM
Event ID:      1069
Task Category: Resource Control Manager
Level:         Error
Keywords:
User:          SYSTEM
Computer:      HQ-HASQL-1.sbgnet.int
Description:
Cluster resource 'Analysis Services (HASQL)' of type 'Generic Service' in clustered role 'SQL Server (HASQL)' failed.
Based on the failure policies for the resource and role, the cluster service may try to bring the resource online on this node or move the group to another node of the cluster and then restart it. Check the resource and group state using Failover Cluster
Manager or the Get-ClusterResource Windows PowerShell cmdlet.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
    <Provider Name="Microsoft-Windows-FailoverClustering" Guid="{BAF908EA-3421-4CA9-9B84-6689B8C6F85F}" />
    <EventID>1069</EventID>
    <Version>1</Version>
    <Level>2</Level>
    <Task>3</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8000000000000000</Keywords>
    <TimeCreated SystemTime="2014-04-10T15:48:49.752168200Z" />
    <EventRecordID>26213</EventRecordID>
    <Correlation />
    <Execution ProcessID="6464" ThreadID="9076" />
    <Channel>System</Channel>
    <Computer>HQ-HASQL-1.sbgnet.int</Computer>
    <Security UserID="S-1-5-18" />
</System>
<EventData>
    <Data Name="ResourceName">Analysis Services (HASQL)</Data>
    <Data Name="ResourceGroup">SQL Server (HASQL)</Data>
    <Data Name="ResTypeDll">Generic Service</Data>
</EventData>
</Event>
With just these generic error messages being present, this has been difficult to diagnose. Some research has yielded possible resolutions of the Event Viewer log being full, .NET corruption, missing registry entries, but none of those seem to be the issue
(Event Viewer logs cleared, Analysis services is working on the same physical servers in a different cluster, and the registry entries was only a supported issue for SQL Server 2008 and 2008 R2).
Any help would be greatly appreciated.

Bring up Configuration Manager, look at binary path for SSAS. Make sure BOTH folders exist. Sometimes with failovers mappings get screwed up.

Cluster Resource Failed in Clustered Role

Hello All.
We’re running our VMs on the Cluster of Hyper-V hosts of Windows Server 2012 R2.
We frequently encounter the following error in the Cluster Events:
“Cluster resource 'Virtual Machine VM123' of type 'Virtual Machine' in clustered role 'VM123' failed.
Based on the failure policies for the resource and role, the cluster service may try to bring the resource online on this node or move the group to another node of the cluster and then restart
it. Check the resource and group state using Failover Cluster Manager or the Get-ClusterResource Windows PowerShell cmdlet.”
VM123 is the hostname of VM. We encounter this error for most of the VMs, however the VMs keep running in the cluster.
Anybody please guide me what this error mean.
Please help and advise.
Regards,
Hasan Bin Hasib

Hi Hasan Bin Hasib,
Since in your description there don’t have others detail clue, in your current information we can get the VM123 resource at failed status, you can first run the cluster validation
to verify whether your cluster configuration is correct and install the Recommended hotfixes and updates first.
Recommended hotfixes and updates for Windows Server 2012 R2-based failover clusters
https://support.microsoft.com/en-us/kb/2920151?wa=wsignin1.0
Understanding how Failover Clustering Recovers from Unresponsive Resources
http://blogs.msdn.com/b/clustering/archive/2013/01/24/10388009.aspx
I’m glad to be of help to you!
Please remember to mark the replies as answers if they help and unmark them if they provide no help. If you have feedback for TechNet Support, contact [email protected]

Cluster resource ' Disk Name' of type 'Physical Disk' in clustered role 'Role Name' failed.

We have been observing issues with our file Cluster (Windows Server 2012 R2 Std Clustered with 2 Nodes) where File Server gets
unresponsive for SMB access request event id 30809 in Microsoft-Windows-SMBClient/Connectivity is observed
and when we try to failover the role clustered disks fail to get offline with an error in event id 1069 Cluster resource ' Disk Name' of type 'Physical Disk' in clustered role 'Role Name' failed, we have to force fully reboot the node which faces this
issue. It works properly for a week and again we get the same issue, this happens with all the disks in different file server roles.
Regards Ajinkya Ghare MCITP-Server Administrator | MCTS

we didn't found any thing in the cluster logs, in the WitnessClientAdmin logs we found errors related to failed registration
Witness Client failed registration request for \\fileserver\sharename with error (The request is not supported.)
Regards Ajinkya Ghare MCITP-Server Administrator | MCTS

Cluster resource 'SQL Network Name (SQLCLUS1)' of type 'Network Name' in clustered role 'SQL Server (DB1)' failed.

Cluster resource 'SQL Network Name (SQLCLUS1)' of type 'Network Name' in clustered role 'SQL Server (DB1)' failed.
Based on the failure policies for the resource and role, the cluster service may try to bring the resource online on this node or move the group to another node of the cluster and then restart it. Check the resource and group state using Failover Cluster
Manager or the Get-ClusterResource Windows PowerShell cmdlet.
I keep getting this error message. Can someone please help. Thank You.
Kranp.

Hi Kranp,
As the issue is more related to Windows Server high availability
, I recommend you post the question in the
Windows Server High Availability (Clustering)
forum. It is appropriate and more experts will assist you.
Besides, there are similar threads regarding to the above error for your reference.
2012 Cluster service name failing
SQL 2012 Failover Cluster - unable to start because
of 'Network Name' failed
Issues with resource creation on W2K12 SQL
failover cluster, confirm procedures
Thanks,
Lydia Zhang
Lydia Zhang
TechNet Community Support

Cluster resource SAPCCM4X.00' in Resource Group 'SAP ABC' failed

Hallo.
I installed SAPCCM4X following the http://www.sdn.sap.com/irj/scn/go/portal/prtroot/docs/library/uuid/f0bcedaa-dfb1-2d10-b3a9-c140aff84dc2?quicklink=index&overridelayout=true
It is registered successfully, but it fails every 5 minutes.
I see in the event viewer
Cluster resource SAPCCM4X.00' in Resource Group 'SAP ABC' failed.
What could I check?
Thanks for your help .
Mario

Hallo.
I followed the note.
The problem is when the CCMS check the status , every 5 minutes.
The registrazion was successful.
I set ccms/enable_agent = -1
I set AgentLocalHost virtualhostname
But I have the same problem.
I don't know what to check.

SCVMM 2012 R2 RU3 - Live storage migration failing with error 12711 if cluster resource name is not in form of "SCVMM vmname "

At the end of live storage migration in SCVMM 2012 R2 RU3 getting error:
Error (12711)
VMM cannot complete the WMI operation on the server (hypervhost1.domain.local) because of an error: [MSCluster_Resource.Name="SCVMM test-vm"] The cluster resource could not be found.
The cluster resource could not be found (0x138F)
Recommended Action
Resolve the issue and then try the operation again.
Storage migration actually succeeds and after Repair/Ignore and Refresh, the VM shows up ok in VMM, with the VHD in new location.
If VM is offline, storage migration always succeeds. Live (non-storage) migration always succeeds.
I noticed in the error message that it searches for cluster resource name "SCVMM test-vm" and for some VMs I've got different cluster resource names.
Like for this VM cluster resource names are:
Virtual Machine test-vm
Virtual Machine Configuration test-vm
Furthermore, those resource names are showing up correctly in [VirtualManagerDB].[dbo].[tbl_WLC_VMInstance] table, columns VMResource and VMConfigResource.
Anyone knows why SCVMM keeps searching for "SCVMM test-vm" and how to fix this without recreating cluster resources?

Ok, you have the same problem and same configuration that this post:
https://social.technet.microsoft.com/Forums/systemcenter/en-US/853c021f-dd0a-4d88-a7c1-72bb8d4d0591/hyperv-cluster-live-migration-does-not-work-anymore-after-ur5-installation?forum=virtualmachingmgrhyperv#b5ae914f-7b52-4cea-86ef-a64ce4b32bb0
This problem occurs will old and new vm?

The cluster resource 'SQL Server' could not be brought online error while in completion stage of sql server 2012 failover installation

While installing sql server 2012 with failover clustering I got the following error:
The cluster resource 'SQL Server' could not be brought online. Error: There was a failure to call cluster code from a provider. Exception message: Generic failure. Status code: 5942. Description: The resource failed to come online due to the failure of one
or more provider resources.
The two nodes of the sql server failover have been successfully prepared. The error came while running the advance cluster completion option.
In the clustered events in failover cluster manager the following message is displayed:
Cluster resource 'SQL Network Name (CPCTestSQLClus)' of type 'Network name' in clustered role 'SQL Server (MSSQLSERVER)' failed.
I have done various checking and the issue remained. I tried to start the sql

Please find below part of the content of the cluster log. I hope this is helpful in resolving the issue:
00000998.000010e8::2013/05/08-20:45:00.233 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: Online called for resource SQL Network Name (CPCTestSQLClus)
000005a4.0000050c::2013/05/08-20:45:00.233 INFO [RCM] HandleMonitorReply: ONLINERESOURCE for 'SQL Network Name (CPCTestSQLClus)', gen(6) result 997/0.
000005a4.0000050c::2013/05/08-20:45:00.233 INFO [RCM] Res SQL Network Name (CPCTestSQLClus): OnlineCallIssued -> OnlinePending( StateUnknown )
000005a4.0000050c::2013/05/08-20:45:00.233 INFO [RCM] TransitionToState(SQL Network Name (CPCTestSQLClus)) OnlineCallIssued-->OnlinePending.
00000998.000012f0::2013/05/08-20:45:00.233 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: Entering Online thread
000005a4.0000050c::2013/05/08-20:45:00.233 INFO [GEM] Sending 1 messages as a batched GEM message
00000998.000012f0::2013/05/08-20:45:00.233 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: Initializing config info
00000998.000012f0::2013/05/08-20:45:00.248 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: Skipping initialization of 'Identity' module because netname is not yet created
00000998.000012f0::2013/05/08-20:45:00.248 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: Performing actual online of resource.
00000998.000012f0::2013/05/08-20:45:00.248 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: Obtaining IP info for resource SQL IP Address 1 (CPCTestSQLClus)
00000998.000012f0::2013/05/08-20:45:00.248 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: Using provider SQL IP Address 1 (CPCTestSQLClus), ip address 10.144.166.160, mask 255.255.255.0, prefix length 24
00000998.000012f0::2013/05/08-20:45:00.248 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: Resource has 1 IPs
00000998.000012f0::2013/05/08-20:45:00.248 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: IP: Type Ipv4, Address 10.144.166.160:~0~, Prefix 10.144.166.160/24, Online true, Transport \Device\NetBt_If3
00000998.000012f0::2013/05/08-20:45:00.248 INFO [RES] Network Name: Agent: Initializing (082d8ef5-62a8-4fee-acbc-e89b769e9181,Configuration)
00000998.000012f0::2013/05/08-20:45:00.248 INFO [RES] Network Name: Agent: Sending request Netname/InitializeIndirect to NN:Agent
00000998.000010e8::2013/05/08-20:45:00.248 INFO [RES] Network Name: Agent: OnInitialize (082d8ef5-62a8-4fee-acbc-e89b769e9181,Configuration)
00000998.000010e8::2013/05/08-20:45:00.248 INFO [RES] Network Name: Agent: CanModuleBeInitializedImp - Module can be initialized, current state Offline
00000998.000010e8::2013/05/08-20:45:00.248 INFO [RES] Network Name: Agent: Sending Initialize to module NN:082d8ef5-62a8-4fee-acbc-e89b769e9181:Configuration
00000998.000010e8::2013/05/08-20:45:00.248 INFO [RES] Network Name: Agent: Sending request Netname/Initialize to NN:082d8ef5-62a8-4fee-acbc-e89b769e9181:Configuration
00000998.000010e8::2013/05/08-20:45:00.248 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: Configuration: Waiting for ongoing slow strand to complete (if any)
00000998.000010e8::2013/05/08-20:45:00.248 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: Configuration: Signaling main strand to initialize module
00000998.000010e8::2013/05/08-20:45:00.248 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: Configuration: Calling initialize of the configuration implementation
000005a4.00001350::2013/05/08-20:45:00.248 INFO [GEM] Sending 1 messages as a batched GEM message
00000998.000010e8::2013/05/08-20:45:00.248 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: Configuration: Setting 'StatusKerberos' in clusdb returned status 0
00000998.000010e8::2013/05/08-20:45:00.248 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: Configuration: Creating a New AD NetName (type Singleton)...
00000998.000010e8::2013/05/08-20:45:00.248 INFO [RES] Network Name: Agent: Initializing (082d8ef5-62a8-4fee-acbc-e89b769e9181,AccountAD)
00000998.000010e8::2013/05/08-20:45:00.248 INFO [RES] Network Name: Agent: Sending request Netname/InitializeIndirect to NN:Agent
00000998.000010e8::2013/05/08-20:45:00.248 INFO [RES] Network Name: Agent: OnInitialize (082d8ef5-62a8-4fee-acbc-e89b769e9181,AccountAD)
00000998.000010e8::2013/05/08-20:45:00.248 INFO [RES] Network Name: Agent: CanModuleBeInitializedImp - Module can be initialized, current state Closing
00000998.000010e8::2013/05/08-20:45:00.248 INFO [RES] Network Name: Agent: Sending Initialize to module NN:082d8ef5-62a8-4fee-acbc-e89b769e9181:AccountAD
00000998.000010e8::2013/05/08-20:45:00.248 INFO [RES] Network Name: Agent: Sending request Netname/Initialize to NN:082d8ef5-62a8-4fee-acbc-e89b769e9181:AccountAD
00000998.000010e8::2013/05/08-20:45:00.248 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: AccountAD: Waiting for ongoing slow strand to complete (if any)
00000998.000010e8::2013/05/08-20:45:00.248 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: AccountAD: Signaling main strand to initialize module
00000998.000010e8::2013/05/08-20:45:00.248 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: AccountAD: Initializing Name: CPCTestSQLClus, NetbiosName: CPCTESTSQLCLUS, Type: Singleton, Created: false
00000998.000010e8::2013/05/08-20:45:00.248 INFO [RES] Network Name: [NNLIB] GetCreatingDC - name CPCTestSQLClus, lookup flags = 1
00000998.000010e8::2013/05/08-20:45:00.452 INFO [RES] Network Name: [NNLIB] GetDCWithFlags - using DC
\\MY01DC03.Domain.net, domain name Domain.net, IsReadOnly 0, Object exists 0
00000998.000010e8::2013/05/08-20:45:00.452 INFO [RES] Network Name: [NN] Setting crypto access members for encrypt. New container = false.
000005a4.00001350::2013/05/08-20:45:00.452 INFO [GEM] Sending 1 messages as a batched GEM message
00000998.000010e8::2013/05/08-20:45:00.452 INFO [RES] Network Name: Agent: Sending request Netname/LastDcChange to NN:082d8ef5-62a8-4fee-acbc-e89b769e9181:Configuration
00000998.0000075c::2013/05/08-20:45:00.452 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: Configuration: LastDc Changed, configOnly: 1
000005a4.00001350::2013/05/08-20:45:00.452 INFO [GEM] Sending 1 messages as a batched GEM message
00000998.0000075c::2013/05/08-20:45:00.468 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: Configuration: Setting last DC in clusdb returned status 0
00000998.0000075c::2013/05/08-20:45:00.468 INFO [RES] Network Name: [NN] got sync reply: 0
00000998.000010e8::2013/05/08-20:45:00.468 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: AccountAD: LastDc Changed (configOnly true), result: 0
00000998.000010e8::2013/05/08-20:45:00.468 INFO [RES] Network Name: Agent: Sending request Netname/PasswordChange to NN:082d8ef5-62a8-4fee-acbc-e89b769e9181:Configuration
00000998.0000075c::2013/05/08-20:45:00.468 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: Configuration: Password Changed, configOnly: 1
000005a4.00001350::2013/05/08-20:45:00.468 INFO [GEM] Sending 1 messages as a batched GEM message
00000998.0000075c::2013/05/08-20:45:00.468 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: Configuration: Setting password in clusdb returned status 0
00000998.0000075c::2013/05/08-20:45:00.468 INFO [RES] Network Name: [NN] got sync reply: 0
00000998.000010e8::2013/05/08-20:45:00.468 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: AccountAD: PasswordChanged (configOnly true), result: 0
00000998.000010e8::2013/05/08-20:45:00.468 INFO [RES] Network Name: [NN] Setting crypto access members for decrypt. New container = false.
00000998.00000ab8::2013/05/08-20:45:00.636 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: Getting Read/Write private properties
00000998.00000ab8::2013/05/08-20:45:00.647 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: Getting Read only private properties
00000998.0000075c::2013/05/08-20:45:00.784 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: Getting Read/Write private properties
00000998.00000ab8::2013/05/08-20:45:00.815 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: Getting Read only private properties
00000998.00000ab8::2013/05/08-20:45:00.815 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: Getting Read/Write private properties
00000998.00000ab8::2013/05/08-20:45:00.830 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: Getting Read only private properties
00000998.00000ab8::2013/05/08-20:45:00.830 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: Getting Read/Write private properties
00000998.00000ab8::2013/05/08-20:45:00.830 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: Getting Read only private properties
00000998.000010e8::2013/05/08-20:45:00.924 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: AccountAD: OU name for VCO is OU=NewComputers,OU=WORKSTATIONS,DC=Domain,DC=net
00000998.000010e8::2013/05/08-20:45:00.924 INFO [RES] Network Name: Agent: Sending request Netname/GetToken to NN:c03fb105-9a50-4d00-8bee-afaf693f7efc:Identity
00000998.00000ab8::2013/05/08-20:45:00.924 INFO [RES] Network Name <Cluster Name>: Identity: Module is not yet initialized. Trying to obtain a new token.
00000998.0000075c::2013/05/08-20:45:00.924 INFO [RES] Network Name <Cluster Name>: Identity: Obtaining new token
00000998.0000075c::2013/05/08-20:45:00.924 INFO [RES] Network Name: [NN] Setting crypto access members for decrypt. New container = false.
00000998.0000075c::2013/05/08-20:45:00.924 INFO [RES] Network Name: [NNLIB] Priming local KDC cache to
\\MY01DC02.Domain.net for domain Domain.net
00000998.0000075c::2013/05/08-20:45:00.924 INFO [RES] Network Name: [NNLIB] PopulateKerbKDCLookupCache - DC flags 0
00000998.0000075c::2013/05/08-20:45:00.940 INFO [RES] Network Name: [NNLIB] LsaCallAuthenticationPackage success with a request of size 94, result size 0 (status: 0, subStatus: 0)
00000998.0000075c::2013/05/08-20:45:00.940 INFO [RES] Network Name: [NNLIB] Priming local KDC cache to
\\MY01DC02.Domain.net for domain label Domain
00000998.0000075c::2013/05/08-20:45:00.940 INFO [RES] Network Name: [NNLIB] LsaCallAuthenticationPackage success with a request of size 86, result size 0 (status: 0, subStatus: 0)
00000998.0000075c::2013/05/08-20:45:01.018 WARN [RES] Network Name: [NNLIB] LogonUserEx fails for user CPCTestWinClus$: 1326 (useSecondaryPassword: 0)
00000998.0000075c::2013/05/08-20:45:01.096 WARN [RES] Network Name: [NNLIB] LogonUserEx fails for user CPCTestWinClus$: 1326 (useSecondaryPassword: 1)
00000998.0000075c::2013/05/08-20:45:01.096 INFO [RES] Network Name: [NNLIB] Logon failed for user CPCTestWinClus$ (Error 1326), DC
\\MY01DC02.Domain.net, domain Domain.net
00000998.0000075c::2013/05/08-20:45:01.096 INFO [RES] Network Name <Cluster Name>: Identity: Obtaining Windows Token for Name: CPCTestWinClus, SamName: CPCTestWinClus$, Type: Singleton, Result: 1326, LastDC:
\\MY01DC02.Domain.net
00000998.0000075c::2013/05/08-20:45:01.096 INFO [RES] Network Name <Cluster Name>: Identity: Slow Operation, FinishWithReply: 1326
00000998.0000075c::2013/05/08-20:45:01.096 INFO [RES] Network Name <Cluster Name>: Identity: InternalReplyHandler with event: 1326
00000998.0000075c::2013/05/08-20:45:01.096 INFO [RES] Network Name <Cluster Name>: Identity: End of Slow Operation, state: Error/Idle, prevWorkState: Idle
00000998.00000ab8::2013/05/08-20:45:01.096 WARN [RES] Network Name <Cluster Name>: Identity: Get Token Request, currently doesnt have a token!
00000998.00000ab8::2013/05/08-20:45:01.096 INFO [RES] Network Name: [NN] got sync reply: 0
00000998.000010e8::2013/05/08-20:45:01.096 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: AccountAD: End of Slow Operation, state: Initializing/Writing, prevWorkState: Writing
00000998.000010e8::2013/05/08-20:45:01.096 WARN [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: AccountAD: Slow operation has exception ERROR_INVALID_HANDLE(6)' because of '::ImpersonateLoggedOnUser( GetToken() )'
00000998.000010e8::2013/05/08-20:45:01.096 INFO [RES] Network Name: Agent: OnInitializeReply, Failure on (082d8ef5-62a8-4fee-acbc-e89b769e9181,AccountAD): 6
00000998.000010e8::2013/05/08-20:45:01.096 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: Configuration: InitializeReplyCreation of NetName (type Singleton), result: 6, IsCanceled: false
000005a4.00000cd8::2013/05/08-20:45:01.096 INFO [GEM] Sending 1 messages as a batched GEM message
00000998.00000ab8::2013/05/08-20:45:01.096 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: Getting Read/Write private properties
00000998.000010e8::2013/05/08-20:45:01.096 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: Configuration: Setting 'StatusKerberos' in clusdb returned status 0
00000998.000010e8::2013/05/08-20:45:01.096 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: Configuration: Deleting ResourceData, CreatingDC, ObjectGUID for a newly created netname from cluster database
000005a4.00000cd8::2013/05/08-20:45:01.096 INFO [GEM] Sending 1 messages as a batched GEM message
000005a4.00000cd8::2013/05/08-20:45:01.112 INFO [GEM] Sending 1 messages as a batched GEM message
00000998.00000ab8::2013/05/08-20:45:01.112 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: Getting Read only private properties
00000998.000010e8::2013/05/08-20:45:01.112 INFO [RES] Network Name: Agent: OnInitializeReply, Failure on (082d8ef5-62a8-4fee-acbc-e89b769e9181,Configuration): 6
00000998.000010e8::2013/05/08-20:45:01.112 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: SyncReplyHandler Configuration, result: 6
00000998.000012f0::2013/05/08-20:45:01.112 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: PerformOnline - Initialization of Configuration module finished with result: 6
00000998.000012f0::2013/05/08-20:45:01.112 ERR   [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: Online thread Failed: ERROR_SUCCESS(0)' because of 'Initializing netname configuration for SQL Network Name (CPCTestSQLClus) failed with
error 6.'
00000998.000012f0::2013/05/08-20:45:01.112 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: All resources offline. Cleaning up.
00000998.000012f0::2013/05/08-20:45:01.112 ERR   [RHS] Online for resource SQL Network Name (CPCTestSQLClus) failed.
000005a4.0000050c::2013/05/08-20:45:01.112 WARN [RCM] HandleMonitorReply: ONLINERESOURCE for 'SQL Network Name (CPCTestSQLClus)', gen(6) result 5018/0.
000005a4.0000050c::2013/05/08-20:45:01.112 INFO [RCM] Res SQL Network Name (CPCTestSQLClus): OnlinePending -> ProcessingFailure( StateUnknown )
000005a4.0000050c::2013/05/08-20:45:01.112 INFO [RCM] TransitionToState(SQL Network Name (CPCTestSQLClus)) OnlinePending-->ProcessingFailure.
000005a4.0000050c::2013/05/08-20:45:01.112 ERR   [RCM] rcm::RcmResource::HandleFailure: (SQL Network Name (CPCTestSQLClus))
000005a4.0000050c::2013/05/08-20:45:01.112 INFO [RCM] resource SQL Network Name (CPCTestSQLClus): failure count: 6, restartAction: 2 persistentState: 1.
000005a4.00000cd8::2013/05/08-20:45:01.112 INFO [GEM] Sending 1 messages as a batched GEM message
000005a4.0000050c::2013/05/08-20:45:01.112 INFO [RCM] Greater than restartPeriod time has elapsed since first failure of SQL Network Name (CPCTestSQLClus), resetting failureTime and failureCount.
000005a4.0000050c::2013/05/08-20:45:01.112 INFO [RCM] Will queue immediate restart (500 milliseconds) of SQL Network Name (CPCTestSQLClus) after terminate is complete.
000005a4.0000050c::2013/05/08-20:45:01.112 INFO [RCM] Res SQL Network Name (CPCTestSQLClus): ProcessingFailure -> WaitingToTerminate( DelayRestartingResource )
000005a4.0000050c::2013/05/08-20:45:01.112 INFO [RCM] TransitionToState(SQL Network Name (CPCTestSQLClus)) ProcessingFailure-->[WaitingToTerminate to DelayRestartingResource].
000005a4.0000050c::2013/05/08-20:45:01.112 INFO [RCM] Res SQL Network Name (CPCTestSQLClus): [WaitingToTerminate to DelayRestartingResource] -> Terminating( DelayRestartingResource )
000005a4.0000050c::2013/05/08-20:45:01.112 INFO [RCM] TransitionToState(SQL Network Name (CPCTestSQLClus)) [WaitingToTerminate to DelayRestartingResource]-->[Terminating to DelayRestartingResource].
00000998.00000ab8::2013/05/08-20:45:01.112 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: Terminate called for resource SQL Network Name (CPCTestSQLClus)
00000998.00000ab8::2013/05/08-20:45:01.112 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: Entering Offline thread
00000998.00000ab8::2013/05/08-20:45:01.112 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: Performing actual offline of resource.
00000998.00000ab8::2013/05/08-20:45:01.112 INFO [RES] Network Name: Agent: Closing (082d8ef5-62a8-4fee-acbc-e89b769e9181,AdminShare)
00000998.00000ab8::2013/05/08-20:45:01.112 INFO [RES] Network Name: Agent: Sending request Netname/CloseIndirect to NN:Agent
00000998.000010e8::2013/05/08-20:45:01.112 INFO [RES] Network Name: Agent: OnClose (082d8ef5-62a8-4fee-acbc-e89b769e9181,AdminShare)
00000998.000010e8::2013/05/08-20:45:01.112 INFO [RES] Network Name: Agent: Sending request Netname/Close to NN:082d8ef5-62a8-4fee-acbc-e89b769e9181:AdminShare
00000998.000010e8::2013/05/08-20:45:01.112 ERR   [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: AdminShare: OnCloseBase, Error Already Closing, previous state: Closing/Ending
00000998.000010e8::2013/05/08-20:45:01.112 INFO [RES] Network Name: Agent: OnCloseReply (082d8ef5-62a8-4fee-acbc-e89b769e9181,AdminShare) result: 1904
00000998.000010e8::2013/05/08-20:45:01.112 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: SyncReplyHandler Configuration, result: 1904
00000998.00000ab8::2013/05/08-20:45:01.112 INFO [RES] Network Name: Agent: Closing (082d8ef5-62a8-4fee-acbc-e89b769e9181,Configuration)
00000998.00000ab8::2013/05/08-20:45:01.112 INFO [RES] Network Name: Agent: Sending request Netname/CloseIndirect to NN:Agent
000005a4.00000cd8::2013/05/08-20:45:01.112 INFO [RCM-rbtr] giving default token to group SQL Server (MSSQLSERVER)
000005a4.00000cd8::2013/05/08-20:45:01.112 INFO [RCM-rbtr] giving default token to group SQL Server (MSSQLSERVER)
000005a4.00000cd8::2013/05/08-20:45:01.112 INFO [RCM-rbtr] giving default token to group SQL Server (MSSQLSERVER)
000005a4.00000cd8::2013/05/08-20:45:01.112 INFO [RCM-rbtr] giving default token to group SQL Server (MSSQLSERVER)
00000998.0000075c::2013/05/08-20:45:01.112 INFO [RES] Network Name: Agent: OnClose (082d8ef5-62a8-4fee-acbc-e89b769e9181,Configuration)
00000998.0000075c::2013/05/08-20:45:01.112 INFO [RES] Network Name: Agent: Sending request Netname/Close to NN:082d8ef5-62a8-4fee-acbc-e89b769e9181:Configuration
00000998.0000075c::2013/05/08-20:45:01.112 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: Configuration: Canceling work, state: Closing/Idle
00000998.0000075c::2013/05/08-20:45:01.112 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: Configuration: OnCloseBase, previous state: Initializing/Idle
00000998.0000075c::2013/05/08-20:45:01.112 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: Configuration: Closing... (PreviousState: Initializing, Created: false, Type: Singleton)
00000998.0000075c::2013/05/08-20:45:01.112 INFO [RES] Network Name: Agent: Closing (082d8ef5-62a8-4fee-acbc-e89b769e9181,AccountAD)
00000998.0000075c::2013/05/08-20:45:01.112 INFO [RES] Network Name: Agent: Sending request Netname/CloseIndirect to NN:Agent
00000998.000010e8::2013/05/08-20:45:01.112 INFO [RES] Network Name: Agent: OnClose (082d8ef5-62a8-4fee-acbc-e89b769e9181,AccountAD)
00000998.000010e8::2013/05/08-20:45:01.112 INFO [RES] Network Name: Agent: Sending request Netname/Close to NN:082d8ef5-62a8-4fee-acbc-e89b769e9181:AccountAD
00000998.000010e8::2013/05/08-20:45:01.112 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: AccountAD: Canceling work, state: Closing/Idle
00000998.000010e8::2013/05/08-20:45:01.112 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: AccountAD: OnCloseBase, previous state: Initializing/Idle
00000998.000010e8::2013/05/08-20:45:01.112 INFO [RES] Network Name: Agent: OnCloseReply (082d8ef5-62a8-4fee-acbc-e89b769e9181,AccountAD) result: 0
00000998.000010e8::2013/05/08-20:45:01.112 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: Configuration: Finish Closing NetName (type Singleton) Module AccountAD, result 0, remaining... 0 (0)
00000998.000010e8::2013/05/08-20:45:01.112 INFO [RES] Network Name: Agent: -- listing 3 instances ---------------------------------------------
00000998.000010e8::2013/05/08-20:45:01.112 INFO [RES] Network Name: Agent: --- 7 modules for instance: 082d8ef5-62a8-4fee-acbc-e89b769e9181:
00000998.000010e8::2013/05/08-20:45:01.112 INFO [RES] Network Name: Agent: ---- Module: Configuration with states: Offline/Ending (BeingBorn)
00000998.000010e8::2013/05/08-20:45:01.112 INFO [RES] Network Name: Agent: ---- Module: AccountAD with states: Closing/Ending (BeingBorn)
00000998.000010e8::2013/05/08-20:45:01.112 INFO [RES] Network Name: Agent: ---- Module: Client with states: Offline/Idle (BeingBorn)
00000998.0000075c::2013/05/08-20:45:01.112 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: Configuration: Closed
00000998.000010e8::2013/05/08-20:45:01.112 INFO [RES] Network Name: Agent: ---- Module: Identity with states: Offline/Idle (BeingBorn)
00000998.000010e8::2013/05/08-20:45:01.112 INFO [RES] Network Name: Agent: ---- Module: Dns with states: Offline/Idle (BeingBorn)
00000998.0000075c::2013/05/08-20:45:01.112 INFO [RES] Network Name: Agent: OnCloseReply (082d8ef5-62a8-4fee-acbc-e89b769e9181,Configuration) result: 0
00000998.000010e8::2013/05/08-20:45:01.112 INFO [RES] Network Name: Agent: ---- Module: Netbios with states: Offline/Idle (BeingBorn)
00000998.0000075c::2013/05/08-20:45:01.112 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: SyncReplyHandler Configuration, result: 0
00000998.000010e8::2013/05/08-20:45:01.112 INFO [RES] Network Name: Agent: ---- Module: AdminShare with states: Closing/Ending (BeingBorn)
00000998.000010e8::2013/05/08-20:45:01.112 INFO [RES] Network Name: Agent: --- 7 modules for instance: 43dec41b-661c-42a8-ae23-b635d7ab11f5:
00000998.000010e8::2013/05/08-20:45:01.112 INFO [RES] Network Name: Agent: ---- Module: Configuration with states: Offline/Ending (BeingBorn)
00000998.000010e8::2013/05/08-20:45:01.112 INFO [RES] Network Name: Agent: ---- Module: AccountAD with states: Closing/Ending (BeingBorn)
00000998.000010e8::2013/05/08-20:45:01.112 INFO [RES] Network Name: Agent: ---- Module: Client with states: Offline/Idle (BeingBorn)
00000998.000010e8::2013/05/08-20:45:01.112 INFO [RES] Network Name: Agent: ---- Module: Identity with states: Offline/Idle (BeingBorn)
00000998.000010e8::2013/05/08-20:45:01.112 INFO [RES] Network Name: Agent: ---- Module: Dns with states: Offline/Idle (BeingBorn)
00000998.000010e8::2013/05/08-20:45:01.112 INFO [RES] Network Name: Agent: ---- Module: Netbios with states: Offline/Idle (BeingBorn)
00000998.000010e8::2013/05/08-20:45:01.112 INFO [RES] Network Name: Agent: ---- Module: AdminShare with states: Closing/Ending (BeingBorn)
00000998.000010e8::2013/05/08-20:45:01.112 INFO [RES] Network Name: Agent: --- 7 modules for instance: c03fb105-9a50-4d00-8bee-afaf693f7efc:
00000998.000010e8::2013/05/08-20:45:01.112 INFO [RES] Network Name: Agent: ---- Module: Configuration with states: Initialized/Idle (Alive)
00000998.000010e8::2013/05/08-20:45:01.112 INFO [RES] Network Name: Agent: ---- Module: AccountAD with states: Initializing/Idle (BeingBorn)
00000998.000010e8::2013/05/08-20:45:01.112 INFO [RES] Network Name: Agent: ---- Module: Client with states: Initialized/Idle (Alive)
00000998.000010e8::2013/05/08-20:45:01.112 INFO [RES] Network Name: Agent: ---- Module: Identity with states: Error/Idle (BeingBorn)
00000998.000010e8::2013/05/08-20:45:01.112 INFO [RES] Network Name: Agent: ---- Module: Dns with states: Initialized/Idle (Alive)
00000998.000010e8::2013/05/08-20:45:01.112 INFO [RES] Network Name: Agent: ---- Module: Netbios with states: Initialized/Idle (Alive)
00000998.000010e8::2013/05/08-20:45:01.112 INFO [RES] Network Name: Agent: ---- Module: AdminShare with states: Initialized/Idle (BeingBorn)
00000998.000010e8::2013/05/08-20:45:01.112 INFO [RES] Network Name: Agent: -- 0 Zombie modules
000005a4.0000050c::2013/05/08-20:45:01.112 INFO [RCM] HandleMonitorReply: TERMINATERESOURCE for 'SQL Network Name (CPCTestSQLClus)', gen(7) result 0/0.
000005a4.0000050c::2013/05/08-20:45:01.112 INFO [RCM] Res SQL Network Name (CPCTestSQLClus): [Terminating to DelayRestartingResource] -> DelayRestartingResource( StateUnknown )
000005a4.0000050c::2013/05/08-20:45:01.112 INFO [RCM] TransitionToState(SQL Network Name (CPCTestSQLClus)) [Terminating to DelayRestartingResource]-->DelayRestartingResource.
000005a4.0000050c::2013/05/08-20:45:01.112 WARN [RCM] Queueing immediate delay restart of resource SQL Network Name (CPCTestSQLClus) in 500 ms.
000005a4.00001248::2013/05/08-20:45:01.112 INFO [RCM-rbtr] giving default token to group SQL Server (MSSQLSERVER)
000005a4.00001248::2013/05/08-20:45:01.112 INFO [RCM-rbtr] giving default token to group SQL Server (MSSQLSERVER)
000005a4.00001248::2013/05/08-20:45:01.112 INFO [RCM-rbtr] giving default token to group SQL Server (MSSQLSERVER)
000005a4.00001248::2013/05/08-20:45:01.112 INFO [RCM-rbtr] giving default token to group SQL Server (MSSQLSERVER)
000005a4.00001350::2013/05/08-20:45:01.112 INFO [GEM] Sending 1 messages as a batched GEM message
00000998.00000b78::2013/05/08-20:45:01.112 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: Getting Read/Write private properties
000005a4.00001350::2013/05/08-20:45:01.112 INFO [GEM] Sending 1 messages as a batched GEM message
00000998.00000b78::2013/05/08-20:45:01.112 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: Getting Read only private properties
00000998.00000b78::2013/05/08-20:45:01.127 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: Getting Read/Write private properties
000005a4.00001350::2013/05/08-20:45:01.127 INFO [GEM] Sending 1 messages as a batched GEM message
00000998.00000b78::2013/05/08-20:45:01.127 INFO [RES] Network Name <SQL Network Name (CPCTestSQLClus)>: Getting Read only private properties
000005a4.00001350::2013/05/08-20:45:01.127 INFO [GEM] Sending 1 messages as a batched GEM message
000005a4.00001350::2013/05/08-20:45:01.127 INFO [GEM] Sending 1 messages as a batched GEM message
000005a4.00001350::2013/05/08-20:45:01.143 INFO [GEM] Sending 1 messages as a batched GEM message
00000998.00000b78::2013/05/08-20:45:01.252 INFO [RES] Network Name: Agent: Sending request Netname/RecheckConfig to NN:c03fb105-9a50-4d00-8bee-afaf693f7efc:Netbios
00000998.00000ab8::2013/05/08-20:45:01.252 INFO [RES] Network Name <Cluster Name>: Netbios: Slow Operation, FinishWithReply: 0
00000998.00000ab8::2013/05/08-20:45:01.252 INFO [RES] Network Name: [NN] got sync reply: 0
00000998.00000ab8::2013/05/08-20:45:01.252 INFO [RES] Network Name <Cluster Name>: Netbios: End of Slow Operation, state: Initialized/Idle, prevWorkState: Idle

Failover Cluster Network Name Failed and Can't be Repaired

I have an issue that seem to be a different problem than any others have encountered.
I've scoured everything I can find and nothing has fixed my problem.
The problem starts with the common problem of the cluster network name failing on my 2 node server 2012 file server cluster. The computer object was still in AD and appeared to be fine so it was not the common problem of the object
getting deleted somehow. At the time, there was no other object with that name in the recycling bin, so I don't think it was mistakenly deleted and quickly recreated to cover any tracks, so to speak.
Following one guide, I tried to find the registry key that corresponded with the GUID of the object, but neither node in the cluster had it in its registry (which may be part of the problem).
Since it was in the failed state, I tried to do the repair on the object to no avail.
We run a "locked down" DC environment so all computer objects have to be pre-provisioned. They were all pre-provisioned successfully and successfully assigned during cluster creation. The cluster was running with no issues for a month
or so before this problem came up.
When I do a repair on the object while taking diagnostic logs the following 4609 error appears:
The action 'Repair' did not complete. - System.ApplicationException: An error occurred resetting the password for 'Cluster Name'. ---> System.ComponentModel.Win32Exception: Unknown error (0x80005000)
There appears to be a corresponding 4771 error with a failure code 0x18 that comes from the security log of the DC that states there was a Kerberos pre-authentication failure for the cluster network name object (Domain\Clustername$)
I believe this is what is causing the repair failure. All the information I found related to security error 4771 was either a bad credentials given for a user account or the fix was to reconnect the computer to the domain. I can't seem to find
a way to do this with the cluster network name. If there's a way please let me know.
I've tried a number of things, like resetting the object, disabling it, deleting and creating a new object with the same name, deleting that new object and recovering the original, etc...
Can anyone shed some light on what is going on and hopefully how to fix it other than rebuilding the cluster? I'm quite close to just tearing it down and building it back up but am hesitant because this cluster in currently in production...
Any help would be appreciated

Hi,
I don’t find out the similar issue with yours, base on my experience, the 4096 error
often caused by the CSV disk issue, and the 0x80005000 error some time caused by the repetitive computer object in OU. Please check the above related part or run the validate test then post the error information.
Although I do have a CSV, there doesn't seem to be any problems with it and it was running just fine for a month or so before the problem started. I double checked and there is no duplicate computer objects, maybe I don't understand what you mean by
repetitive, could you explain further?
The cluster validates successfully with a few warnings:
Validating cluster resource Name: DT-FileCluster.
This resource is marked with a state of 'Failed' instead of
'Online'. This failed state indicates that the resource had a problem either
coming online or had a failure while it was online. The event logs and cluster
logs may have information that is helpful in identifying the cause of the
failure.
- This is because the cluster name is in the failed state
Validating the service principal names for Name:
DT-FileCluster.
The network name Name: DT-FileCluster does not have a valid
value for the read-only property 'ObjectGUID'. To validate the service principal
name the read-only private property 'ObjectGuid' must have a valid value. To
correct this issue make sure that the network name has been brought online at
least once. If this does not correct this issue you will need to delete the
network name and re-create it.
- This is definitely related to the problem and the GUID probably got removed when we attempted a fix by resetting the object and trying the repair from the failover cluster manager.
The user running validate, does not have permissions to create
computer objects in the 'ad.unlv.edu' domain.
- This is correct, we run a restricted domain. I have a delegated OU that I can pre-provision accounts in. The account was pro-provisioned successfully and was at one point setup and working just fine.
There are no other errors nor warnings.

Cluster Quorum Disk failing inside Guest cluster VMs in Hyper-V Cluster using Virtual Disk Sharing Windows Server 2012 R2

Hi, I'm having a problem in a VM Guest cluster using Windows Server 2012 R2 and virtual disk sharing enabled.
It's a SQL 2012 cluster, which has around 10 vhdx disks shared this way. all the VHDX files are inside LUNs on a SAN. These LUNs are presented to all clustered members of the Windows Server 2012 R2 Hyper-V cluster, via Cluster Shared Volumes.
Yesterday happened a very strange problem, both the Quorum Disk and the DTC disks got the information completetly erased. The vhdx disks themselves where there, but the info inside was gone.
The SQL admin had to recreated both disks, but now we don't know if this issue was related to the virtualization platform or another event inside the cluster itself.
Right now I'm seen this errors on one of the VM Guest:
Log Name: System
Source: Microsoft-Windows-FailoverClustering
Date: 3/4/2014 11:54:55 AM
Event ID: 1069
Task Category: Resource Control Manager
Level: Error
Keywords:
User: SYSTEM
Computer: ServerDB02.domain.com
Description:
Cluster resource 'Quorum-HDD' of type 'Physical Disk' in clustered role 'Cluster Group' failed.
Based on the failure policies for the resource and role, the cluster service may try to bring the resource online on this node or move the group to another node of the cluster and then restart it. Check the resource and group state using Failover Cluster
Manager or the Get-ClusterResource Windows PowerShell cmdlet.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Microsoft-Windows-FailoverClustering" Guid="{BAF908EA-3421-4CA9-9B84-6689B8C6F85F}" />
<EventID>1069</EventID>
<Version>1</Version>
<Level>2</Level>
<Task>3</Task>
<Opcode>0</Opcode>
<Keywords>0x8000000000000000</Keywords>
<TimeCreated SystemTime="2014-03-04T17:54:55.498842300Z" />
<EventRecordID>14140</EventRecordID>
<Correlation />
<Execution ProcessID="1684" ThreadID="2180" />
<Channel>System</Channel>
<Computer>ServerDB02.domain.com</Computer>
<Security UserID="S-1-5-18" />
</System>
<EventData>
<Data Name="ResourceName">Quorum-HDD</Data>
<Data Name="ResourceGroup">Cluster Group</Data>
<Data Name="ResTypeDll">Physical Disk</Data>
</EventData>
</Event>
Log Name: System
Source: Microsoft-Windows-FailoverClustering
Date: 3/4/2014 11:54:55 AM
Event ID: 1558
Task Category: Quorum Manager
Level: Warning
Keywords:
User: SYSTEM
Computer: ServerDB02.domain.com
Description:
The cluster service detected a problem with the witness resource. The witness resource will be failed over to another node within the cluster in an attempt to reestablish access to cluster configuration data.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Microsoft-Windows-FailoverClustering" Guid="{BAF908EA-3421-4CA9-9B84-6689B8C6F85F}" />
<EventID>1558</EventID>
<Version>0</Version>
<Level>3</Level>
<Task>42</Task>
<Opcode>0</Opcode>
<Keywords>0x8000000000000000</Keywords>
<TimeCreated SystemTime="2014-03-04T17:54:55.498842300Z" />
<EventRecordID>14139</EventRecordID>
<Correlation />
<Execution ProcessID="1684" ThreadID="2180" />
<Channel>System</Channel>
<Computer>ServerDB02.domain.com</Computer>
<Security UserID="S-1-5-18" />
</System>
<EventData>
<Data Name="NodeName">ServerDB02</Data>
</EventData>
</Event>
We don't know if this can happen again, what if this happens on disk with data?! We don't know if this is related to the virtual disk sharing technology or anything related to virtualization, but I'm asking here to find out if it is a possibility.
Any ideas are appreciated.
Thanks.
Eduardo Rojas

Hi,
Please refer to the following link:
http://blogs.technet.com/b/keithmayer/archive/2013/03/21/virtual-machine-guest-clustering-with-windows-server-2012-become-a-virtualization-expert-in-20-days-part-14-of-20.aspx#.Ux172HnxtNA
Best Regards,
Vincent Wu
Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.

Grid Infrastructure Does Not Start Cluster Resources

Hello Gurus,
I configured a 2 node RAC cluster using VirtualBox.It has been running fine all along and each time I started one of the nodes, I will definitely see all of other Cluster Resources will be started eventually.
However, after I left it untouched for a month (VM is stopped), I found out that after starting up the machine, only local resource which is ONLINE.
This is what I get:
[grid@oel6-112-rac1 bin]$ ./crsctl status resource -t
NAME           TARGET STATE        SERVER                   STATE_DETAILS
Local Resources
ora.CRS.dg
               ONLINE ONLINE       oel6-112-rac1
ora.DATADG.dg
               ONLINE ONLINE       oel6-112-rac1
ora.FRADG.dg
               ONLINE ONLINE       oel6-112-rac1
ora.LISTENER.lsnr
               OFFLINE OFFLINE      oel6-112-rac1
ora.asm
               ONLINE ONLINE       oel6-112-rac1            Started
ora.gsd
               OFFLINE OFFLINE      oel6-112-rac1
ora.net1.network
               ONLINE ONLINE       oel6-112-rac1
ora.ons
               ONLINE ONLINE       oel6-112-rac1
Cluster Resources
ora.LISTENER_SCAN1.lsnr
      1        OFFLINE OFFLINE
ora.LISTENER_SCAN2.lsnr
      1        OFFLINE OFFLINE
ora.LISTENER_SCAN3.lsnr
      1        OFFLINE OFFLINE
ora.cvu
      1        OFFLINE OFFLINE
ora.oc4j
      1        OFFLINE OFFLINE
ora.oel6-112-rac1.vip
      1        OFFLINE OFFLINE
ora.oel6-112-rac2.vip
      1        OFFLINE OFFLINE
ora.racdb.db
      1        OFFLINE OFFLINE                               Instance Shutdown
      2        OFFLINE OFFLINE
ora.scan1.vip
      1        OFFLINE OFFLINE
ora.scan2.vip
      1        OFFLINE OFFLINE
ora.scan3.vip
      1        OFFLINE OFFLINE
and these are my other resources
[grid@oel6-112-rac1 bin]$ ./crsctl status resource -t -init
NAME           TARGET STATE        SERVER                   STATE_DETAILS
Cluster Resources
ora.asm
      1        ONLINE ONLINE       oel6-112-rac1            Started
ora.cluster_interconnect.haip
      1        ONLINE ONLINE       oel6-112-rac1
ora.crf
      1        ONLINE ONLINE       oel6-112-rac1
ora.crsd
      1        ONLINE ONLINE       oel6-112-rac1
ora.cssd
      1        ONLINE ONLINE       oel6-112-rac1
ora.cssdmonitor
      1        ONLINE ONLINE       oel6-112-rac1
ora.ctssd
      1        ONLINE ONLINE       oel6-112-rac1            ACTIVE:0
ora.diskmon
      1        OFFLINE OFFLINE
ora.evmd
      1        ONLINE ONLINE       oel6-112-rac1
ora.gipcd
      1        ONLINE ONLINE       oel6-112-rac1
ora.gpnpd
      1        ONLINE ONLINE       oel6-112-rac1
ora.mdnsd
      1        ONLINE ONLINE       oel6-112-rac1
Where do I supposed to check to see why the Cluster Resource like SCAN Listener, Database and etc not running?
I've been checking on the logs but I haven't figured out what I should be looking at.
Can some body help me?
Thank you in advanced,
Adhika

Hi Freddie,
I saw these lines on that Clusterware alert log:
2013-07-01 22:39:20.084
[crsd(3338)]CRS-1012:The OCR service started on node oel6-112-rac1.
2013-07-01 22:39:20.208
[evmd(3145)]CRS-1401:EVMD started on node oel6-112-rac1.
2013-07-01 22:39:21.549
[crsd(3338)]CRS-1201:CRSD started on node oel6-112-rac1.
2013-07-01 22:39:22.715
[/u01/app/11203/grid/bin/oraagent.bin(3450)]CRS-5016:Process "/u01/app/11203/grid/bin/lsnrctl" spawned by agent "/u01/app/11203/grid/bin/oraagent.bin" for action "check" failed: details at "(:CLSN00010:)" in "/u01/app/11203/grid/log/oel6-112-rac1/agent/crsd/oraagent_grid/oraagent_grid.log"
2013-07-01 22:39:22.728
[/u01/app/11203/grid/bin/oraagent.bin(3450)]CRS-5016:Process "/u01/app/11203/grid/bin/lsnrctl" spawned by agent "/u01/app/11203/grid/bin/oraagent.bin" for action "check" failed: details at "(:CLSN00010:)" in "/u01/app/11203/grid/log/oel6-112-rac1/agent/crsd/oraagent_grid/oraagent_grid.log"
2013-07-01 22:39:22.772
[/u01/app/11203/grid/bin/oraagent.bin(3450)]CRS-5016:Process "/u01/app/11203/grid/bin/lsnrctl" spawned by agent "/u01/app/11203/grid/bin/oraagent.bin" for action "check" failed: details at "(:CLSN00010:)" in "/u01/app/11203/grid/log/oel6-112-rac1/agent/crsd/oraagent_grid/oraagent_grid.log"
2013-07-01 22:39:22.811
[/u01/app/11203/grid/bin/oraagent.bin(3450)]CRS-5016:Process "/u01/app/11203/grid/bin/lsnrctl" spawned by agent "/u01/app/11203/grid/bin/oraagent.bin" for action "check" failed: details at "(:CLSN00010:)" in "/u01/app/11203/grid/log/oel6-112-rac1/agent/crsd/oraagent_grid/oraagent_grid.log"
2013-07-01 22:39:23.069
[/u01/app/11203/grid/bin/oraagent.bin(3450)]CRS-5016:Process "/u01/app/11203/grid/opmn/bin/onsctli" spawned by agent "/u01/app/11203/grid/bin/oraagent.bin" for action "check" failed: details at "(:CLSN00010:)" in "/u01/app/11203/grid/log/oel6-112-rac1/agent/crsd/oraagent_grid/oraagent_grid.log"
2013-07-01 22:39:23.567
[crsd(3338)]CRS-2772:Server 'oel6-112-rac1' has been assigned to pool 'Generic'.
2013-07-01 22:39:23.568
[crsd(3338)]CRS-2772:Server 'oel6-112-rac1' has been assigned to pool 'ora.racdb'.
The I started looking in the /u01/app/11203/grid/log/oel6-112-rac1/agent/crsd/oraagent_grid/oraagent_grid.log file and found out that at the same time (2013-07-01 22:39:22),
I saw the following lines:
2013-07-01 22:39:22.433: [ora.CRS.dg][1644160768] {1:13152:2} [check] DgpAgent::getConnxn connection failure 1
2013-07-01 22:39:22.433: [ora.CRS.dg][1644160768] {1:13152:2} [check] DgpAgent::getConnxn failed CRS-5000: Expected resource ora.asm does not exist in agent process
2013-07-01 22:39:22.434: [ora.CRS.dg][1644160768] {1:13152:2} [check] DgpAgent::getConnxn try getInstanceInforWhenASMFail
2013-07-01 22:39:22.434: [ora.CRS.dg][1644160768] {1:13152:2} [check] CrsCmd::ClscrsCmdData::stat entity 1 statflag 33 useFilter 0
But that does not prevent the asm from being started properly.
The only local resource that didn't start up automatically was the LISTENER.
The following command shows that the local LISTENER has hard dependency on ora.cluster_vip_net1.type which is the ora.oel6-112-rac1.vip
[grid@oel6-112-rac1 bin]$ ./crsctl status resource ora.LISTENER.lsnr -p | grep -ie dependencies
START_DEPENDENCIES=hard(type:ora.cluster_vip_net1.type) pullup(type:ora.cluster_vip_net1.type)
STOP_DEPENDENCIES=hard(intermediate:type:ora.cluster_vip_net1.type)
NAME=ora.oel6-112-rac1.vip
TYPE=ora.cluster_vip_net1.type
START_DEPENDENCIES=hard(ora.net1.network) pullup(ora.net1.network)
STOP_DEPENDENCIES=hard(ora.net1.network)
[grid@oel6-112-rac1 bin]$ ./crsctl status resource ora.net1.network -p | grep -ie dependencies
START_DEPENDENCIES=
STOP_DEPENDENCIES=
The ora.net1.network resource started properly and I didn't see that this prevent the ora.oel6-112-rac1.vip from starting up.
The following lines also show that the ora.asm resource is has a weak dependency only against the ora.LISTENER.lsnr
[grid@oel6-112-rac1 bin]$ ./crsctl status resource ora.racdb.db -p | grep -ie dependencies
START_DEPENDENCIES=hard(ora.DATADG.dg,ora.FRADG.dg) weak(type:ora.listener.type,global:type:ora.scan_listener.type,uniform:ora.ons,global:ora.gns) pullup(ora.DATADG.dg,ora.FRADG.dg)
STOP_DEPENDENCIES=hard(intermediate:ora.asm,shutdown:ora.DATADG.dg,shutdown:ora.FRADG.dg)
[grid@oel6-112-rac1 bin]$ ./crsctl status resource ora.CRS.dg -p | grep -ie dependenci
START_DEPENDENCIES=hard(ora.asm) pullup(ora.asm)
STOP_DEPENDENCIES=hard(intermediate:ora.asm)
[grid@oel6-112-rac1 bin]$ ./crsctl status resource ora.DATADG.dg -p | grep -ie dependencies
START_DEPENDENCIES=hard(ora.asm) pullup(ora.asm)
STOP_DEPENDENCIES=hard(intermediate:ora.asm)
[grid@oel6-112-rac1 bin]$ ./crsctl status resource ora.FRADG.dg -p | grep -ie dependencies
START_DEPENDENCIES=hard(ora.asm) pullup(ora.asm)
STOP_DEPENDENCIES=hard(intermediate:ora.asm)
[grid@oel6-112-rac1 bin]$ ./crsctl status resource ora.asm -p | grep -ie dependencies
START_DEPENDENCIES=weak(ora.LISTENER.lsnr)
STOP_DEPENDENCIES=
I'm a little lost here.
A suggestion would be very much appreciated.
Thank you,
Adhika

One VM Cluster Resources Regularly Failing

Similar Messages

Maybe you are looking for