Operations manager failed to run a wmi query for wmi events (0x800706ba)

Question

Hi everyone,
I've been working on this issue for a while and I am still no closer to finding out what the problem is.  If anybody can offer any other advice or things to check, I'm all ears.
I'm running SCOM 2012 R2 with UR2, and the Cluster Management Pack v6.0.7063.0
My problem is on one particular batch of cluster servers where I am getting the following error.
Name: Operations Manager failed to run a WMI query for WMI events
Alert Description:
Module was unable to enumerate the WMI data
Error: 0x800706ba
Details: The RPC server is unavailable
Workflow name: Microsoft.Windows.Cluster.Node.StateMonitoring
Instance Name: servername.domain.local
Instance ID: {instance_id}
Management group: SCOM_Management_Grp_Name
I am getting this alert regardless of whether I run the Windows Cluster Action Account as Local System, or as a domain user with full local admin privileges on all the cluster nodes.
When looking at the management pack and the workflow in particular (Microsoft.Windows.Cluster.Node.StateMonitoring), I can see that it's trying to access
MSCluster_Node in the root\MSCLUSTER WMI namespace.
This is the workflow for your information...
<UnitMonitor> ID="Microsoft.Windows.Cluster.Node.StateMonitoring" Accessibility="Public" Enabled="onEssentialMonitoring" Target="ClusLibrary!Microsoft.Windows.Cluster.Node" ParentMonitorID="Health!System.Health.AvailabilityState" Remotable="true" Priority="Normal" TypeID="ClusLibrary!Microsoft.Windows.Cluster.CheckState" ConfirmDelivery="false">
<Category>AvailabilityHealth</Category>
<AlertSettings AlertMessage="Microsoft.Windows.Cluster.Node.StateMonitoring.AlertMessage">
<AlertOnState>Warning</AlertOnState>
<AutoResolve>true</AutoResolve>
<AlertPriority>Normal</AlertPriority>
<AlertSeverity>MatchMonitorHealth</AlertSeverity>
<AlertParameters>
<AlertParameter1>$Target/Property[Type="Windows!Microsoft.Windows.Computer"]/NetworkName$</AlertParameter1>
<AlertParameter2>$Target/Property[Type="ClusLibrary!Microsoft.Windows.Cluster.Node"]/ClusterName$</AlertParameter2>
</AlertParameters>
</AlertSettings>
<OperationalStates>
<OperationalState ID="Success" MonitorTypeStateID="Online" HealthState="Success" />
<OperationalState ID="Warning" MonitorTypeStateID="Partial" HealthState="Warning" />
<OperationalState ID="Error" MonitorTypeStateID="NotOnline" HealthState="Error" />
</OperationalStates>
<Configuration>
<ClusterObjectName>$Target/Property[Type='ClusLibrary!Microsoft.Windows.Cluster.Node']/NodeName$</ClusterObjectName>
<PollInterval>60</PollInterval>
<ClusterObjectClass>MSCLUSTER_Node</ClusterObjectClass>
<OnlineExpression>
<SimpleExpression>
<ValueExpression>
<XPathQuery Type="String">Property[@Name='EventNewState']</XPathQuery>
</ValueExpression>
<Operator>Equal</Operator>
<ValueExpression>
<Value Type="String">0</Value>
</ValueExpression>
</SimpleExpression>
</OnlineExpression>
<OnlineExpressionOnDemand>
<SimpleExpression>
<ValueExpression>
<XPathQuery Type="String">Property[@Name='State']</XPathQuery>
</ValueExpression>
<Operator>Equal</Operator>
<ValueExpression>
<Value Type="String">0</Value>
</ValueExpression>
</SimpleExpression>
</OnlineExpressionOnDemand>
<PartialExpression>
<Or>
<Expression>
<SimpleExpression>
<ValueExpression>
<XPathQuery Type="String">Property[@Name='EventNewState']</XPathQuery>
</ValueExpression>
<Operator>Equal</Operator>
<ValueExpression>
<Value Type="String">2</Value>
</ValueExpression>
</SimpleExpression>
</Expression>
<Expression>
<SimpleExpression>
<ValueExpression>
<XPathQuery Type="String">Property[@Name='EventNewState']</XPathQuery>
</ValueExpression>
<Operator>Equal</Operator>
<ValueExpression>
<Value Type="String">3</Value>
</ValueExpression>
</SimpleExpression>
</Expression>
</Or>
</PartialExpression>
<PartialExpressionOnDemand>
<Or>
<Expression>
<SimpleExpression>
<ValueExpression>
<XPathQuery Type="String">Property[@Name='State']</XPathQuery>
</ValueExpression>
<Operator>Equal</Operator>
<ValueExpression>
<Value Type="String">2</Value>
</ValueExpression>
</SimpleExpression>
</Expression>
<Expression>
<SimpleExpression>
<ValueExpression>
<XPathQuery Type="String">Property[@Name='State']</XPathQuery>
</ValueExpression>
<Operator>Equal</Operator>
<ValueExpression>
<Value Type="String">3</Value>
</ValueExpression>
</SimpleExpression>
</Expression>
</Or>
</PartialExpressionOnDemand>
<NotOnlineExpression>
<And>
<Expression>
<SimpleExpression>
<ValueExpression>
<XPathQuery Type="String">Property[@Name='EventNewState']</XPathQuery>
</ValueExpression>
<Operator>NotEqual</Operator>
<ValueExpression>
<Value Type="String">0</Value>
</ValueExpression>
</SimpleExpression>
</Expression>
<Expression>
<SimpleExpression>
<ValueExpression>
<XPathQuery Type="String">Property[@Name='EventNewState']</XPathQuery>
</ValueExpression>
<Operator>NotEqual</Operator>
<ValueExpression>
<Value Type="String">2</Value>
</ValueExpression>
</SimpleExpression>
</Expression>
<Expression>
<SimpleExpression>
<ValueExpression>
<XPathQuery Type="String">Property[@Name='EventNewState']</XPathQuery>
</ValueExpression>
<Operator>NotEqual</Operator>
<ValueExpression>
<Value Type="String">3</Value>
</ValueExpression>
</SimpleExpression>
</Expression>
</And>
</NotOnlineExpression>
<NotOnlineExpressionOnDemand>
<And>
<Expression>
<SimpleExpression>
<ValueExpression>
<XPathQuery Type="String">Property[@Name='State']</XPathQuery>
</ValueExpression>
<Operator>NotEqual</Operator>
<ValueExpression>
<Value Type="String">0</Value>
</ValueExpression>
</SimpleExpression>
</Expression>
<Expression>
<SimpleExpression>
<ValueExpression>
<XPathQuery Type="String">Property[@Name='State']</XPathQuery>
</ValueExpression>
<Operator>NotEqual</Operator>
<ValueExpression>
<Value Type="String">2</Value>
</ValueExpression>
</SimpleExpression>
</Expression>
<Expression>
<SimpleExpression>
<ValueExpression>
<XPathQuery Type="String">Property[@Name='State']</XPathQuery>
</ValueExpression>
<Operator>NotEqual</Operator>
<ValueExpression>
<Value Type="String">3</Value>
</ValueExpression>
</SimpleExpression>
</Expression>
</And>
</NotOnlineExpressionOnDemand>
<WMIFields>Name, State</WMIFields>
</Configuration>
</UnitMonitor>
I can confirm that I am able to browse the MSCluster_Node class locally, as well as remotely using WMIEXPLORER and WBEMTEST,
however it only works when I set the Authentication Level to
Packet Privacy.  If I do not select Packet Privacy, a WMI event log error 5605 is logged on the remote servers application log that says...
The root\mscluster namespace is marked with the RequiresEncryption flag.  Access to this namespace might be denied if the script or application does not have the appropriate authentication level.  Change the authentication level to Pkt_Privacy
and run the script or application again.
I can confirm that all firewalls are turned off, and there are no firewalls between the management servers and the agents in question.  AV exclusions have been done and appear to be in place.  The nodes are all Windows 2008 R2 with SP1.  As
far as I can tell there is plenty of memory available on each of the nodes in question (50%+) of RAM is available. 
If I manually run the "Discover the Windows Server 2008 R2 Cluster Components" task in the Cluster Service State section of the management pack in the Monitoring Pane in the console, on the nodes in question - the discovery runs successfully.
Does anybody have any other ideas or suggestions I could try?
Many thanks in advance,
Noel.
http://www.dreamension.net

Answer 1

Hi,
Common causes of RPC errors include:
Errors resolving a DNS or NetBIOS name.
The RPC service or related services may not be running.
Problems with network connectivity.
File and printer sharing is not enabled.
For more information, please review the link below:
Windows Server Troubleshooting: "The RPC server is unavailable"
http://social.technet.microsoft.com/wiki/contents/articles/4494.windows-server-troubleshooting-the-rpc-server-is-unavailable.aspx#Identify
Troubleshooting RPC Errors
http://technet.microsoft.com/en-us/magazine/2007.07.howitworks.aspx
We
are trying to better understand customer views on social support experience, so your participation in this
interview project would be greatly appreciated if you have time.
Thanks for helping make community forums a great place.