Issue in Global Services Monitor GSM

Question

Dear all,
I'm facing a problem in Global Services Monitor.
The Resources pool contains 4 Management Servers; 2 old and 2 recently installed.
The GSM was installed and was working normally on the old Management Servers.
But, after increasing the number of management servers to be 4 instead of two, the problem appeared.
The GSM is firing alerts on the new MS's and their state are critical (old 2 servers are healthy).
The alert description is as below:
Global Service Monitor Modules: Failed to discover Global Service Monitor locations.
Failure step: 'Couldn't get the ACS endpoint from discovery service. SubscriptionId: 'a6846da0-e5d7-4bea-ab13-836d89364b60', OutsideInServiceBaseUri: 'https://gsm-prod.systemcenter.microsoft.com/''
Message: 'Could not establish trust relationship for the SSL/TLS secure channel with authority 'gsm-prod.systemcenter.microsoft.com'.'
Details: 'System.ServiceModel.Security.SecurityNegotiationException: Could not establish trust relationship for the SSL/TLS secure channel with authority 'gsm-prod.systemcenter.microsoft.com'. ---> System.Net.WebException: The underlying connection was closed:
Could not establish trust relationship for the SSL/TLS secure channel. ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.
at System.Net.Security.SslState.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, Exception exception)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Net.TlsStream.ProcessAuthentication(LazyAsyncResult result)
at System.Net.TlsStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.PooledStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.ConnectStream.WriteHeaders(Boolean async)
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.GetResponse()
at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)
--- End of inner exception stack trace ---
Server stack trace: 
at System.ServiceModel.Channels.HttpChannelUtilities.ProcessGetResponseWebException(WebException webException, HttpWebRequest request, HttpAbortReason abortReason)
at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)
at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
Exception rethrown at [0]: 
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at Microsoft.SystemCenter.Cloud.SharedLibrary.Discovery.IDiscovery.GetEndpoints(String subscriptionId)
at Microsoft.SystemCenter.Cloud.SharedLibrary.Discovery.DiscoveryHelper.<>c__DisplayClass1.<DiscoverAcsEndpoint>b__0(IDiscovery service)
at Microsoft.SystemCenter.Cloud.SharedLibrary.RestCallHelper.ExecuteRestCall[TContract](Uri endpointUri, WebProxy webProxy, String accessToken, RestMethod`1 method)
at Microsoft.SystemCenter.Cloud.SharedLibrary.Discovery.DiscoveryHelper.DiscoverAcsEndpoint(String subscriptionId, Uri outsideInServiceBaseUri, WebProxy proxy)
at Microsoft.SystemCenter.Cloud.OutsideInUnitModule.DiscoveryWriteActionModule.Execute()'
Any clue?
Regards,
Khaled A. Hamad

Answer 1

Hi
Is there any requirement to install the Microsoft Root Certificate on the server where SCOM console is working? Shall I need to purchase Windows Azure Subscription also for GSM? Please let me know.
The scenario is - I have one SCOM server (Including all the server roles on the single server) and other server where VMM server and SCOM console is installed. I have installed GSM Management Packs on the SCOM server and configured one Web Availability Monitor
to be monitored from external servers (e.g. Chicago).
I am getting the below error:-
Log Name:      Operations Manager
Source:        Health Service Modules Ex
Date:          9/11/2014 7:14:26 PM
Event ID:      10001
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      SCOMCLOUD.abc.in
Description:
Global Service Monitor Modules:  Failed step: 'Couldn't get the ACS endpoint from discovery service. SubscriptionId: '1f156904-532e-416f-b570-1141438392a3', OutsideInServiceBaseUri: 'https://gsm-prod.systemcenter.microsoft.com/''. Diagnostic context: RequestId
= '0fe72d85-989c-4c1b-89c1-1f4b641c1578', New ConfigHash = '65afc4b6-c18d-5e68-56d3-482e2db1851a', '1' tests, Last ConfigHash = '00000000-0000-0000-0000-000000000000'. Exception: 'There was no endpoint listening at https://gsm-prod.systemcenter.microsoft.com/DiscoveryService/1f156904-532e-416f-b570-1141438392a3/Endpoints
that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details.'
One or more workflows were affected by this. 
Workflow name: Microsoft.SystemCenter.Omonline.OutsideIn.Discovery.ConfigUploaderRule 
Instance name: Global Service Monitor 
Instance ID: {298CB0DA-4453-EFD2-A7AC-C2E8F2F7100D} 
Management group: SCOMGROUP
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Health Service Modules Ex" />
    <EventID Qualifiers="0">10001</EventID>
    <Level>3</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2014-09-11T13:44:26.000000000Z" />
    <EventRecordID>149790</EventRecordID>
    <Channel>Operations Manager</Channel>
    <Computer>SCOMCLOUD.abc.in</Computer>
    <Security />
  </System>
  <EventData>
    <Data>SCOMGROUP</Data>
    <Data>Microsoft.SystemCenter.Omonline.OutsideIn.Discovery.ConfigUploaderRule</Data>
    <Data>Global Service Monitor</Data>
    <Data>{298CB0DA-4453-EFD2-A7AC-C2E8F2F7100D}</Data>
    <Data>Failed step: 'Couldn't get the ACS endpoint from discovery service. SubscriptionId: '1f156904-532e-416f-b570-1141438392a3', OutsideInServiceBaseUri: 'https://gsm-prod.systemcenter.microsoft.com/''. Diagnostic context: RequestId = '0fe72d85-989c-4c1b-89c1-1f4b641c1578',
New ConfigHash = '65afc4b6-c18d-5e68-56d3-482e2db1851a', '1' tests, Last ConfigHash = '00000000-0000-0000-0000-000000000000'. Exception: 'There was no endpoint listening at https://gsm-prod.systemcenter.microsoft.com/DiscoveryService/1f156904-532e-416f-b570-1141438392a3/Endpoints
that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details.'</Data>
    <Data>Global Service Monitor Modules</Data>
    <Data>Couldn't get the ACS endpoint from discovery service. SubscriptionId: '1f156904-532e-416f-b570-1141438392a3', OutsideInServiceBaseUri: 'https://gsm-prod.systemcenter.microsoft.com/'</Data>
    <Data>RequestId = '0fe72d85-989c-4c1b-89c1-1f4b641c1578', New ConfigHash = '65afc4b6-c18d-5e68-56d3-482e2db1851a', '1' tests, Last ConfigHash = '00000000-0000-0000-0000-000000000000'</Data>
    <Data>There was no endpoint listening at https://gsm-prod.systemcenter.microsoft.com/DiscoveryService/1f156904-532e-416f-b570-1141438392a3/Endpoints that could accept the message. This is often caused by an incorrect address or SOAP action.
See InnerException, if present, for more details.</Data>
    <Data>System.ServiceModel.EndpointNotFoundException: There was no endpoint listening at https://gsm-prod.systemcenter.microsoft.com/DiscoveryService/1f156904-532e-416f-b570-1141438392a3/Endpoints that could accept the message. This is often
caused by an incorrect address or SOAP action. See InnerException, if present, for more details. ---&gt; System.Net.WebException: The remote name could not be resolved: 'gsm-prod.systemcenter.microsoft.com'
   at System.Net.HttpWebRequest.GetResponse()
   at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)
   --- End of inner exception stack trace ---
Server stack trace: 
   at System.ServiceModel.Channels.HttpChannelUtilities.ProcessGetResponseWebException(WebException webException, HttpWebRequest request, HttpAbortReason abortReason)
   at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)
   at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
Exception rethrown at [0]: 
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData&amp; msgData, Int32 type)
   at Microsoft.SystemCenter.Cloud.SharedLibrary.Discovery.IDiscovery.GetEndpoints(String subscriptionId)
   at Microsoft.SystemCenter.Cloud.SharedLibrary.Discovery.DiscoveryHelper.&lt;&gt;c__DisplayClass1.&lt;DiscoverAcsEndpoint&gt;b__0(IDiscovery service)
   at Microsoft.SystemCenter.Cloud.SharedLibrary.RestCallHelper.ExecuteRestCall[TContract](Uri endpointUri, WebProxy webProxy, String accessToken, RestMethod`1 method)
   at Microsoft.SystemCenter.Cloud.SharedLibrary.Discovery.DiscoveryHelper.DiscoverAcsEndpoint(String subscriptionId, Uri outsideInServiceBaseUri, WebProxy proxy)
   at Microsoft.SystemCenter.Cloud.OutsideInUnitModule.ConfigUploaderWriteActionModule.Execute()</Data>
  </EventData>
</Event>
Any HELP would be really Appreciated.
Thanks in advance.
Abhinav | MCTS-Server Virtualization