FrontendTransport health set unhealthy (OnPremisesSmtpClientSubmissionMonitor)
FrontendTransport health set unhealthy (OnPremisesSmtpClientSubmissionMonitor) - The client submission probe failed 3 times over 15 minutes.
Seems like these alerts have started comming for some of the servers, where mailbox and CAS role is installed together. when i cehcked the queue, all seems to be fine. Performed the below mentioned steps, but the issue didn't fixed:
1. invoke-monitoringprobe" command doesn't work.
2. Have restarted "health manager service" didn't work.
Still the alert value is in uhealthy state, have anyone come across the same issue, if so, can you share what are the steps that we have take?
Your answers are much appreciated!
Hi,
Please check the Monitor Result and Probe Result in the following path and see if there is any related message.
Event Viewer\Applications and Services Logs\Microsoft\Exchange\ActiveMonitoring\ProbeResult( or MonitorResult).
Based on your description, everthing works well except this alert. However, there is a way to hide the alert by overriding the monitor using the command below:
Add-GlobalMonitoringOverride -Identity "FrontendTransport\OnPremisesSmtpClientSubmissionMonitor" -PropertyName Enabled -PropertyValue 0 -ItemType Monitor -ApplyVersion "version"
Hope this is helpful to you.
Best regards,
Belinda Ma
TechNet Community Support
Similar Messages
-
Exchange 2013 CU2, Alert for OWA Health set unhealthy from SCOM 2012
I am facing issue in Exchange 2013 CU2, I got this alert from SCOM 2012 atleast 5-6 times a day, OWA health set is unhealthy, I have done all the steps mentioned in this web link. Authentication type for OWA Virtual directory is integrated windows and Basic.
I have 2 CAS servers, and this alert generated from both of them.
http://technet.microsoft.com/en-us/library/ms.exch.scom.OWA(EXCHG.150).aspx?v=15.0.712.24
Alert: Health Set unhealthy
Source: EX-CAS - OWA
Path: EX-CAS;EX-CAS
Last modified by: System
Last modified time: 1/5/2014 8:15:08 PM
Alert description: Outlook Web Access logon is failing on ClientAccess server EX-CAS.
Availability has dropped to 0%. You can find protocol level traces for the failures on C:\Program Files\Microsoft\Exchange Server\V15\Logging\Monitoring\OWA\ClientAccessProbe.
Incident start time: 1/6/2014 4:05:08 AM
Last failed result:
Failing Component - Owa
Failure Reason - CafeFailure
Exception:
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> Microsoft.Exchange.Net.MonitoringWebClient.ScenarioException:
Microsoft.Exchange.Net.MonitoringWebClient.ScenarioException:
Failure source: Owa
Failure reason: CafeFailure
Failing component:Owa
Exception hint: CafeErrorPage: CafeFailure Unauthorized Inner exception: Microsoft.Exchange.Net.MonitoringWebClient.CafeErrorPageException
ErrorPageFailureReason: CafeFailure, RequestFailureContext: FailurePoint=FrontEnd, HttpStatusCode=401, Error=Unauthorized, Details=, HttpProxySubErrorCode=, WebExceptionStatus=
Microsoft.Exchange.Net.MonitoringWebClient.CafeErrorPageException: An error occurred on the Client Access server while processing the request
WebExceptionStatus: Success
GET https://localhost/owa/ HTTP/1.1
User-Agent: Mozilla/4.0 (compatible; MSIE 9.0; Windows NT 6.1; MSEXCHMON; ACTIVEMONITORING; OWACTP)
Accept: */*
Cache-Control: no-cache
X-OWA-ActionName: Monitoring
Cookie:
HTTP/1.1 401 Unauthorized
request-id: 211474d2-a43e-4fab-8038-3aab35353568
X-FailureContext: FrontEnd;401;VW5hdXRob3JpemVk;;;
Server: Microsoft-IIS/7.5
WWW-Authenticate: Negotiate,NTLM,Basic realm="localhost"
X-Powered-By: ASP.NET
X-FEServer: EX-CAS
Date: Mon, 06 Jan 2014 04:14:47 GMT
Content-Length: 0
Response time: 0s
---> Microsoft.Exchange.Net.MonitoringWebClient.CafeErrorPageException: Microsoft.Exchange.Net.MonitoringWebClient.CafeErrorPageException
ErrorPageFailureReason: CafeFailure, RequestFailureContext: FailurePoint=FrontEnd, HttpStatusCode=401, Error=Unauthorized, Details=, HttpProxySubErrorCode=, WebExceptionStatus=
Microsoft.Exchange.Net.MonitoringWebClient.CafeErrorPageException: An error occurred on the Client Access server while processing the request
WebExceptionStatus: Success
GET https://localhost/owa/ HTTP/1.1
User-Agent: Mozilla/4.0 (compatible; MSIE 9.0; Windows NT 6.1; MSEXCHMON; ACTIVEMONITORING; OWACTP)
Accept: */*
Cache-Control: no-cache
X-OWA-ActionName: Monitoring
Cookie:
HTTP/1.1 401 Unauthorized
request-id: 211474d2-a43e-4fab-8038-3aab35353568
X-FailureContext: FrontEnd;401;VW5hdXRob3JpemVk;;;
Server: Microsoft-IIS/7.5
WWW-Authenticate: Negotiate,NTLM,Basic realm="localhost"
X-Powered-By: ASP.NET
X-FEServer: EX-CAS
Date: Mon, 06 Jan 2014 04:14:47 GMT
Content-Length: 0
Response time: 0s
--- End of inner exception stack trace ---
at Microsoft.Exchange.Net.MonitoringWebClient.BaseExceptionAnalyzer.Analyze(TestId currentTestStep, HttpWebRequestWrapper request, HttpWebResponseWrapper response, Exception exception, Action`1 trackingDelegate)
at Microsoft.Exchange.Net.MonitoringWebClient.HttpSession.AnalyzeResponse[T](HttpWebRequestWrapper request, HttpWebResponseWrapper response, Exception exception, HttpStatusCode[] expectedStatusCodes, Func`2 processResponse)
at Microsoft.Exchange.Net.MonitoringWebClient.HttpSession.EndSend[T](IAsyncResult result, HttpStatusCode[] expectedStatusCodes, Func`2 processResponse, Boolean fireResponseReceivedEvent)
at Microsoft.Exchange.Net.MonitoringWebClient.HttpSession.EndGet[T](IAsyncResult result, HttpStatusCode[] expectedStatusCodes, Func`2 processResponse)
at Microsoft.Exchange.Net.MonitoringWebClient.Authenticate.AuthenticationResponseReceived(IAsyncResult result)
--- End of inner exception stack trace ---
at Microsoft.Exchange.Net.MonitoringWebClient.BaseTestStep.EndExecute(IAsyncResult result)
at Microsoft.Exchange.Net.MonitoringWebClient.Owa.OwaLogin.AuthenticationCompleted(IAsyncResult result)
--- End of inner exception stack trace ---
at Microsoft.Exchange.Net.MonitoringWebClient.BaseTestStep.EndExecute(IAsyncResult result)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Bool
States of all monitors within the health set:
Note: Data may be stale. To get current data, run: Get-ServerHealth -Identity 'EX-CAS' -HealthSet 'OWA'
State
Name
TargetResource HealthSet
AlertValue ServerComponent
NotApplicable OwaCtpMonitor
OWA
Unhealthy None
States of all health sets:
Note: Data may be stale. To get current data, run: Get-HealthReport -Identity 'EX-CAS'
State
HealthSet
AlertValue LastTransitionTime
MonitorCount
NotApplicable ActiveSync
Healthy 1/3/2014 5:21:13 AM
2
NotApplicable AD
Healthy 11/24/2013 6:54:18 AM
10
NotApplicable ECP
Healthy 1/5/2014 3:03:05 AM
1
Online
Autodiscover.Proxy
Healthy 11/20/2013 10:06:37 AM
1
NotApplicable Autodiscover
Healthy 1/3/2014 10:18:17 PM
2
Online
ActiveSync.Proxy
Healthy 11/20/2013 10:06:37 AM
1
Online
ECP.Proxy
Healthy
11/21/2013 6:16:08 PM 4
Online
EWS.Proxy
Healthy 11/20/2013 10:06:37 AM
1
Online
OutlookMapi.Proxy
Healthy 11/24/2013 6:54:28 AM
4
Online
OAB.Proxy
Healthy 11/19/2013 7:14:34 PM
1
Online
OWA.Proxy
Healthy 11/20/2013 10:06:37 AM
2
NotApplicable EDS
Healthy 1/3/2014 5:19:56 AM
10
Online
RPS.Proxy
Healthy 1/3/2014 5:21:27 AM
13
Online
RWS.Proxy Healthy
1/3/2014 5:20:09 AM 10
Online
Outlook.Proxy
Healthy 1/3/2014 5:21:12 AM
4
NotApplicable EWS
Healthy 1/3/2014 10:18:17 PM
2
Online
FrontendTransport
Healthy 1/5/2014 3:47:09 AM
11
Online
HubTransport
Healthy 1/5/2014 3:47:09 AM
29
NotApplicable Monitoring
Unhealthy 1/5/2014 4:05:57 AM
9
NotApplicable DataProtection
Healthy 1/3/2014 5:25:42 AM
1
NotApplicable Network Healthy
1/4/2014 1:51:16 PM 1
NotApplicable OWA
Unhealthy 1/5/2014 8:05:08 PM
1
NotApplicable FIPS
Healthy 1/3/2014 5:21:12 AM
3
Online
Transport
Healthy 1/5/2014 4:11:00 AM
9
NotApplicable RPS
Healthy 11/20/2013 10:07:12 AM
2
NotApplicable Compliance
Healthy 11/20/2013 10:08:10 AM
2
NotApplicable Outlook
Healthy 11/21/2013 6:12:54 PM
2
Online
UM.CallRouter
Healthy 1/5/2014 3:47:10 AM
7
NotApplicable UserThrottling
Healthy 1/5/2014 4:16:42 AM
7
NotApplicable Search
Healthy
11/24/2013 6:55:06 AM 9
NotApplicable AntiSpam
Healthy 1/3/2014 5:16:43 AM
3
NotApplicable Security
Healthy 1/3/2014 5:19:28 AM
3
NotApplicable IMAP.Protocol
Healthy 1/3/2014 5:21:14 AM
3
NotApplicable Datamining
Healthy 1/3/2014 5:18:34 AM
3
NotApplicable Provisioning
Healthy 1/3/2014 5:19:56 AM
3
NotApplicable POP.Protocol
Healthy 1/3/2014 5:20:44 AM
3
NotApplicable Outlook.Protocol
Healthy 1/3/2014 5:19:46 AM
3
NotApplicable ProcessIsolation
Healthy 1/3/2014 5:19:26 AM
9
NotApplicable Store
Healthy 1/3/2014 5:20:38 AM
6
NotApplicable TransportSync
Healthy 11/24/2013 6:53:09 AM
3
NotApplicable MailboxTransport
Healthy 1/3/2014 5:21:11 AM
6
NotApplicable EventAssistants
Healthy 11/21/2013 6:22:01 PM
2
NotApplicable MRS
Healthy 1/3/2014 5:20:29 AM
3
NotApplicable MessageTracing
Healthy 1/3/2014 5:18:15 AM
3
NotApplicable CentralAdmin
Healthy 1/3/2014 5:17:25 AM
3
NotApplicable UM.Protocol
Healthy 1/3/2014 5:17:08 AM
3
NotApplicable Autodiscover.Protocol
Healthy 1/3/2014 5:17:13 AM
3
NotApplicable OAB
Healthy 1/3/2014 5:20:51 AM
3
NotApplicable OWA.Protocol
Healthy 1/3/2014 5:20:52 AM
3
NotApplicable Calendaring
Healthy 11/24/2013 6:56:59 AM
3
NotApplicable PushNotifications.Protocol
Healthy 11/21/2013 6:16:05 PM
3
NotApplicable EWS.Protocol
Healthy 1/3/2014 5:19:07 AM
3
NotApplicable ActiveSync.Protocol
Healthy
1/3/2014 5:20:16 AM 3
NotApplicable RemoteMonitoring
Healthy 1/5/2014 3:47:09 AM
3
Any solution for this alert, how to rectify it, but OWA is running perfect for all users.Hi,
Sorry for the late reply.
Do we have Exchange 2010 coexistence?
If it is the case, I know the following known issue:
Release Notes for Exchange 2013
http://technet.microsoft.com/en-us/library/jj150489%28v=exchg.150%29.aspx
Please note the "Exchange 2010 coexistence" session.
If it is not related to our problem, please check the IIS log.
If there is any detailed error code, like 401.1, 401.2, please let me know.
Hope it is helpful
Thanks
Mavis
If you have feedback for TechNet Subscriber Support, contact
[email protected]
Mavis Huang
TechNet Community Support -
Alert: Health Set unhealthy - Clustering
We have SCOM 2012 R2 setup to monitor our Exchange 2013 CU5 enviroment and we have gotten this error message about our Clustering going in to an unhealthy state a couple of times. We have checked the FSW and everything seems OK on its end. I
cannot find much out there on this message, so any help would be greatly appreciated:
Alert: Health Set unhealthy
Source: EXCHANGE04 - Clustering
Path: EXCHANGE04.company.com;EXCHANGE04.company.com
Last modified by: System
Last modified time: 8/24/2014 1:36:35 PM Alert description: The Cluster Group has not been healthy for 7200 minutes. The most recent probe failure message is: Check 'Microsoft.Exchange.Monitoring.QuorumGroupCheck' thrown an Exception!
Exception - Microsoft.Exchange.Monitoring.ReplicationCheckFailedException: QuorumGroup has failed. Specific error is: Quorum resource 'Cluster Group' is not online on server 'exchange06'. Database availability group 'exchDAG' might not be reachable or may have
lost redundancy. Error:
File Share Witness (\\FSW01.company.com\exchDAG.company.com): Offline is offline. Please verify that the Cluster service is running on the server.
at Microsoft.Exchange.Monitoring.ReplicationCheck.Fail(LocalizedString error)
at Microsoft.Exchange.Monitoring.QuorumGroupCheck.RunCheck()
at Microsoft.Exchange.Monitoring.DagMemberCheck.InternalRun()
at Microsoft.Exchange.Monitoring.ReplicationCheck.Run()
at Microsoft.Exchange.Monitoring.ActiveMonitoring.HighAvailability.Probes.ReplicationHealthChecksProbeBase.RunReplicationCheck(Type checkType) Check 'Microsoft.Exchange.Monitoring.QuorumGroupCheck' did not Pass!
Detail Message - Quorum resource 'Cluster Group' is not online on server 'exchange06'. Database availability group 'exchDAG' might not be reachable or may have lost redundancy. Error:
File Share Witness (\\FSW01.company.com\exchDAG.company.com): Offline is offline. Please verify that the Cluster service is running on the server.
To add some additional information, when I look in Failover cluster manager this is what I see. I know when we setup the servers the correct FSW information was being displayed.Hi,
According to the error message, "Offline is offline. Please verify that the Cluster service is running on the server.",
I suggest double check whether the Cluster service is running as well. If not, please restart the service manually to verify whether this issue exists.
Please also refer the blog below to double check whether the FSW online:
Verifying the file share witness server / directory in use for Exchange 2010
http://blogs.technet.com/b/timmcmic/archive/2012/03/12/verifying-the-file-share-witness-server-directory-in-use-for-exchange-2010.aspx
If there is nothing abnormal on the Exchange server, it seems an issue on the SCOM side. Please contact SCOM Forum for help so that you can get more professional suggestions. For your convenience:
http://social.technet.microsoft.com/Forums/systemcenter/en-US/home?category=systemcenteroperationsmanager
Thanks
Mavis
Mavis Huang
TechNet Community Support -
Hi,
I am currently running a 2013 CU2 DAG with 2 database and 2 cas servers. SCOM is reporting the following but i can find very little info on it;
Alert: Health Set unhealthy
Source: <server name> - Outlook.Protocol
Last modified by: System
Last modified time: 4/2/2014 3:49:58 PM
Alert description: EMSMDB.DoRpc(Logon) step of OutlookRpcDeepTestProbe/<database name> has failed against <server name> proxying to <server name> for HealthMailboxb63d235bb56b428ebf56ea594d3ca0c7@CEOSMTPServer.
Latency: 00:00:00.0520000
ActivityContext: I32:ADS.C[Apollo]=1;F:ADS.AL[Apollo]=3.3585;I32:ADR.C[Apollo]=1;F:ADR.AL[Apollo]=3.0093;I32:ADS.C[Razor]=2;F:ADS.AL[Razor]=2.0185
Outline: [50] EMSMDB.Connect(); [1][FAILED!] EMSMDB.DoRpc(Logon); Likely root cause: Momt
Details:
Error: Error returned in LogonCallResult. Error code = WrongServer (0x00000478)
Log: Mailbox logon verification
EMSMDB.Connect()
Task produced output:
- TaskStarted = 2/04/2014 3:49:25 PM
- TaskFinished = 2/04/2014 3:49:25 PM
- ErrorDetails =
- RespondingRpcClientAccessServerVersion = 15.0.712.4012
- Latency = 00:00:00.0505291
- ActivityContext = I32:ADS.C[Apollo]=1;F:ADS.AL[Apollo]=3.3585;I32:ADR.C[Apollo]=1;F:ADR.AL[Apollo]=3.0093;I32:ADS.C[Razor]=2;F:ADS.AL[Razor]=2.0185
EMSMDB.Connect() completed successfully.
EMSMDB.DoRpc(Logon)
Task produced output:
- TaskStarted = 2/04/2014 3:49:25 PM
- TaskFinished = 2/04/2014 3:49:25 PM
- Exception = Microsoft.Exchange.RpcClientAccess.RopExecutionException: Error returned in LogonCallResult. Error code = WrongServer (0x00000478)
- ErrorDetails =
- Latency = 00:00:00.0010381
- ActivityContext = I32:ADS.C[Apollo]=1;F:ADS.AL[Apollo]=3.3585;I32:ADR.C[Apollo]=1;F:ADR.AL[Apollo]=3.0093;I32:ADS.C[Razor]=2
Any help would be greatly appreciated.
ThanksHi,
Please run the following command and post the output:
Get-ServerHealth -Identity Servername -HealthSet Outlook.Protocol
In addition, I recommend you run "test-mapiconnectivity" and check event viewer on exchange server.
http://technet.microsoft.com/en-us/library/bb123681(v=exchg.150).aspx
Use the Test-MapiConnectivity cmdlet to verify server functionality by logging on to the mailbox that you specify. If you don't specify a mailbox, the cmdlet logs on to the SystemMailbox on the database that you specify.
Thanks.
Niko Cheng
TechNet Community Support -
Health set components seems to be unhealthy
Hi,
In my environment health sets components seems to be unhealthy but there is no problem with user side
Below are the components
HealthSet AlertValue
MailboxTransport Unhealthy
HubTransport Unhealthy
ECP Unhealthy
Search Unhealthy
Store Unhealthy
MSExchangeCertif... Disabled
DataProtection Unhealthy
RPS Unhealthy
RWS Unhealthy
Compliance Unhealthy
Outlook Unhealthy
Can somebody help me through this please.Hello,
I think you can combine the heltht reports with the application log?
Is there any warning or error reprot in it about these unhealty items. If no, I think we can safely ingore these errors.
Thanks,
Simon Wu
TechNet Community Support -
HubTransport and FrontendTransport marked as Unhealthy. How to diagnose and resolve?
I am trying to do an Exchange 2007 to 2013 migration. Before doing all the final repointing of DNS and proxying clients, I just want to make sure that the new Exchange 2013 is healthy. So I run a Get-HealthReport and it shows both the Online
states of HubTransport and FrontendTransport as Unhealthy.
In the event logs under Microsoft>Exchange>ManagedAvailability>Monitoring I see Error events for them about the client submission probe failed x times over 15 minutes. No connection because the target machine actively refused it 127.0.0.1:587.
There was also another one that said Unable to relay. For the Hub Transport it does not give as detailed information. It just says The HubAvailibilityProbe has failed 5 or more times in 15 minutes. Last failing server: ".
So it would sound like a connectivity problem, so with TELNET I tried to connect to port 25 and I get this:
telnet 127.0.0.1 25
421 4.3.2 Service not available
BUT if I put the hostname instead of 127.0.0.1, I get the 220 EMAIL.domain.com Microsoft ESMTP MAIL Service ready.
I tried temporarily disabling the windows firewall and its the same problem. I know SMTP is running because from another machine I can telnet to the mail server on port 25 and I get the SMTP banner.
So for me it seems like the Managed Availability is trying to use the IP address of 127.0.0.1 and that fails fundamentally for me when I try that in the command line, so no wonder why its marking the FrontendTransport as unhealthy. How can I resolve
this? I would think a resolution would be either somehow allow 127.0.0.1 access to relay which I DID put in my Frontend Transport receive connector, but it made no difference. Or the other option is somehow configure the probe to use the
hostname instead of 127.0.0.1.
As far as the HubTransport being unhealthy, I have a feeling its the same type of issue, but I am not seeing as detailed information in this path of the Event Viewer.Bump....
No issues with mail flow but still getting this in the MOnitoring log from Managed Availability.
Log Name: Microsoft-Exchange-ManagedAvailability/Monitoring
Source: Microsoft-Exchange-ManagedAvailability
Date: 12/8/2014 11:11:29 AM
Event ID: 4
Task Category: Monitoring
Level: Error
Keywords:
User: SYSTEM
Computer: email.domain.com
Description:
The inbound proxy probe failed 3 times over 15 minutes.
Server email.domain.com on port 587 did not respond with expected response (ServiceReady). The actual response was: 421 4.3.2 Service not available
Probe Exception: 'System.Exception: Server email.domain.com on port 587 did not respond with expected response (ServiceReady). The actual response was: 421 4.3.2 Service not available
at Microsoft.Forefront.Monitoring.ActiveMonitoring.Smtp.Probes.SmtpConnectionProbe.AssertExpectedResponse(SmtpExpectedResponse expectedResponse)
at Microsoft.Forefront.Monitoring.ActiveMonitoring.Smtp.Probes.SmtpConnectionProbe.TestConnection()
at Microsoft.Forefront.Monitoring.ActiveMonitoring.Smtp.Probes.SmtpConnectionProbe.DoWork(CancellationToken cancellationToken)
at Microsoft.Office.Datacenter.WorkerTaskFramework.WorkItem.Execute(CancellationToken joinedToken)
at Microsoft.Office.Datacenter.WorkerTaskFramework.WorkItem.<>c__DisplayClass2.<StartExecuting>b__0()
at System.Threading.Tasks.Task.Execute()'
Failure Context: 'Server email.domain.com on port 587 did not respond with expected response (ServiceReady). The actual response was: 421 4.3.2 Service not available
Execution Context: ''
Probe Result Name: 'OnPremisesInboundProxy'
Probe Result Type: 'Failed'
Monitor Total Value: '3'
Monitor Total Sample Count: '3'
Monitor Total Failed Count: '0'
Monitor Poisoned Count: '0'
Monitor First Alert Observed Time: '9/25/2014 1:27:15 PM'
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Microsoft-Exchange-ManagedAvailability" Guid="{C424A887-A89F-455F-8319-960917152221}" />
<EventID>4</EventID>
<Version>0</Version>
<Level>2</Level>
<Task>2</Task>
<Opcode>0</Opcode>
<Keywords>0x8000000000000000</Keywords>
<TimeCreated SystemTime="2014-12-08T16:11:29.045528200Z" />
<EventRecordID>7358</EventRecordID>
<Correlation />
<Execution ProcessID="7604" ThreadID="1800" />
<Channel>Microsoft-Exchange-ManagedAvailability/Monitoring</Channel>
<Computer>email.domain.com</Computer>
<Security UserID="S-1-5-18" />
</System>
<UserData>
<EventXML xmlns:auto-ns2="http://schemas.microsoft.com/win/2004/08/events" xmlns="myNs">
<HealthSet>FrontendTransport</HealthSet>
<Subject>The inbound proxy probe failed 3 times over 15 minutes.</Subject>
<Message>The inbound proxy probe failed 3 times over 15 minutes.
Server email.domain.com on port 587 did not respond with expected response (ServiceReady). The actual response was: 421 4.3.2 Service not available
Probe Exception: 'System.Exception: Server email.domain.com on port 587 did not respond with expected response (ServiceReady). The actual response was: 421 4.3.2 Service not available
at Microsoft.Forefront.Monitoring.ActiveMonitoring.Smtp.Probes.SmtpConnectionProbe.AssertExpectedResponse(SmtpExpectedResponse expectedResponse)
at Microsoft.Forefront.Monitoring.ActiveMonitoring.Smtp.Probes.SmtpConnectionProbe.TestConnection()
at Microsoft.Forefront.Monitoring.ActiveMonitoring.Smtp.Probes.SmtpConnectionProbe.DoWork(CancellationToken cancellationToken)
at Microsoft.Office.Datacenter.WorkerTaskFramework.WorkItem.Execute(CancellationToken joinedToken)
at Microsoft.Office.Datacenter.WorkerTaskFramework.WorkItem.<>c__DisplayClass2.<StartExecuting>b__0()
at System.Threading.Tasks.Task.Execute()'
Failure Context: 'Server email.domain.com on port 587 did not respond with expected response (ServiceReady). The actual response was: 421 4.3.2 Service not available
Execution Context: ''
Probe Result Name: 'OnPremisesInboundProxy'
Probe Result Type: 'Failed'
Monitor Total Value: '3'
Monitor Total Sample Count: '3'
Monitor Total Failed Count: '0'
Monitor Poisoned Count: '0'
Monitor First Alert Observed Time: '9/25/2014 1:27:15 PM'
</Message>
<Monitor>OnPremisesInboundProxyMonitor</Monitor>
</EventXML>
</UserData>
</Event> -
Hello,
I'm getting several times per day the following alert in the Event Viewer of both my Mailbox Role servers which then raise an alert in SCOM 2012:
Log Name: Microsoft-Exchange-ManagedAvailability/Monitoring
Source: ManagedAvailability
Event ID: 4
Mean Scan Time has been greater than 60 seconds for the last 15 mins. Please investigate.
I'm not able to find information about what is being monitored and how to get rid of this events.
Any help appreciated.
Thank you in advance.
Raffaele ZamunerHi Amit,
here's the output from the EMS and the Event Viewer:
[PS] C:\Windows\system32>Get-ServerHealth [ServerName] | ?{$_.alertvalue -ne "Healthy" -and $_.alertvalue -ne "Disabled"}
Server State Name TargetResource
HealthSetName AlertValue ServerComp
onent
[ServerName] NotApplicable ScanTimeMeanMonitor FIPS
Unhealthy None
[ServerName] NotApplicable MaintenanceFailur... DAL
Unknown None
[ServerName] NotApplicable DatabaseLogicalPh... FINIT MailboxSpace Repairing None
[ServerName] NotApplicable DatabaseLogicalPh... MailboxSpace
Unhealthy None
[ServerName] Online Total.Shadow.Queu...
HubTransport Unhealthy HubTran...
[ServerName] NotApplicable MailboxDeliveryAv... MailboxTrans... Unhealthy
None
[ServerName] NotApplicable DatabaseLogicalPh... DB1 MailboxSpace Repairing None
[ServerName] NotApplicable DatabaseLogicalPh... MCS1 MailboxSpace Repairing None
[PS] C:\Windows\system32>get-serverhealth -Server [ServerName] -HealthSet AMScanTimeou
[PS] C:\Windows\system32>
- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
- <System>
<Provider Name="Microsoft-Exchange-ManagedAvailability" Guid="{C424A887-A89F-455F-8319-960917152221}" />
<EventID>4</EventID>
<Version>0</Version>
<Level>2</Level>
<Task>2</Task>
<Opcode>0</Opcode>
<Keywords>0x8000000000000000</Keywords>
<TimeCreated SystemTime="2014-06-03T07:23:16.626852900Z" />
<EventRecordID>1753</EventRecordID>
<Correlation />
<Execution ProcessID="1860" ThreadID="21496" />
<Channel>Microsoft-Exchange-ManagedAvailability/Monitoring</Channel>
<Computer>[ServerName].finit.local</Computer>
<Security UserID="S-1-5-18" />
</System>
- <UserData>
- <EventXML xmlns:auto-ns2="http://schemas.microsoft.com/win/2004/08/events" xmlns="myNs">
<HealthSet>FIPS</HealthSet>
<Subject>FIPS health set unhealthy (ScanTimeMeanMonitor) - High Mean Scan Time</Subject>
<Message>Mean Scan Time has been greater than 60 seconds for the last 15 mins. Please investigate. ------------------------------------------------------------------------------- States of all monitors within the health set: Note: Data may be stale.
To get current data, run: Get-ServerHealth -Identity '[ServerName]' -HealthSet 'FIPS' State Name TargetResource HealthSet AlertValue ServerComponent ----- ---- -------------- --------- ---------- --------------- NotApplicable ScanProcessesRunningMonitor FIPS
Healthy None NotApplicable ScanProcessBelowMinimumMonitor FIPS Healthy None NotApplicable ScanRequestsTimedOutMonitor FIPS Healthy None NotApplicable ScanRequestsTimedOutUrgentMonitor FIPS Healthy None NotApplicable ScanRequestQueueTimeoutUrgentMonitor FIPS
Healthy None NotApplicable RecoveryQueueTimeoutUrgentMonitor FIPS Healthy None NotApplicable ScanTimeMeanMonitor FIPS Healthy None NotApplicable ScanRequestErrorRatioMonitor FIPS Healthy None NotApplicable ScanRequestErrorsMonitor FIPS Healthy None NotApplicable
TextExtractionErrorsMonitor FIPS Healthy None NotApplicable ObsoleteEngineInUseEventMonitor FIPS FIPS Healthy None NotApplicable MaintenanceFailureMonitor.FIPS FIPS Healthy None NotApplicable EngineNotEnabledForUpdatesEventMonitor FIPS FIPS Healthy None NotApplicable
EUSTerminationEventMonitor FIPS FIPS Healthy None NotApplicable EUSUnableToStartEventMonitor FIPS FIPS Healthy None NotApplicable ADFilteringSettingsWatcherStartFaile... FIPS FIPS Healthy None NotApplicable ScanningProcessRepeatedlyCrashingMon... FIPS FIPS
Healthy None NotApplicable FilteringManagementServiceDownMonitor FIPS Healthy None NotApplicable PrivateWorkingSetWarning.fms fms FIPS Healthy None NotApplicable PrivateWorkingSetError.fms fms FIPS Healthy None NotApplicable ProcessProcessorTimeWarning.fms
fms FIPS Healthy None NotApplicable ProcessProcessorTimeError.fms fms FIPS Healthy None NotApplicable ExchangeCrashEventError.fms fms FIPS Healthy None NotApplicable LongRunningWatsonWarning.fms fms FIPS Healthy None NotApplicable LongRunningWerMgrWarning.fms
fms FIPS Healthy None NotApplicable MicrosoftFailedUpdatesMonitor FIPS Healthy None NotApplicable MicrosoftEngineErrorsMonitor FIPS Healthy None NotApplicable MSClassificationEngineErrorsMonitor FIPS Healthy None States of all health sets: Note: Data may be
stale. To get current data, run: Get-HealthReport -Identity '[ServerName]' State HealthSet AlertValue LastTransitionTime MonitorCount ----- --------- ---------- ------------------ ------------ NotApplicable ActiveSync.Protocol Healthy 03/06/2014 05.04.35 9
NotApplicable Autodiscover.Protocol Healthy 03/06/2014 01.18.16 9 NotApplicable ActiveSync Healthy 03/06/2014 08.24.19 1 NotApplicable Antimalware Unhealthy 03/06/2014 09.18.14 11 NotApplicable ECP Healthy 03/06/2014 08.40.07 5 NotApplicable Ediscovery.Protocol
Healthy 03/06/2014 08.55.21 1 NotApplicable AD Healthy 03/06/2014 07.46.15 19 NotApplicable EDS Healthy 03/06/2014 08.52.30 90 NotApplicable EventAssistants Healthy 03/06/2014 09.21.03 15 NotApplicable EWS.Protocol Healthy 03/06/2014 09.15.04 11 NotApplicable
FEP Healthy 02/06/2014 09.04.40 2 NotApplicable FIPS Healthy 03/06/2014 09.15.34 28 NotApplicable Monitoring Healthy 03/06/2014 07.53.12 17 NotApplicable MailboxMigration Healthy 02/06/2014 14.03.19 3 Online UM.CallRouter Healthy 03/06/2014 04.44.11 8 NotApplicable
UM.Protocol Healthy 03/06/2014 05.05.45 22 NotApplicable OAB Healthy 03/06/2014 09.22.26 8 Online HubTransport Unhealthy 03/06/2014 09.22.35 133 NotApplicable EAS Healthy 02/05/2014 10.33.27 1 NotApplicable Autodiscover Healthy 03/06/2014 05.56.59 1 NotApplicable
DataProtection Healthy 03/06/2014 09.18.11 66 NotApplicable FreeBusy Healthy 02/05/2014 10.30.56 1 NotApplicable Clustering Healthy 03/06/2014 08.21.27 5 NotApplicable BitlockerDeployment Healthy 02/06/2014 22.00.36 1 NotApplicable ClientAccess.Proxy Healthy
02/06/2014 14.20.31 1 NotApplicable Compliance Healthy 02/05/2014 10.29.01 1 NotApplicable EWS Healthy 01/06/2014 12.57.38 1 NotApplicable FfoRws Healthy 01/06/2014 09.15.31 1 NotApplicable Places Healthy 03/06/2014 01.16.29 1 Online Transport Healthy 03/06/2014
08.23.33 10 NotApplicable RemoteMonitoring Healthy 02/06/2014 15.11.34 4 NotApplicable Search Healthy 03/06/2014 07.04.56 140 NotApplicable Inference Healthy 01/06/2014 23.57.09 1 NotApplicable OWA Healthy 03/06/2014 09.16.39 2 NotApplicable MailboxSpace Repairing
03/06/2014 09.22.02 17 NotApplicable MRS Healthy 03/06/2014 09.20.58 19 NotApplicable MessageTracing Healthy 03/06/2014 07.03.30 8 NotApplicable Network Healthy 03/06/2014 09.09.30 2 NotApplicable Calendaring Healthy 03/06/2014 09.19.16 8 NotApplicable PeopleConnect
Healthy 02/06/2014 14.10.08 1 NotApplicable IMAP Healthy 03/06/2014 08.10.29 1 NotApplicable POP Healthy 03/06/2014 05.54.04 1 NotApplicable ProcessIsolation Healthy 03/06/2014 07.39.59 22 NotApplicable Provisioning Healthy 03/06/2014 08.35.16 8 NotApplicable
Psws Healthy 03/06/2014 08.51.44 15 NotApplicable PublicFolders Healthy 03/06/2014 09.21.54 5 NotApplicable Outlook Healthy 03/06/2014 05.56.00 1 NotApplicable RPS Healthy 03/06/2014 08.22.48 1 NotApplicable RWS Healthy 03/06/2014 08.11.44 2 NotApplicable
Security Healthy 03/06/2014 08.11.33 8 NotApplicable SiteMailbox Healthy 03/06/2014 09.07.10 4 NotApplicable Store Healthy 03/06/2014 09.19.09 110 NotApplicable TransportSync Healthy 03/06/2014 08.11.56 8 NotApplicable UserThrottling Healthy 03/06/2014 09.21.49
8 NotApplicable FfoWebService Healthy 03/06/2014 07.07.43 1 NotApplicable DAL Healthy 03/06/2014 08.08.02 12 NotApplicable FfoWebstore Healthy 03/06/2014 07.07.33 1 NotApplicable AntiSpam Healthy 03/06/2014 06.02.52 8 NotApplicable OWA.Protocol.Dep Healthy
30/05/2014 17.45.35 1 NotApplicable MigrationMonitor Healthy 03/06/2014 08.33.07 3 NotApplicable MailboxTransport Unhealthy 03/06/2014 09.05.01 63 NotApplicable MSExchangeCertificateDeplo... Disabled 01/01/0001 01.00.00 2 NotApplicable OWA.Protocol Healthy
03/06/2014 07.33.09 9 NotApplicable Outlook.Protocol Healthy 02/06/2014 18.07.34 11 NotApplicable RPS.Protocol Healthy 02/05/2014 10.33.10 2 NotApplicable IMAP.Protocol Healthy 03/06/2014 07.31.55 7 NotApplicable Datamining Healthy 03/06/2014 05.06.42 7 NotApplicable
POP.Protocol Healthy 01/06/2014 19.02.12 7 Online FrontendTransport Healthy 02/06/2014 09.06.30 7 NotApplicable CentralAdmin Healthy 03/06/2014 07.37.18 7 Online ECP.Proxy Healthy 03/06/2014 06.30.43 7 Online RPS.Proxy Healthy 03/06/2014 09.22.16 28 NotApplicable
PushNotifications.Protocol Healthy 03/06/2014 08.34.46 7 Online RWS.Proxy Healthy 02/06/2014 22.55.09 7 Online Outlook.Proxy Healthy 03/06/2014 07.51.37 7 Online OutlookMapi.Proxy Healthy 03/06/2014 08.09.14 7 Note: Subsequent detected alerts are suppressed
until the health set is healthy again.</Message>
<Monitor>ScanTimeMeanMonitor</Monitor>
</EventXML>
</UserData>
</Event>
Raffaele Zamuner -
Exchange 2013 CU3 Databases only activate on one mailbox server
Hi, guys
I have two Exchange 2013 CU3 Mailbox servers installed, one DAG, 5 databases, each has one copy. I found that if I activated three databases on Mailboxserver1 or Mailboxserver2, then after a few hours, all databases will be activated on the mailbox
server which has three databases activated. All the databases can be activated on Mailboxserver1 or Mailboxserver2, and they work well. I disabled DAC mode for preventing Event 4133 and 4376. And it has the same problem if I enable DAC mode.
From the event log, I found the log when activate one database on another mailbox server, it is Event 3169:
Managed availability system failover initiated by Responder=OutlookMapiHttpDeepTestFailover Component=Outlook.
This caused the database activated on another server.
And I got the message from SCOM, like this:
Alert: Health Set unhealthy
Source: test-mbx - Outlook.Protocol
Path: test-mbx.contoso.local;test-mbx.contoso.local
Last modified by: System
Last modified time: 11/12/2013 5:15:46 AM Alert description: EMSMDB.DoRpc(Logon) step of OutlookRpcDeepTestProbe/DB-01 has failed against test-mbx.contoso.local proxying to test-mbx.contoso.local for [email protected].
Latency: 00:00:00.0320000
ActivityContext:
Outline: [30] EMSMDB.Connect(); [1][FAILED!] EMSMDB.DoRpc(Logon); Likely root cause: Momt
Details:
Error: Error returned in LogonCallResult. Error code = WrongServer (0x00000478)
Log: Mailbox logon verification
EMSMDB.Connect()
Task produced output:
- TaskStarted = 11/12/2013
5:15:25 AM
- TaskFinished = 11/12/2013
5:15:25 AM
- ErrorDetails =
- RespondingRpcClientAccessServerVersion
= 15.0.712.4012
Latency = 00:00:00.0303884
- ActivityContext =
EMSMDB.Connect() completed successfully.
EMSMDB.DoRpc(Logon)
Task produced output:
- TaskStarted = 11/12/2013
5:15:25 AM
- TaskFinished = 11/12/2013
5:15:25 AM
- Exception = Microsoft.Exchange.RpcClientAccess.RopExecutionException:
Error returned in LogonCallResult. Error code = WrongServer (0x00000478)
- ErrorDetails =
- Latency = 00:00:00.0018801
- ActivityContext =
EMSMDB.DoRpc(Logon) failed.
Task produced output:
- TaskStarted = 11/12/2013 5:15:25 AM
- TaskFinished = 11/12/2013 5:15:25 AM
- Exception = Microsoft.Exchange.RpcClientAccess.RopExecutionException:
Error
States of all monitors within the health set:
Note: Data may be stale. To get current data, run: Get-ServerHealth -Identity 'test-mbx' -HealthSet 'Outlook.Protocol'
State Name
TargetResource HealthSet
AlertValue ServerComponent
NotApplicable OutlookMapiHttpDeepTestMonitor
Outlook.Protocol Unhealthy None
NotApplicable OutlookRpcDeepTestMonitor
Outlook.Protocol Healthy None
NotApplicable OutlookRpcSelfTestMonitor
Outlook.Protocol Healthy None
NotApplicable OutlookMapiHttpSelfTestMonitor Outlook.Protocol
Healthy None
NotApplicable PrivateWorkingSetWarning....cclienta... microsoft.exchange.rpcclientacc... Outlook.Protocol Healthy
None
NotApplicable PrivateWorkingSetError....rpcclienta... microsoft.exchange.rpcclientacc... Outlook.Protocol Healthy
None
NotApplicable ProcessProcessorTimeWarning....ienta... microsoft.exchange.rpcclientacc... Outlook.Protocol Healthy
None
NotApplicable ProcessProcessorTimeError....clienta... microsoft.exchange.rpcclientacc... Outlook.Protocol Healthy
None
NotApplicable ExchangeCrashEventError....pcclienta... microsoft.exchange.rpcclientacc... Outlook.Protocol Healthy
None
NotApplicable LongRunningWatsonWarning....cclienta... microsoft.exchange.rpcclientacc... Outlook.Protocol Healthy
None
NotApplicable LongRunningWerMgrWarning....cclienta... microsoft.exchange.rpcclientacc... Outlook.Protocol Healthy
None
This test is a cause that mailbox databases in DAG is doing failover to another server
Log Name: Application
Source: MSExchangeRepl
Date: 12.11.2013 4:49:46
Event ID: 3169
Task Category: Service
Level: Information
Keywords: Classic
User: N/A
Computer: test-mbx-2
Description:
(Active Manager) Database DB-01 was successfully moved from test-mbx.contoso.local to test-mbx-1.contoso.local. Move comment: Managed availability system failover initiated by Responder=OutlookRpcDeepTestFailover Component=Outlook.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="MSExchangeRepl" />
<EventID Qualifiers="16388">3169</EventID>
<Level>4</Level>
<Task>1</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2013-11-12T00:49:46.000000000Z" />
<EventRecordID>1606248</EventRecordID>
<Channel>Application</Channel>
<Computer>test-mbx-2.contoso.local</Computer>
<Security />
</System>
<EventData>
<Data>DB-01</Data>
<Data>test-mbx.contoso.local</Data>
<Data>test-mbx-1.contoso.local</Data>
<Data>Managed availability system failover initiated by Responder=OutlookRpcDeepTestFailover Component=Outlook.</Data>
</EventData>
</Event>
I don't know why, anyone know what's the problem?
Thank you.
Nile Jiang- Please mark the post as answer if it answers your question.
http://www.usefulshare.comHi,
After deleting all the health mailboxes and restart
the Exchange Health Manager service, the health mailboxes are recreated successfullly, but when I check the outlook.protocol health, the OutlookRpcDeepTestMonitor or the OutlookMapiHttpDeepTestMonitor is still unhealthy. How can I fix it?
[PS] C:\Windows\system32> Get-ServerHealth -Identity 'MAILBOX1' -HealthSet 'Outlook.Protocol' | ft server,state,name,ale
rtvalue -AutoSize
Server state Name AlertValue
MAILBOX1 OutlookRpcDeepTestMonitor Healthy
MAILBOX1 OutlookMapiHttpDeepTestMonitor Unhealthy
MAILBOX1 OutlookRpcSelfTestMonitor Healthy
MAILBOX1 OutlookMapiHttpSelfTestMonitor Healthy
MAILBOX1 PrivateWorkingSetWarning....cclientaccess.service Healthy
MAILBOX1 PrivateWorkingSetError....rpcclientaccess.service Healthy
MAILBOX1 ProcessProcessorTimeWarning....ientaccess.service Healthy
MAILBOX1 ProcessProcessorTimeError....clientaccess.service Healthy
MAILBOX1 ExchangeCrashEventError....pcclientaccess.service Healthy
MAILBOX1 LongRunningWatsonWarning....cclientaccess.service Healthy
MAILBOX1 LongRunningWerMgrWarning....cclientaccess.service Healthy
Nile Jiang- Please mark the post as answer if it answers your question.
http://www.usefulshare.com -
Managed Availability MapiExceptionMultiMailboxSearchFailed
Hello Everyone,
Have a fresh Exchange 2013 SP1 install setup and before we start to make any changes towards coexistence we would like to know how to resolve the below alerts on our 4 mailbox servers.
First Event:
Probe: EdiscoveryDeepTestProbe/Mailbox Database 2031363269
Latency: 0ms
Error: MapiExceptionMultiMailboxSearchFailed: Multi Mailbox Keyword Stats Search failed. (hr=0x80004005, ec=2802)
Diagnostic context:
Lid: 1494 ---- Remote Context Beg ----
Lid: 55392 StoreEc: 0xAF2
Lid: 32864 StoreEc: 0xAF2
Lid: 42621 StoreEc: 0xAF2
Lid: 1750 ---- Remote Context End ----
Lid: 63080
Lid: 38504 StoreEc: 0xAF2
Lid: 58984
Lid: 34408 StoreEc: 0xAF2
Details:
Log:
States of all monitors within the health set:
Note: Data may be stale. To get current data, run: Get-ServerHealth -Identity 'NCITSMBX02' -HealthSet 'Ediscovery.Protocol'
State Name
TargetResource HealthSet
AlertValue ServerComponent
NotApplicable EdiscoveryDeepTestMonitor
Ediscovery.Protocol Healthy None
Second Event:
Search has unhealthy copy status for mounted database 'Mailbox Database 2031363269'. The last probe error was:
Search catalog is in 'FailedAndSuspended' state for mounted database 'Mailbox Database 2031363269'. Failover has been tried already but did not solve the problem.
Status of all copies of the database among servers:
Copy: 'Mailbox Database 2031363269\NCITSMBX02', Database Status: 'Mounted', Content Index Status: 'FailedAndSuspended', Error Message: 'The content index is corrupted.'.
Please let me know if you need anymore information. Thanks for any assistance!Hi Gulab,
Thanks for the response. These are standalone servers (total of 4 mailbox servers) with no DAG.
These errors are happening on all databases (which on on the servers right now).
Error 1:
Log Name: Microsoft-Exchange-ManagedAvailability/Monitoring
Source: Microsoft-Exchange-ManagedAvailability
Date: 6/12/2014 9:38:04 AM
Event ID: 4
Task Category: Monitoring
Level: Error
Keywords:
User: SYSTEM
Computer: NCITSMBX02.domain.local
Description:
Probe: EdiscoveryDeepTestProbe/Mailbox Database 2031363269
Latency: 0ms
Error: MapiExceptionMultiMailboxSearchFailed: Multi Mailbox Keyword Stats Search failed. (hr=0x80004005, ec=2802)
Diagnostic context:
Lid: 1494 ---- Remote Context Beg ----
Lid: 55392 StoreEc: 0xAF2
Lid: 32864 StoreEc: 0xAF2
Lid: 42621 StoreEc: 0xAF2
Lid: 1750 ---- Remote Context End ----
Lid: 63080
Lid: 38504 StoreEc: 0xAF2
Lid: 58984
Lid: 34408 StoreEc: 0xAF2
Details:
Log:
Note: Subsequent detected alerts are suppressed until the health set is healthy again.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Microsoft-Exchange-ManagedAvailability" Guid="{C424A887-A89F-455F-8319-960917152221}" />
<EventID>4</EventID>
<Version>0</Version>
<Level>2</Level>
<Task>2</Task>
<Opcode>0</Opcode>
<Keywords>0x8000000000000000</Keywords>
<TimeCreated SystemTime="2014-06-12T13:38:04.712079000Z" />
<EventRecordID>830</EventRecordID>
<Correlation />
<Execution ProcessID="11740" ThreadID="5864" />
<Channel>Microsoft-Exchange-ManagedAvailability/Monitoring</Channel>
<Computer>NCITSMBX02.domain.local</Computer>
<Security UserID="S-1-5-18" />
</System>
<UserData>
<EventXML xmlns:auto-ns2="http://schemas.microsoft.com/win/2004/08/events" xmlns="myNs">
<HealthSet>Ediscovery.Protocol</HealthSet>
<Subject>Ediscovery.Protocol health set unhealthy (EdiscoveryDeepTestMonitor) - EdiscoveryDeepTestMonitor is failing on NCITSMBX02</Subject>
<Message>Probe: EdiscoveryDeepTestProbe/Mailbox Database 2031363269
Latency: 0ms
Error: MapiExceptionMultiMailboxSearchFailed: Multi Mailbox Keyword Stats Search failed. (hr=0x80004005, ec=2802)
Diagnostic context:
Lid: 1494 ---- Remote Context Beg ----
Lid: 55392 StoreEc: 0xAF2
Lid: 32864 StoreEc: 0xAF2
Lid: 42621 StoreEc: 0xAF2
Lid: 1750 ---- Remote Context End ----
Lid: 63080
Lid: 38504 StoreEc: 0xAF2
Lid: 58984
Lid: 34408 StoreEc: 0xAF2
Details:
Log:
States of all monitors within the health set:
Note: Data may be stale. To get current data, run: Get-ServerHealth -Identity 'NCITSMBX02' -HealthSet 'Ediscovery.Protocol'
State Name
TargetResource HealthSet
AlertValue ServerComponent
NotApplicable EdiscoveryDeepTestMonitor
Ediscovery.Protocol Healthy None
States of all health sets:
Note: Subsequent detected alerts are suppressed until the health set is healthy again.</Message>
<Monitor>EdiscoveryDeepTestMonitor</Monitor>
</EventXML>
</UserData>
</Event>
Event 2:
Log Name: Microsoft-Exchange-ManagedAvailability/Monitoring
Source: Microsoft-Exchange-ManagedAvailability
Date: 6/12/2014 9:22:20 AM
Event ID: 4
Task Category: Monitoring
Level: Error
Keywords:
User: SYSTEM
Computer: NCITSMBX02.domain.local
Description:
Search has unhealthy copy status for mounted database 'Mailbox Database 2031363269'. The last probe error was:
Search catalog is in 'FailedAndSuspended' state for mounted database 'Mailbox Database 2031363269'. Failover has been tried already but did not solve the problem.
Status of all copies of the database among servers:
Copy: 'Mailbox Database 2031363269\NCITSMBX02', Database Status: 'Mounted', Content Index Status: 'FailedAndSuspended', Error Message: 'The content index is corrupted.'.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Microsoft-Exchange-ManagedAvailability" Guid="{C424A887-A89F-455F-8319-960917152221}" />
<EventID>4</EventID>
<Version>0</Version>
<Level>2</Level>
<Task>2</Task>
<Opcode>0</Opcode>
<Keywords>0x8000000000000000</Keywords>
<TimeCreated SystemTime="2014-06-12T13:22:20.783351100Z" />
<EventRecordID>829</EventRecordID>
<Correlation ActivityID="{BACB36F3-F55B-49E8-ACEF-0D549F2154DE}" />
<Execution ProcessID="11740" ThreadID="6988" />
<Channel>Microsoft-Exchange-ManagedAvailability/Monitoring</Channel>
<Computer>NCITSMBX02.domain.local</Computer>
<Security UserID="S-1-5-18" />
</System>
<UserData>
<EventXML xmlns:auto-ns2="http://schemas.microsoft.com/win/2004/08/events" xmlns="myNs">
<HealthSet>Search</HealthSet>
<Subject>E15: Search has unhealthy copy status for mounted database 'Mailbox Database 2031363269'</Subject>
<Message>Search has unhealthy copy status for mounted database 'Mailbox Database 2031363269'. The last probe error was:
Search catalog is in 'FailedAndSuspended' state for mounted database 'Mailbox Database 2031363269'. Failover has been tried already but did not solve the problem.
Status of all copies of the database among servers:
Copy: 'Mailbox Database 2031363269\NCITSMBX02', Database Status: 'Mounted', Content Index Status: 'FailedAndSuspended', Error Message: 'The content index is corrupted.'.
</Message>
<Monitor>SearchMountedCopyStatusMonitor/Mailbox Database 2031363269</Monitor>
</EventXML>
</UserData>
</Event> -
Database 'Servername' is low on log volume space. 'DBname' is low on log volume space.
Hello Everyone,
I have Exchange 2013 up and running with some users on working with no issues. However, I keep receiving these alerts from SCOM 2012. I also see the below Event in the Monitoring logs.
I have attempted to run the following cmdlets but I have seen no change.
How can I change the threshold of this alert? I read some places to change in SCOM and others in Exchange. I have no success.
Attempted Overrides:
Add-ServerMonitoringOverride
-ItemType Monitor –Server mbx01 -Identity
dataprotection\lowlogvolumespacemonitor\E: -PropertyName MonitoringThreshold
-PropertyValue 50000 -ApplyVersion 15.0.913.22
Add-ServerMonitoringOverride
-ItemType Monitor –Server mbx01 -Identity
diskspace\lowlogvolumespacemonitor\E: -PropertyName MonitoringThreshold
-PropertyValue 50000 -ApplyVersion 15.0.913.22
Alerts:
SCOM Email
Alert: Health Set unhealthy
Source: MBX03 - DiskSpace
Path: MBX03.domain.local;MBX03.domain.local
Last modified by: System
Last modified time: 10/24/2014 11:06:59 AM Alert description: Database 'MBX03-DB1' is low on log volume space. 'MBX03-DB1' is low on log volume space. Current=150.53 GB, Threshold=195.31 GB
Event log alert:
Database 'MBX01-DB3' is low on log volume space. 'MBX01-DB3' is low on log volume space. Current=161.68 GB, Threshold=195.31 GB
Thanks for any information,
Ian...Hi,
I searched and found that the issue is happening because we have a hard coded limit of 195.31GB for log drive space. If there isn’t that much free, the this error happens.
To work around this issue, you can modify the value of a registry key entry by changing the default threshold from 195.31GB to 50GB. It appears to be system defined that we can only change with a registry key entry.
HKLM\Software\Microsoft\ExchangeServer\v15\Replay\Parameters\SpaceMonitorLowSpaceThresholdInMB
REG_DWORD c350 (50000)
In order for this to take effect, a restart of the server is needed.
Best regards,
If you have feedback for TechNet Subscriber Support, contact
[email protected]
Belinda Ma
TechNet Community Support -
Since we installed CU6 last thursday evening, we are now getting the below healthset alerts:
Alert: Health Set unhealthy
Source: SERVNAME - Compliance
Path: Servername.domain.com;servname.domain.com
Last modified by: System
Last modified time: 9/30/2014 9:33:22 AM Alert description: Hold errors. Exception Details: .
Probe Exception: 'Microsoft.Exchange.Data.Storage.DumpsterOperationException: Copy to dumpster failed. The error message is Move 1 item(s) to the dumpster folder RecoverableItemsDeletions failed on mailbox /o=Contoso/ou=CS/cn=Recipients/cn=finance/cn=JKorth,
which is on hold. This operation is from client Client=MSExchangeRPC and its result is PartiallySucceeded. ---> Microsoft.Exchange.Data.Storage.PartialCompletionException: Move/Copy messages failed. ---> Microsoft.Mapi.MapiExceptionPartialCompletion:
MapiExceptionPartialCompletion: Unable to copy message(s). (hr=0x40680, ec=0) Diagnostic context:
Is anyone else experiencing this? I've traversed all the logs on the servers that its happening on but cant find much to go on. Just that the process failed, but no reason as to why it failed. This started immediately after CU6 was installed.
RichHi Rich,
How about the Exchange 2013 CU6 workflow, does it work fine? If there is anything abnormal or some operations can't be performed, please give a response.
I suggest move JKorth's mailbox to another database and restart MSExchangeMailboxAssistants service for test.
Thanks
If you have feedback for TechNet Subscriber Support, contact
[email protected]
Mavis Huang
TechNet Community Support -
Wac discovery has failed Alerts form all mailbox/cas servers after installing CU5
Is anyone else seeing this alert after installing CU5?
Everything works normally.. Just the odd new alert.
Each of the 6 servers issues this alert about once a day.
Will simply setting the WACdiscoveryEndpoint clear this up?
Like this?
Set-OrganizationConfig -WACDiscoveryEndpoint "https://officeappserver.domain.locla/hosting/discovery"
Alert below
Subject: Alert: Health Set unhealthy Resolution state: New
Alert: Health Set unhealthy
Source: MBX-1 - OWA.Protocol.Dep
Path: MBX-1.rl.gov;MXB-1.rl.gov
Last modified by: System
Last modified time: 7/7/2014 6:53:11 AM
Alert description: Wac discovery has failed on server MBX-1. Please
check the Owa web.config on this exchange server for
BlockWacViewingThroughUI app setting. It should not be set to true. Also
please manually hit the Wac Discovery Url configured under
WacUrlHostName in owa web.config and see if it is returning a valid xml
with external-https net zone. If not, please contact wac team (wac911)
regarding the same. If yes, please debug why our exchange server is not
able to hit this Url and parse the external-https net zone.
States of all monitors within the health set:
Note: Data may be stale. To get current data, run: Get-ServerHealth -Identity 'MBX-1' -HealthSet 'OWA.Protocol.Dep'
State Name
TargetResource HealthSet
AlertValue ServerComponent
NotApplicable OwaIMInitializationFailedMonitor
OWA.Protocol.Dep OWA.Protocol.Dep Healthy
None
NotApplicable WacDiscoveryFailureEventMonitor
OWA.Protocol.Dep OWA.Protocol.Dep Healthy
NoneWe do not have OWAS and do not integrate it with Exchange, so we do not need to monitor health of this item.
After checking exchange health I decided to disable monitoring for
WacDiscoveryFailureEventDiscovery by editing xml file.
Browse to %ExchangeInstallationPath%\Microsoft\Exchange\V15\Bin\Monitoring\Config\
and edit WacDiscoveryFailureEventDiscovery.xml file by changing:
the line
Enabled ="true" /> to Enabled ="false" />
Save the file and restart the Microsoft Exchange Health Management service on the server where you modified the .xml file. There should be no downtime applying this change and restarting the service.
This needs to be changed on all servers. -
Help with Autodiscover.Proxy Unhealthy state.
Hello, I am trying to diagnose unhealthy systems in Exchange 2013. Here is my command and output. Lets start with the first one, Autodiscover.Proxy.
[PS] C:\Windows\system32>Get-HealthReport -Server email| where {$_.alertvalue -ne "Healthy" }
Server State HealthSet AlertValue LastTransitionTime MonitorCount
email Offline Autodiscover.Proxy Unhealthy 11/19/2014 10:52... 1
email Online HubTransport Unhealthy 11/24/2014 6:38:... 96
email Online FrontendTransport Unhealthy 9/25/2014 9:28:3... 12
email NotApplicable MSExchangeCertif... Disabled 1/1/0001 12:00:0... 2
I go to follow this article here: http://technet.microsoft.com/en-us/library/ms.exch.scom.autodiscover.proxy%28v=exchg.150%29.aspx
But the issue is that Invoke-MonitoringProbe does not return anything of value to me. Can you help me analyze this output?
[PS] C:\Windows\system32>Invoke-MonitoringProbe Autodiscover.Proxy\AutoDiscoverProxyTestProbe -Server email | Format-Lis
t
RunspaceId : bfa8f7cf-cc0b-4395-b3c8-75ab16fc227c
Server : email
MonitorIdentity : Autodiscover.Proxy\AutoDiscoverProxyTestProbe
RequestId : d677ac2a-43fa-4147-b806-b2f433c5a6e3
ExecutionStartTime : 11/25/2014 3:23:33 PM
ExecutionEndTime : 11/25/2014 3:23:33 PM
Error : Unknown app pool name:
Exception : System.InvalidOperationException: Unknown app pool name:
at Microsoft.Exchange.Monitoring.ActiveMonitoring.ClientAccess.CafeLocalProbe.DoWork(Cancellati
onToken cancellationToken)
at Microsoft.Office.Datacenter.WorkerTaskFramework.WorkItem.Execute(CancellationToken
joinedToken)
at
Microsoft.Office.Datacenter.WorkerTaskFramework.WorkItem.<>c__DisplayClass2.<StartExecuting>b__0()
at System.Threading.Tasks.Task.Execute()
PoisonedCount : 0
ExecutionId : 60170839
SampleValue : 0
ExecutionContext : Probe Absolute Timeout=60000ms, Timeout Value=60000ms, Calculated HttpRequest Timeout=59000ms
FailureContext :
ExtensionXml :
ResultType : Failed
RetryCount : 0
ResultName : d677ac2a43fa4147b806b2f433c5a6e3-AutoDiscoverProxyTestProbe
IsNotified : False
ResultId : 27004887
ServiceName : InvokeNow
StateAttribute1 :
StateAttribute2 :
StateAttribute3 :
StateAttribute4 :
StateAttribute5 :
StateAttribute6 : 0
StateAttribute7 : 0
StateAttribute8 : 0
StateAttribute9 : 0
StateAttribute10 : 0
StateAttribute11 :
StateAttribute12 :
StateAttribute13 :
StateAttribute14 :
StateAttribute15 :
StateAttribute16 : 0
StateAttribute17 : 0
StateAttribute18 : 0
StateAttribute19 : 0
StateAttribute20 : 0
StateAttribute21 :
StateAttribute22 :
StateAttribute23 :
StateAttribute24 :
StateAttribute25 :
Identity : 956989c13cc44e6faf102491a8d7a11b
IsValid : True
ObjectState : New
I'm not seeing any issue right now with Autodiscover but I don't want a larger issue to show up in the near future.Ok, I guess we can do that. A new health set came up today Unhealthy. Its Compliance and in a NotApplicable state. I will try to determine the impact of this and start a different thread on that if I can't figure that one out.
Otherwise I don't see any problem with mail.
More info, not sure if I posted this, but if this helps, the Event Viewer states this:
Log Name: Microsoft-Exchange-ManagedAvailability/Monitoring Source: Microsoft-Exchange-ManagedAvailability Date: 12/23/2014 7:03:08 AM Event ID: 4 Task Category: Monitoring Level: Error Keywords: User: SYSTEM Computer: email.domain.com Description: The Autodiscover.Proxy
health set has detected a problem on EMAIL beginning at 12/22/2014 3:01:12 PM (UTC). The health manager is reporting that recycling the MSExchangeAutodiscoverAppPool app pool has failed to restore health and it has requested the protocol be marked offline.
Attempts to auto-recover from this condition have failed and administrator attention is required.
Details below: MachineName: EMAIL
ServiceName: Autodiscover.Proxy
ResultName: AutodiscoverProxyTestProbe/MSExchangeAutodiscoverAppPool
Error: The remote server returned an error: (500) Internal Server Error.
Exception: System.ApplicationException: The remote server returned an error: (500) Internal Server Error. at Microsoft.Exchange.Monitoring.ActiveMonitoring.ClientAccess.CafeLocalProbe.DoWork(CancellationToken cancellationToken) at Microsoft.Office.Datacenter.WorkerTaskFramework.WorkItem.Execute(CancellationToken
joinedToken) at Microsoft.Office.Datacenter.WorkerTaskFramework.WorkItem.<>c__DisplayClass2.<startexecuting>b__0() at System.Threading.Tasks.Task.Execute() </startexecuting>
<startexecuting>ExecutionContext: Probe Absolute Timeout=60000ms, Timeout Value=60000ms, Calculated HttpRequest Timeout=59000ms FailedResponse after 0 milliseconds. The remote server returned an error: (500) Internal Server Error.
[000.000] Probe Absolute Timeout=60000ms, Timeout Value=60000ms, Calculated HttpRequest Timeout=59000ms [000.000] Starting HTTP request task [000.000] Waiting 59000 ms [000.000] Issuing GET against https://autodiscover.domain.com/AutoDiscover/ [000.000] Awaiting
GET response [000.000] Performing SSL validation [000.000] Failed with exception: The remote server returned an error: (500) Internal Server Error. </startexecuting>
<startexecuting>FailureContext:</startexecuting>
<startexecuting></startexecuting>ResultType: Failed
IsNotified: False
DeploymentId: 0
RetryCount: 0
ExtensionXml:
StateAttribute1: No response headers available.
StateAttribute2: [email protected] cfj>M!T@O-;XNkj+=u8[SL#f8Oby*S:(&Bg@GTal_=R@3YXtGi=%Vj832L_AE|l>Jhy18K/an^cNHv7i*3-8d*9?#FQa8u!IUoAai-mr(&PG|ZALs2&?6hI2N]9NKK][
StateAttribute3:
StateAttribute4:
StateAttribute5:
StateAttribute6: 0
StateAttribute7: 0
StateAttribute8: 0
StateAttribute9: 0
StateAttribute10: 0
StateAttribute11:
StateAttribute12:
StateAttribute13:
StateAttribute14:
StateAttribute14:
StateAttribute16: 0
StateAttribute17: 0
StateAttribute18: 0
StateAttribute19: 0
StateAttribute20: 0
StateAttribute21:
StateAttribute22:
StateAttribute23:
StateAttribute24:
StateAttribute25:
PoisonedCount: 0
Client Access Array: Client Access Array name could not be retrieved.
ExecutionId: 30334093
ExecutionStartTime: 12/23/2014 12:03:08 PM
ExecutionEndTime: 12/23/2014 12:03:08 PM
ResultId: 32263287
SampleValue: 0 Event Xml:
<event style="font-size:0.75em;line-height:1.5;" xmlns="http://schemas.microsoft.com/win/2004/08/events/event"><system><provider guid="{C424A887-A89F-455F-8319-960917152221}" name="Microsoft-Exchange-ManagedAvailability"><eventid>4</eventid>
<version>0</version> <level>2</level> <task>2</task> <opcode>0</opcode> <keywords>0x8000000000000000</keywords> <timecreated systemtime="2014-12-23T12:03:08.889029200Z"><eventrecordid>7753</eventrecordid>
<correlation activityid="{ED377619-21A3-44A7-9444-751CDE95B0A1}"><execution processid="4204" threadid="14216"><channel>Microsoft-Exchange-ManagedAvailability/Monitoring</channel> <computer>email.domain.com</computer>
<security userid="S-1-5-18"></security></execution></correlation></timecreated></provider></system> <userdata><eventxml xmlns="myNs" xmlns:auto-ns2="http://schemas.microsoft.com/win/2004/08/events"><healthset>Autodiscover.Proxy</healthset>
<subject>Exchange Server Alert: The Autodiscover.Proxy health set is unhealthy.</subject> <message>The Autodiscover.Proxy health set has detected a problem on EMAIL beginning at 12/22/2014 3:01:12 PM (UTC). The health manager is reporting
that recycling the MSExchangeAutodiscoverAppPool app pool has failed to restore health and it has requested the protocol be marked offline. Attempts to auto-recover from this condition have failed and administrator attention is required. Details below: <b>MachineName:</b>
EMAIL <b>ServiceName:</b> Autodiscover.Proxy <b>ResultName:</b> AutodiscoverProxyTestProbe/MSExchangeAutodiscoverAppPool <b>Error:</b> The remote server returned an error: (500) Internal Server Error. <b>Exception:</b>
System.ApplicationException: The remote server returned an error: (500) Internal Server Error. at Microsoft.Exchange.Monitoring.ActiveMonitoring.ClientAccess.CafeLocalProbe.DoWork(CancellationToken cancellationToken) at Microsoft.Office.Datacenter.WorkerTaskFramework.WorkItem.Execute(CancellationToken
joinedToken) at Microsoft.Office.Datacenter.WorkerTaskFramework.WorkItem.<>c__DisplayClass2.<StartExecuting>b__0() at System.Threading.Tasks.Task.Execute() <b>ExecutionContext:</b> Probe Absolute Timeout=60000ms, Timeout Value=60000ms,
Calculated HttpRequest Timeout=59000ms FailedResponse after 0 milliseconds. The remote server returned an error: (500) Internal Server Error. [000.000] Probe Absolute Timeout=60000ms, Timeout Value=60000ms, Calculated HttpRequest Timeout=59000ms [000.000]
Starting HTTP request task [000.000] Waiting 59000 ms [000.000] Issuing GET against https://autodiscover.domain.com/AutoDiscover/ [000.000] Awaiting GET response [000.000] Performing SSL validation [000.000] Failed with exception: The remote server returned
an error: (500) Internal Server Error. <b>FailureContext:</b> <b>ResultType:</b> Failed <b>IsNotified:</b> False <b>DeploymentId:</b> 0 <b>RetryCount:</b> 0 <b>ExtensionXml:</b> <b>StateAttribute1:</b>
No response headers available. <b>StateAttribute2:</b> [email protected] cfj>M!T@O-;XNkj+=u8[SL#f8Oby*S:(&Bg@GTal_=R@3YXtGi=%Vj832L_AE|l>Jhy18K/an^cNHv7i*3-8d*9?#FQa8u!IUoAai-mr(&PG|ZALs2&?6hI2N]9NKK][
<b>StateAttribute3:</b> <b>StateAttribute4:</b> <b>StateAttribute5:</b> <b>StateAttribute6:</b> 0 <b>StateAttribute7:</b> 0 <b>StateAttribute8:</b> 0 <b>StateAttribute9:</b>
0 <b>StateAttribute10:</b> 0 <b>StateAttribute11:</b> <b>StateAttribute12:</b> <b>StateAttribute13:</b> <b>StateAttribute14:</b> <b>StateAttribute14:</b> <b>StateAttribute16:</b>
0 <b>StateAttribute17:</b> 0 <b>StateAttribute18:</b> 0 <b>StateAttribute19:</b> 0 <b>StateAttribute20:</b> 0 <b>StateAttribute21:</b> <b>StateAttribute22:</b> <b>StateAttribute23:</b>
<b>StateAttribute24:</b> <b>StateAttribute25:</b> <b>PoisonedCount:</b> 0 <b>Client Access Array:</b> Client Access Array name could not be retrieved. <b>ExecutionId:</b> 30334093 <b>ExecutionStartTime:</b>
12/23/2014 12:03:08 PM <b>ExecutionEndTime:</b> 12/23/2014 12:03:08 PM <b>ResultId:</b> 32263287 <b>SampleValue:</b> 0</message> <monitor>AutodiscoverProxyTestMonitor/MSExchangeAutodiscoverAppPool</monitor></eventxml></userdata></event> -
Office Web Apps - "Could not find trace string in ULS logs" unhealthy?
I have reviewed everything I could find on unhealthy WAC clusters as my problem seems unrelated to certificate or missing components. I've already digested
http://www.wictorwilen.se/office-web-apps-server-2013---machines-are-always-reported-as-unhealthy (Thanks Wictor).
The particular configuration is an Office Web Apps 2013 ([X-OfficeVersion, 15.0.4551.1005]), running on top of Windows Server 2012, configured for http access (SSL offloaded NLB cluster) and finally linked to Exchange 2013, Lync 2013 and SharePoint
2013. Everything works as expected from client side after setting IIS ARR to handle all reverse proxy bits.
FarmOU :
InternalURL : https://officeapps.fqdn/
ExternalURL : https://officeapps.fqdn/
AllowHTTP : True
SSLOffloaded : True
CertificateName :
EditingEnabled : True
LogLocation : C:\ProgramData\Microsoft\OfficeWebApps\Data\Logs\ULS
LogRetentionInDays : 7
LogVerbosity : Unexpected
Proxy :
CacheLocation : C:\ProgramData\Microsoft\OfficeWebApps\Working\d
MaxMemoryCacheSizeInMB : 75
DocumentInfoCacheSize : 5000
CacheSizeInGB : 15
ClipartEnabled : False
TranslationEnabled : False
MaxTranslationCharacterCount : 125000
TranslationServiceAppId :
TranslationServiceAddress :
RenderingLocalCacheLocation : C:\ProgramData\Microsoft\OfficeWebApps\Working\waccache
RecycleActiveProcessCount : 5
AllowCEIP : False
ExcelRequestDurationMax : 300
ExcelSessionTimeout : 450
ExcelWorkbookSizeMax : 50
ExcelPrivateBytesMax : -1
ExcelConnectionLifetime : 1800
ExcelExternalDataCacheLifetime : 300
ExcelAllowExternalData : True
ExcelWarnOnDataRefresh : True
OpenFromUrlEnabled : False
OpenFromUncEnabled : True
OpenFromUrlThrottlingEnabled : True
PicturePasteDisabled : True
RemovePersonalInformationFromLogs : False
AllowHttpSecureStoreConnections : False
Machines : {WAC15PD-02, WAC15PD-01}
The problem however is an incessant logging on the WAC cluster nodes of event 1204,2204 followed almost immediately by 1004,2004. This repeats every 4min or so...
- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
- <System>
<Provider
Name="Office Web Apps Monitoring" />
<EventID
Qualifiers="0">1204</EventID>
<Level>2</Level>
<Task>1</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated
SystemTime="2014-02-04T20:49:37.000000000Z" />
<EventRecordID>3043246</EventRecordID>
<Channel>Microsoft Office Web Apps</Channel>
<Computer>wac15pd-01.fqdn</Computer>
<Security
/>
</System>
- <EventData>
<Data><?xml version="1.0" encoding="utf-16"?> <HealthReport xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <HealthMessage>UlsControllerWatchdog reported status for UlsController in category 'Verify Trace Logging'. Reported status: Could not find trace string in ULS logs in C:\ProgramData\Microsoft\OfficeWebApps\Data\Logs\ULS.</HealthMessage>
<ComponentOwner>ServicesInfrastructure</ComponentOwner>
</HealthReport></Data>
</EventData>
</Event>
- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
- <System>
<Provider
Name="Office Web Apps Monitoring" />
<EventID
Qualifiers="0">2204</EventID>
<Level>2</Level>
<Task>1</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated
SystemTime="2014-02-04T20:49:37.000000000Z" />
<EventRecordID>3043247</EventRecordID>
<Channel>Microsoft Office Web Apps</Channel>
<Computer>wac15pd-01.fqdn</Computer>
<Security
/>
</System>
- <EventData>
<Data><?xml version="1.0" encoding="utf-16"?> <HealthReport xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <HealthMessage>UlsControllerWatchdog reported status for UlsController in category 'Verify Trace Logging'. Reported status: Could not find trace string in ULS logs in
C:\ProgramData\Microsoft\OfficeWebApps\Data\Logs\ULS.</HealthMessage> <ComponentOwner>ServicesInfrastructure</ComponentOwner>
</HealthReport></Data>
</EventData>
</Event>
- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
- <System>
<Provider
Name="Office Web Apps Monitoring" />
<EventID
Qualifiers="0">1004</EventID>
<Level>2</Level>
<Task>10002</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated
SystemTime="2014-02-04T20:49:39.000000000Z" />
<EventRecordID>3043266</EventRecordID>
<Channel>Microsoft Office Web Apps</Channel>
<Computer>wac15pd-01.fqdn</Computer>
<Security
/>
</System>
- <EventData>
<Data><?xml version="1.0" encoding="utf-16"?> <HealthReport xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <HealthMessage>AgentManagerWatchdog reported status for
AgentManagerWatchdog in category 'Recent Watchdog Reports'. Reported status: Machine health is Unhealthy</HealthMessage> </HealthReport></Data>
</EventData>
</Event>
- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
- <System>
<Provider
Name="Office Web Apps Monitoring" />
<EventID
Qualifiers="0">2004</EventID>
<Level>2</Level>
<Task>10002</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated
SystemTime="2014-02-04T20:49:39.000000000Z" />
<EventRecordID>3043267</EventRecordID>
<Channel>Microsoft Office Web Apps</Channel>
<Computer>wac15pd-01.fqdn</Computer>
<Security
/>
</System>
- <EventData>
<Data><?xml version="1.0" encoding="utf-16"?> <HealthReport xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <HealthMessage>AgentManagerWatchdog reported status for
AgentManagerWatchdog in category 'Recent Watchdog Reports'. Reported status: Machine health is Unhealthy</HealthMessage> </HealthReport></Data>
</EventData>
</Event>
Further exploration of ULS log files (C:\ProgramData\Microsoft\OfficeWebApps\Data\Logs\ULS) did not yield particularly much, except the following;
02/04/2014 20:48:04.48 UlsControllerWatchdog.exe (0x1244) 0x0F60 Services Infrastructure Uls Controller Watchdog ajbam Assert
We're about to trace a string for category MsoSpUlsControllerWatchdog at level Info and we expect to find in the log later, but it appears that the category has been throttled. We will never be able to find the string and this watchdog will always fail.
StackTrace: at Microsoft.Office.Web.UlsControllerWatchdog.Program.CheckServiceInstance(ServiceInstance serviceInstance) at Microsoft.Office.Web.Common.WatchdogHelperThreadManager.GetHealthResults(WatchdogExecutionContext
context, ServiceInstance si) at Microsoft.Office.Web.Common.WatchdogHelperThreadManager.WatchingThreadMethod(Object o) at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback
callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Ru... 345fbec5-e958-4f1f-bf56-d65c1c0d472a
02/04/2014 20:48:04.48* UlsControllerWatchdog.exe (0x1244) 0x0F60 Services Infrastructure Uls Controller Watchdog ajbam Assert
...n(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
at System.Threading.ThreadPoolWorkQueue.Dispatch() 345fbec5-e958-4f1f-bf56-d65c1c0d472a
02/04/2014 20:48:05.52 UlsControllerWatchdog.exe (0x1244) 0x0F60 Services Infrastructure Services Infrastructure Health adhog Unexpected Health report
by UlsControllerWatchdog: Agent: UlsController, eventId: 1204, eventType: Error, categoryId: 1, eventMessage: <?xml version="1.0" encoding="utf-16"?> <HealthReport xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <HealthMessage>UlsControllerWatchdog reported status for UlsController in category 'Verify Trace Logging'. Reported
status: Could not find trace string in ULS logs in C:\ProgramData\Microsoft\OfficeWebApps\Data\Logs\ULS.</HealthMessage> <ComponentOwner>ServicesInfrastructure</ComponentOwner> </HealthReport> 345fbec5-e958-4f1f-bf56-d65c1c0d472a
02/04/2014 20:48:05.52 UlsControllerWatchdog.exe (0x1244) 0x0F60 Services Infrastructure Services Infrastructure Health adhoh Unexpected Health report
by UlsControllerWatchdog (persistent): Agent: UlsController, eventId: 2204, eventType: Error, categoryId: 1 345fbec5-e958-4f1f-bf56-d65c1c0d472a
I suspect these might be related, but can't seem to find any logical explanation why this should cause the Get-OfficeWebAppsMachine to report HealthStatus of Unhealthy. If related, is there a way to disable this check or remove throttling in a safe
way? Alternatively if this is some coding issue (I've not found any other blog/QA dealing with this particularly) it would be nice to get confirmation of this and potentially a fix/solution.
Any help would be greatly appreciated. Thank you!Hi ChristiaanB,
You get this ULS error because you change the log verbosity of the OWA farm. I wrote an article for this on my blog : OWA unhealthy uls issue
Regards,
Wes -
Performance Counter monitors stay unhealthy even when values drop below thresholds
I'm investigating some (SCOM 2012) alerts on our Exchange 2013 (SP1) environment:
Malware filtering is taking too long (90th percentile)
Mailbox Transport Submission is not keeping up with the work...
Queue Alert: Internal Aggregate Delivery Queue (Normal Priority) exceeds threshold
The total number of messages in shadow queues exceeds 1500
All the monitors are Performance Counter based monitors. When I investigate those performance counters (found via the Crimson Channels in the Eventlogs ../ActiveMonitoring/Monitordefinition) all affected counters have already dropped below threshold values
but the associated Health Sets stay unhealthy
IE: Get-ServerHealth -Identity '<Server>' -HealthSet '<HealthSet>' still reports Total.Shadow.Queue.Length.Above.Threshold.Monitor as UnHealthy while perfmon reports values way below 1500 for that particular server.
How to (re)evaluate the monitors ?For investigation purposes I've added some perfmoncounters in SCOM
Another server now alerts 'Total number of messages in shadow queues exceeds 1500'. On the specific server the treshold never reached 1500; it was about 5 at the time the alert fired ?!?
Now it seems the alerts fire for no reason and cannot be reset (I want Managed Availability to reset the alert, not scom)
The XML of Eventlog\Applications and Serviices/Microsoft/Exchange/ActiveMonitoring/MonitorDefinition/
- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
- <System>
<Provider Name="Microsoft-Exchange-ActiveMonitoring" Guid="{ECD64F52-A3BC-47B8-B681-A11B7A1C8770}" />
<EventID>7</EventID>
<Version>0</Version>
<Level>4</Level>
<Task>7</Task>
<Opcode>0</Opcode>
<Keywords>0x4000000000000000</Keywords>
<TimeCreated SystemTime="2014-07-08T12:08:40.460212000Z" />
<EventRecordID>15316181</EventRecordID>
<Correlation />
<Execution ProcessID="38980" ThreadID="30424" />
<Channel>Microsoft-Exchange-ActiveMonitoring/MonitorDefinition</Channel>
<Computer>XXXX</Computer>
<Security UserID="S-1-5-18" />
</System>
- <UserData>
- <EventXML xmlns:auto-ns2="http://schemas.microsoft.com/win/2004/08/events" xmlns="myNs">
<Id>577</Id>
<AssemblyPath>C:\Program Files\Microsoft\Exchange Server\V15\Bin\Microsoft.Office.Datacenter.ActiveMonitoringLocal.dll</AssemblyPath>
<TypeName>Microsoft.Office.Datacenter.ActiveMonitoring.OverallConsecutiveSampleValueAboveThresholdMonitor</TypeName>
<Name>Total.Shadow.Queue.Length.Above.Threshold.Monitor</Name>
<WorkItemVersion>[null]</WorkItemVersion>
<ServiceName>HubTransport</ServiceName>
<DeploymentId>0</DeploymentId>
<ExecutionLocation>[null]</ExecutionLocation>
<CreatedTime>2014-07-08T12:08:40.4602120Z</CreatedTime>
<Enabled>1</Enabled>
<TargetPartition>[null]</TargetPartition>
<TargetGroup>[null]</TargetGroup>
<TargetResource />
<TargetExtension>[null]</TargetExtension>
<TargetVersion>[null]</TargetVersion>
<RecurrenceIntervalSeconds>0</RecurrenceIntervalSeconds>
<TimeoutSeconds>30</TimeoutSeconds>
<StartTime>2014-07-08T12:08:40.4602120Z</StartTime>
<UpdateTime>2014-07-08T12:04:53.7722193Z</UpdateTime>
<MaxRetryAttempts>0</MaxRetryAttempts>
<ExtensionAttributes>[null]</ExtensionAttributes>
<SampleMask>EDS/Performance Counter/MSExchangeTransport Shadow Redundancy Host Info\Shadow Queue Length\_total</SampleMask>
<MonitoringIntervalSeconds>600</MonitoringIntervalSeconds>
<MinimumErrorCount>0</MinimumErrorCount>
<MonitoringThreshold>1500</MonitoringThreshold>
<SecondaryMonitoringThreshold>1</SecondaryMonitoringThreshold>
<ServicePriority>2</ServicePriority>
<ServiceSeverity>0</ServiceSeverity>
<IsHaImpacting>0</IsHaImpacting>
<CreatedById>50</CreatedById>
<InsufficientSamplesIntervalSeconds>28800</InsufficientSamplesIntervalSeconds>
<StateAttribute1Mask>[null]</StateAttribute1Mask>
<FailureCategoryMask>0</FailureCategoryMask>
<ComponentName>ServiceComponents/HubTransport/High</ComponentName>
<StateTransitionsXml>[null]</StateTransitionsXml>
<AllowCorrelationToMonitor>0</AllowCorrelationToMonitor>
<ScenarioDescription>[null]</ScenarioDescription>
<SourceScope>[null]</SourceScope>
<TargetScopes>[null]</TargetScopes>
<Version>65536</Version>
</EventXML>
</UserData>
</Event>
Maybe you are looking for
-
I am planning to create custom defined DSO Object & Info cube
Hi , i am planning to create custom defined DSO Object & Info cube.what ratio i can calculate what is the keyfields & what are the data fields in DSO.How can i calculate. 2. how can i create suitable dimensions, su
-
Reference field in editable alv
hi to all experts, in my alv report there are 2 editable fields chckbox and menge im building up the catalog with the fields description as QTY TO PRINT if i assign the reference fieldname and reference tabname to field catalog im getting error descr
-
SQL Syntax for hour/date range in Query
Hi I am trying to set up an query for sales order documents procesed in the last 30 minutes to be set as an alert to be run every 30 minutes to the sales manager. I am having difficulty getting the syntax for the last 30 minutes Any suggestions? Dav
-
Can I run master collection CS6 on a Mac and PC? What options do I have?
I'm thinking about purchasing the Master Collection of CS6. My school is offering a great deal, but I have a Mac right now and would like to change over to a PC in 1-2 years, so I was wondering what I should do since I either have to buy a Mac or Win
-
Multiple channel data with different time stamp logging
I have peaks and valleys from multiple channels with different time stamp but I am not able to log to a single file. Any idea on how to do this? Attachments: Single Point Peak Valley Log.vi 53 KB