Tuning Exchange 2013 HealthSet Monitors (example: MailboxTransport - Mapi.Submit.Monitor)

Question

Hello everyone,
I was hoping some of you could help me out a bit with the new monitors in Exchange 2013.
Especially after updating to CU6 we are getting a lot of warnings from SCOM with different HealthSets that show as unhealthy. These are usually “OK” again when we log on to the server and run Get-ServerHealth or Get-HealthReport. The most common healthset is
Compliance, but also get warnings from MailboxTransport and ActiveSync.
If we do manage to log on before the healthset returns to healthy again, like in the example below, we can drill down a bit. We are however very unsure about what to do with this information. Perhaps we should change the threshold, but how do we find out exactly
what this monitor is reporting on? We know it has something to do with Mapi Submit, and that the threshold is 8, but not much more than that?
Get-ServerHealth SERVER | where {($_.healthsetname -eq "mailboxtransport") -and ($_.alertvalue -eq "unhealthy")} | ft name,alertvalue -auto
Name                AlertValue
Mapi.Submit.Monitor  Unhealthy
(Get-WinEvent -LogName Microsoft-Exchange-ActiveMonitoring/MonitorDefinition | % {[XML]$_.toXml()}).event.userData.eventXml | ?{$_.Name -like "Mapi.Submit.Monitor"}
auto-ns2                           : http://schemas.microsoft.com/win/2004/08/events
xmlns                              : myNs
Id                                 : 266
AssemblyPath                       : D:\Program Files\Microsoft\Exchange Server\V15\Bin\Microsoft.Office.Datacenter.ActiveMonitoringLocal.dll
TypeName                           : Microsoft.Office.Datacenter.ActiveMonitoring.OverallXFailuresMonitor
Name                               : Mapi.Submit.Monitor
WorkItemVersion                    : [null]
ServiceName                        : MailboxTransport
DeploymentId                       : 0
ExecutionLocation                  : [null]
CreatedTime                        : 2014-11-20T08:51:03.5576888Z
Enabled                            : 1
TargetPartition                    : [null]
TargetGroup                        : [null]
TargetResource                     : MailboxTransport
TargetExtension                    : [null]
TargetVersion                      : [null]
RecurrenceIntervalSeconds          : 0
TimeoutSeconds                     : 30
StartTime                          : 2014-11-20T08:51:03.5576888Z
UpdateTime                         : 2014-11-20T08:42:58.9416436Z
MaxRetryAttempts                   : 0
ExtensionAttributes                : [null]
SampleMask                         : Mapi.Submit.Probe
MonitoringIntervalSeconds          : 3600
MinimumErrorCount                  : 0
MonitoringThreshold                : 8
SecondaryMonitoringThreshold       : 0
MonitoringSamplesThreshold         : 100
ServicePriority                    : 2
ServiceSeverity                    : 0
IsHaImpacting                      : 0
CreatedById                        : 57
InsufficientSamplesIntervalSeconds : 28800
StateAttribute1Mask                : [null]
FailureCategoryMask                : 0
ComponentName                      : ServiceComponents/MailboxTransport/High
StateTransitionsXml                : <StateTransitions>
                                       <Transition ToState="Unrecoverable" TimeoutInSeconds="0" />
                                     </StateTransitions>
AllowCorrelationToMonitor          : 0
ScenarioDescription                : [null]
SourceScope                        : [null]
TargetScopes                       : [null]
HaScope                            : Server
Version                            : 65536
We have an Exchange 2007 server coexisting in the environment, but it is not in use and will be decommissioned.
                         ServerRole    Edition AdminDisplayVersion
Mailbox, ClientAccess, HubTransport   Standard Version 8.3 (Build 83.6)
              Mailbox, ClientAccess Enterprise Version 15.0 (Build 995.29)
              Mailbox, ClientAccess Enterprise Version 15.0 (Build 995.29)

Answer 1

Thanks Belinda.
We had a look in the log you mentioned, and found some events that appear to be related.
Mailbox Transport Submission - the verification that the probe messages were successfully submitted within 5 minutes to HUB has failed. 
      This could indicate that the mailbox submissions are taking too long or not progressing at all.
      SequenceNumber: 635515592255510017
      Error: MapiSubmitLAMProbe finished with CheckPreviousMail failure.
      FailureContext: MapiSubmitLAMProbe finished with CheckPreviousMail failure.
      ExecutionContext: MapiSubmitLAMProbe started. This performs - 1. Submits a new message to Store 2. Checks results from previous Send Mail operation. Sequence # = 635515592255510017. First Run? = False. Previous mail submission to store
was successful. Results -  # of previous results: 0.  Could Not Find stages that ran.  Previous SendMail failure -  Mail submitted to Store during the previous run never reached SendAsCheck. 
                        This may indicate a latency from Store to Submission Service. Investigating.  Found lower SA latency. Indicates an issue in Submission service. 
                            Investigate. In SendMail -  NotificationID=00000044-0000-0000-0000-0000659c9601 Sending mail. SendMail finished. MapiSubmitLAMProbe finished with CheckPreviousMail
failure. 
      CheckPreviousRunMail exception details: 
      Current SendMail exception details: 
      MailWasSentInLastRun (0(no)/1(yes)): 1
      CheckMailSuccess(0(no)/1(yes)): 0
      SendMailSuccess(0(no)/1(yes)/2(no-noMDBs)/3(no-noActiveMDBs)/4(no-ignorableException)): 1
      PreviousMailLatencies: <[email protected]>,,2014-11-20T08:37:58.817Z;LSRV=SERVER001.DOMAIN.locale:TOTAL=0.811|SA=0.343|MTSSDMO=0.426|MTSSDPL=0.002|MTSSDSS=0.008|MTSSDS=0.031|MTSS=0.466;;<[email protected]>,,2014-11-20T08:32:58.772Z;LSRV=SERVER001.DOMAIN.locale:TOTAL=0.702|SA=0.203|MTSSDMO=0.451|MTSSDPL=0.003|MTSSDSS=0.007|MTSSDS=0.047|MTSS=0.500;;<[email protected]>,,2014-11-20T08:27:58.664Z;LSRV=SERVER001.DOMAIN.locale:TOTAL=1.170|SA=0.530|MTSSDMO=0.548|MTSSDPL=0.002|MTSSDSS=0.035|MTSSDS=0.078|MTSS=0.647;;
      GetPreviousMailLatencyFailures: 
      (The following are applicable if the mail was actually sent)
      MailSentAt: 20.11.2014 08:42:58
      TimeInSendingToStore: 62,3989
      Sent Mail Result Details:
        (InternetMessageId:<[email protected]>;
        ItemEntryId:0 0 0 0 2E 57 BD C1 1C 4B 5C 47 9E 54 6E B8 B2 C5 75 38 7 0 FA 78 3 5 D1 D4 6 48 B2 FF B 4 D7 25 55 92 0 0 0 0 1 B 0 0 FA 78 3 5 D1 D4 6 48 B2 FF B 4 D7 25 55 92 0 0 18 1A CD F1 0 0 )
        Properties of Sending mail 
        (Subject:00000044-0000-0000-0000-0000659c9601-MapiSubmitLAMProbe;
        Sender:[email protected];
        MbxGuid:340b204b-c716-4dfe-bbe1-cf09b5c73934;
        MdbGuid:2a6b9801-1647-4378-bc11-6ff342d22fcf;
        Recipient:[email protected];
        Body and MessageClass:MessageClass:IPM.Note.MapiSubmitLAMProbe;MessageBody:This is a Probe Mapi message that's Submitted from Store to Mailbox transport Submission service to Hub transport service;
        DoNotDeliver, DropMessageInHub and DeleteAfterSent:DoNotDeliver:True;DropMessageInHub:True;DeleteAfterSubmit:True;)
Probe Exception: 'System.ApplicationException: MapiSubmitLAMProbe finished with CheckPreviousMail failure.
   at Microsoft.Forefront.Monitoring.ActiveMonitoring.Transport.Probes.MapiSubmitLAMProbe.PerformProbeFinalAction(Boolean potentialForAlertBasedOnPreviousRun, Boolean potentialForAlertBasedOnCurrentRun, Exception previousRunVerificationException,
Exception currentRunException, DateTime timeMessageSentToStore)
   at Microsoft.Forefront.Monitoring.ActiveMonitoring.Transport.Probes.MapiSubmitLAMProbe.DoWorkInternal(CancellationToken cancellationToken)
   at Microsoft.Forefront.Monitoring.ActiveMonitoring.Transport.Probes.MapiSubmitLAMProbe.DoWork(CancellationToken cancellationToken)
   at Microsoft.Office.Datacenter.WorkerTaskFramework.WorkItem.Execute(CancellationToken joinedToken)
   at Microsoft.Office.Datacenter.WorkerTaskFramework.WorkItem.<>c__DisplayClass2.<StartExecuting>b__0()
   at System.Threading.Tasks.Task.Execute()'
Probe Result Name: 'Mapi.Submit.Probe'
Probe Result Type: 'Failed'
Monitor Total Value: '0'
Monitor Total Sample Count: '12'
Monitor Total Failed Count: '12'
Monitor Poisoned Count: '0'
Monitor First Alert Observed Time: '16.11.2014 21:40:10'