Have Web App/Transaction monitoring alert only when majority of watcher nodes are in error?

Question

We do a lot of web app/transaction monitoring and we have watcher nodes distributed globally. When we have connectivity issues at a site hosting a watcher node, the web monitors that target the watcher node in that site all alert even though all the other
watcher nodes are fine.
We'd like to reduce the noise from our web monitoring by having alerts fire only when a majority, or some other defined ratio, of watcher nodes are in an error condition. Is this possible, and if so, how?
For reference, we're running 2012 R2 and monitor hundreds of URLs. This list of URLs and watcher nodes is fairly dynamic, so we'd prefer to find a solution that doesn't require manual development (e.g. creating distributed apps), however if that's the only
path then it would also be good to know.

Answer 1

First, disable the alerts from the web application that you have created.
Next, create a new distributed application.  Use the 'Blank (Advanced)' template.  When it comes to choosing the management pack, set it to be the same as the one used for the web monitor.
Once the distributed application designer has loaded, client the 'Add Component' button.
Give it a name - Something like 'Appname web monitor component group'
Choose the 'objects of the following type(s)' radio button.  Navigate the tree:
Entity -> Logical Entity -> Perspective -> Web Application Perspective -> YOURWEBMONITOR
Tick the box next to the name of your web monitor.
Now, on the left, in the Objects bar, you should see the watcher nodes for your web application.  Add ALL the watcher nodes into the component group by dragging them in.
Now save your distributed application.
Click the title of the component group that you have just created, down the bottom right of the window there will be a link called 'Configure Health Rollup'.  Click that.
In the override properties window, Tick the box next to the 'Rollup Algorithm' and change the Override Value to Worst state of a percentage of members in good health state' and override 'percentage' value into your expected value.
OK the changes and then save the distributed application once again.
Before you exit the designer, note the name of the component group that you created.
Finally, we have to enable alerting for that component group.  To do this, exit the designer and you should be back in the Authoring pane of the console.
Go to "Management Pack Objects\Monitors" in the console.  Click on the Scope toolbar button to  scope the view -
if a scope was previously active, this will disable the scoped view, so click the button again.  You should be presented with a "Scope Management Pack Objects Window"
Click the "Clear All" button at the bottom of that window, then type in the
name of the component group that you noted down earlier into the search box.  A single line should be displayed in the search results.  Tick the box and click OK.
There should now be a yellow bar saying that the view has been scoped to your web application component group.  Underneath will be a tree view of the monitors in your web applicaion.
Expand the tree view until you find a monitor called "Component Group Health Roll-up for type..."
Open the properties of that item, go to the alerting tab and
enable the "generate alerts for this monitor" checkbox.
You can then fill in the alert name and description as well as other things like severity and whether or not the alert will close when the monitor is healthy.
Roger