Custom Logical Disk monitor incorrectly flapping between healthy and unhealthy

Question

One of the clients Ops Mgr 2012 SP1 UR8 environments I am supporting has had some custom logical disk monitoring setup; there are 5 groups dynamically populated by logical drives depending on their size (1st group has small drives up to the last group with
very large drives). There is a 'Warning' and 'Critical' Monitor setup per server OS version, the Monitors are not Enabled. There are Overrides applied to each group to enable the Monitor and apply a threshold - different threshold for each group.
During some BAU tuning I could see that some of the above Monitors were appearing as Top-Talking alerts. Further investigation showed that alerts were being triggered by drives that momentarily dropped below the applied threshold. I re-created the Monitors
from 'Simple Threshold' to 'Consecutive Samples' and set the 'Number of Samples' to 6 @ 3 minute intervals.
What I am seeing is that alerts from the above Monitors are still appearing as Top Talkers. When I check the Health Explorer of repeating alerts I can see the disk space is staying the same, below the applied threshold but the health is turning healthy then
back to unhealthy. I have confirmed each noisy Object has the expected threshold as per its dynamic group allocation and have also confirmed the drives are not fluctuating above and below the threshold. One thing I have noticed is that some drives Performance
View is patchy - lots of dotted lines between the coloured lines.
Its almost like the Monitor moves a Logical Disk Object into unhealthy state in the correct (and expected) manner, then it somehow picks up an incorrect threshold which is below the current usage level. This moves it into a healthy state only for the
whole process to repeat. For example: Drive X: on a server is very large, the Group that it sits in has a threshold of 102400MB, its current usage is ~stable at 45500MB. Looking in Health Explorer I can see 3:01pm green state/ 45573 last sampled value/ # of
samples 1 | 3:16pm yellow state/ 45573/ 6 samples | 3:34pm green state/ 45572/ 1 samples | 3:49pm yellow state/ 45571/ 6 samples | 4:01pm green state/ 45425/ 1 sample etc etc.
I'm scratching my head on this one and would appreciate any suggestions or assistance.
Thanks
BT

Answer 1

Thanks for the reply. It is not just one server / drive this is happening on. I am seeing it on everything; once they go into an unhealthy state they periodically go healthy and back again with no change in disk free space. Just to elaborate on how it is
setup; a Monitor has been created for each OS version (2003, 2008 and 2012) and a separate Monitor for Warning and Critical so 6 Monitors in total. Looking at the Warning Monitors; they are created with a threshold of 5120MB for 6 samples and set to disabled.
The following groups have been created and the following thresholds added:
Group 1 (less than 60GB size): override added to enable. This group will then pick up the 5120MB threshold.
Group 2 (60 – 250GB size): override added to enable and override added for 10240MB threshold
Group 3 (250 – 500GB size): override added to enable and override added for 20480MB threshold
Group 4 (500 – 1TB size): override added to enable and override added for 51200MB threshold
Group 5 (>1TB size): override added to enable and override added for 102400MB threshold
One drive I was looking at was in Group 2 (threshold of 10240MB), it was staying at approx. 8500MB but periodically going into healthy state then after 10mins (6 polls @ 2min intervals) back to unhealthy. This process repeats once or twice per day.
I am wondering if the Object is somehow picking up the threshold of the Monitor (5120MB) then going back to its correct overridden threshold. I have setup some test groups and monitors in a lab and will review the results over the coming days.
When the monitors were setup as 'Simple Threshold' this worked fine but were noisy due to drives spiking downwards. It was only when I re-wrote them as 'Consecutive Samples over Threshold' Monitors that this issue has started occurring.
Thanks