Alerting Diskspace on several hosts after 5 runs

[siegema1]

Dear all,
I'm new to Splunk (eval version) and struggeling with many things.
One thing is and that I don't understand completly is:
I would like to run a search over all hosts and would like to get an alert only (the values are stupid so I get an alert) if one of those hosts has reached the threshold the 5th time in a row. I thought I can do that with:

source="Perfmon:Windows__LogicalDisk" sourcetype="Perfmon:Windows__LogicalDisk" counter="% Free Space" (Value<48 Value>5) | stats count by host

The Alert settings are the follwoing:
Schedule, Basic, every minute
Number of Events > 0
Once per result
Throttling: 5 minutes host

Like it seems I get only one Alert all 6 minutes and not an alert for each host. If I open the View results link then I see both host where the problem exists.
What I do wrong.
Would appreciate that if someone could help me.
Cheers
Markus

[siegema1]

Hi Martin,
sorry I saw your answer only now 😞
Thanks for your reply.
Do you have an idea how I've to setup this search that the first alert occurs only if the problem occurs the fifth time per server?
Thanks and cheers
Markus

[martin_mueller]

You should express that in your search. Compute the last five instances, and write a condition that yields true if the last five instances are above your threshold. The trigger the alert based on that condition.

[martin_mueller · Accepted solution]

Throttling by a field value and sending emails per result works fine over here when using your alert settings.
Don't expect many entries in the Splunk UI ("Trigger History") though, an Alert with ten results is only triggered once. However, the actions for the alert, ie sending an email, were executed ten times.

[martin_mueller]

Yes - the Alert has triggered once, so it appears once in the GUI. It's the alert actions run after triggering that are duplicated per event.

Alert throttling isn't capable of selecting the 5th triggered Alert, it's rather for keeping the Alert quiet for a while after the first triggering.
To make your use case "only select hosts that have five consecutive free space violations" work you need to modify the search accordingly.

[siegema1]

Sorry, I've forgot the other point: with this mentioned setup of the search I get an alert as soon as the problem occurs, and only then each 5th time. What can I do that I don't get an alert already at the first time, but instead after the 5th time?
Thanks

[siegema1]

Hi Martin,
thanks for your reply. Only to be sure that I've understood everything well:
If I setup ONE search, no matter how, and I have for exampele 100 hosts, ten of them have no more enough free disk space, I get only ONE Alert in the GUI, but if setup, for each server a mail action.
Thanks a lot and cheers
Markus