Alerting

Raise alert on status change only

atelesca
Explorer

Hello,
I would like to be able to raise an alert when the status of a condition changes?
For example:
- when the CPU user percentage is higher than 80% I would like to raise an alert that launch a command raising the alarm_on to an external program. This should be done only if the previous CPU user percentage was lower than 80%. There is no need to raise the alarm again.
- when the CPU user percentage goes back to below 80% I would like to send a command to my external program that set the alarm_off. This has to be done only if the previous CPU user percentage was higher than 80%.
Do you know how can I do it in Splunk?
I could raise an alert based on a search such as
sourcetype="cpu"| where CPU="all" and pctUser>20
and add the conditions to
sourcetype="cpu"| where CPU="all" and pctUser>20 | stats count=1
so that I am sure that the condition has been met only once.
The problem is:
- I should specify the time, like last 2 minutes
- How can I make sure that this is met by host?

Thanks for your help.
Cheers,
Adriana

Tags (1)
0 Karma

atelesca
Explorer

Well, at the end the only way I found is to create one alert that raise the alarm when the following condition is met:

sourcetype=cpu earliest=-2m | multikv | where CPU="all" | stats count(eval(pctUser > 80)) as a ,count(eval(pctUser < 80)) as b, latest(pctUser) as c, by host | search a =1 AND b = 1 AND c>80

that runs every minute and it looks at the past 2 minutes.

In order to unraise the alarm, I created another alert that is launched when the following condition is met:
sourcetype=cpu earliest=-2m | multikv | where CPU="all" | stats count(eval(pctUser > 80)) as a ,count(eval(pctUser < 80)) as b, latest(pctUser) as c, by host | search a =1 AND b = 1 AND c<80

it also runs every minute and looks at the past 2 minutes.

Get Updates on the Splunk Community!

Stay Connected: Your Guide to May Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars this month. This ...

They're back! Join the SplunkTrust and MVP at .conf24

With our highly anticipated annual conference, .conf, comes the fez-wearers you can trust! The SplunkTrust, as ...

Enterprise Security Content Update (ESCU) | New Releases

Last month, the Splunk Threat Research Team had two releases of new security content via the Enterprise ...