Alerting

How to : Schedule an Alert everytime a job fails more than once within an hour

christinaef07
Loves-to-Learn Everything

Hello, I am trying to create an Alert on Splunk. I want to create an alert so that I am alerted every time a job fails 2 times or more within an hour. We have several different jobs running. Right now, I have a table displaying each job with the amount of failures of each. 

 

 

index=?? uuid=* |search status=success | rex "message=(?<message>.*)" | stats count(eval(status=="failed")) AS Failures by workflow_name | table workflow_name, Failures

 

 

This displays something like : 

workflow_name        Failures

workflow_1                 3

workflow_2                 1

workflow_3                7

How can I fix this to filter and only include the workflows that have failed more than once (workflow_1 & workflow_3) and within a specific time frame - 1 hr.  Additionally, I want to pull in info about the specific workflow with the latest failure (for ex: message, uuid, etc). For ex:

 

workflow_name        Failures.       Latest message       Latest uuid 

workflow_1                 3                        error msg                    12345678

workflow_3                7                          error msg                  98765432

 

Labels (3)
0 Karma

aohls
Contributor

A where clause at the end of you query should do it; | where Failures > 1. Then you could schedule the job to run on whatever time frame you need.

0 Karma
Get Updates on the Splunk Community!

Index This | I’m short for "configuration file.” What am I?

May 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with a Special ...

New Articles from Academic Learning Partners, Help Expand Lantern’s Use Case Library, ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Your Guide to SPL2 at .conf24!

So, you’re headed to .conf24? You’re in for a good time. Las Vegas weather is just *chef’s kiss* beautiful in ...