How to use stats to find abnormal values for an alert?

[drodman29]

I need an efficient design pattern to move a report that generates a table of values on a 24 hr basis to an alert when these values are statistically "abnormal".

Given a report search like this:

{my search} | eval Loaded=mvzip(key,doc_count)  | mvexpand Loaded  | eval Loaded= split(Loaded,",")  | eval KeyName=mvindex(Loaded,0) | eval Doc_Count=tonumber(mvindex(Loaded,1))  | stats   last(Doc_Count) as Mylast first(Doc_Count) as MyFirst by KeyName | eval 24hTotalDiff=MyFirst-Mylast | fields KeyName, 24hTotalDiff

What is the best way to identify abnormal (daily) values of 24hTotalDiff over a xx day sample size?
Should I create a lookup table of average daily values by KeyName? or is there a more elegant solution?

[sundareshr · Accepted solution]

See if one of these will work for you

{my search} | eval Loaded=mvzip(key,doc_count) | mvexpand Loaded | eval Loaded= split(Loaded,",") | eval KeyName=mvindex(Loaded,0) | eval Doc_Count=tonumber(mvindex(Loaded,1)) | bin span=1d _time as t | eval Key=t."#".KeyName | stats last(Doc_Count) as Mylast first(Doc_Count) as MyFirst by Key | eval 24hTotalDiff=MyFirst-Mylast | trendline sma2(24hTotalDiff) as Trend | where Trend>=24hTotalDiff*2 | rex field=Key "(?<Time>[^#]+)#(?<KeyName>.*)" | table Time KeyName 24hTotalDiff Trend

*OR*

{my search} earliest=-7d@d | eval Loaded=mvzip(key,doc_count) | mvexpand Loaded | eval Loaded= split(Loaded,",") | eval KeyName=mvindex(Loaded,0) | eval Doc_Count=tonumber(mvindex(Loaded,1)) | bin span=1d _time as t | eval Key=t."#".KeyName | stats last(Doc_Count) as Mylast first(Doc_Count) as MyFirst by Key | eval 24hTotalDiff=MyFirst-Mylast | eventstats stdev(24hTotalDiff) as sd by Key | where 24hTotalDif>=sd*2 | rex field=Key "(?<Time>[^#]+)#(?<KeyName>.*)" | ...

[drodman29]

The design pattern seems to be "concatenate a time value to the Stats by-clause variable that is consistent with the time based aggregation that you want, then separate the compound key and run eventstats to get sd and avg by the original key."

Posting corrected answer syntax.

{mySearch} | eval Loaded=mvzip(key,doc_count) | mvexpand Loaded | eval Loaded= split(Loaded,",") | eval KeyName=mvindex(Loaded,0) | eval Doc_Count=tonumber(mvindex(Loaded,1)) |  search KeyName!="AFI*" | eval MasterKey=strftime(_time,"%Y-%m-%d" )."#".KeyName | stats latest(Doc_Count) as MyLast earliest(Doc_Count) as MyFirst by MasterKey | eval TotalDiff=tonumber(MyLast-MyFirst)| rex field=MasterKey "(?<YearDate>[^#]+)#(?<KeyName>.*)" | eventstats stdev(TotalDiff) as SD avg(TotalDiff) as AVG by KeyName | table MasterKey, KeyName, TotalDiff, SD , AVG | eval KeyStatus=if(abs(TotalDiff -AVG ) >  SD*2, "RED", "GREEN") | fields MasterKey, KeyName, TotalDiff, SD , AVG, KeyStatus