Hi all!
I feel as if I'm overcomplicating an issue, but I haven't gotten any built-in Splunk tools to work.
Here's the situation: I have a field that I extract from my logs using rex. I want to be able to take an average AND a standard deviation count of each field's occurrence over the days to be able to detect any new abnormalities of this field. Here's the field extraction:
earliest=-7d@d
latest=-0d@d
index=prod
"<b>ERROR:</b>"
| rex "foo:\ (?<my_foo>[^\ ]*)"
| rex "bar:\ (?<my_bar>[^\<]*)"
| eval my_foo = coalesce(my_foo,"-")
| eval my_bar = coalesce(my_bar, "-")
| rex mode=sed field=my_bar "s/[\d]{2,50}/*/g"
| strcat my_foo " - " my_bar my_foobar
I can use stats to get a total count by my_foobar. And I can use timechart to get a count by day for my_foobar. However, if I try to average by day after timechart, I'll get no output unless I give up my my_foobar discretion.
| timechart span=1d@d count as my_count by my_foobar
| stats avg(my_count)
No output
| bin span=1d@d my_chunk
| stats count(my_script_message) by my_chunk
No output
I did come up with a solution, but it's hideous. I basically made my own bins using joins
<initial search above>
| chart count as my_count1 by my_foobar
| join my_foobar [search
<initial search above with my_count iterated>
<x5 more joins>
| eval my_avg = SUM(my_count1 + my_count2 + my_count3 + my_count4 + my_count5 + my_count6 + my_count7)/7
| eval my_std = (POW((my_count1 - my_avg),2) + POW((my_count2 - my_avg),2) + POW((my_count3 - my_avg),2) + POW((my_count4 - my_avg),2) + POW((my_count5 - my_avg),2) + POW((my_count6 - my_avg),2) + POW((my_count7 - my_avg),2))/7
| eval my_last_day_dev = ABS(my_count1 - my_mess_avg)
| table my_foobar my_avg my_std my_last_day_dev
| search my_last_day_dev > my_std
I hate it and need to use this methodology for many of my monitoring plans. Any ideas on how to make this more sleek?
... View more