Hello!
I'm trying to run many queries on a log every day. Is it possible to bundle these searches together, so Splunk doesn't have to iterate over the whole log every time?
I tried searching for an answere here and in the documentation, but I didn't manage to find anything.
Thanks in advance!
Okay, here is how I would do some of these
sourcetype="nginx_log" earliest=-1d@d latest=@d
| eval urlGroup = ""
| eval urlGroup = case(match(url,"^/ws/2/label/"),"/ws/2/label/",
match(url,"^/ws/2/artist/"),"/ws/2/artist/",
match(url,"^/ws/2/release-group/"),"/ws/2/release-group/",
match(url,"^/ws/2/release/"),"/ws/2/release/",
match(url,"^/ws/2/recording/"),"/ws/2/recording/",
match(url,"^/ws/2/work/"),"/ws/2/work/")
| top 50 inc by urlGroup
Would do the first 7 searches as a single search. The eighth search can be, in and of itself, dramatically simplified as follows:
sourcetype="nginx_log" query="*" earliest=-1d@d latest=@d
| fields url
| eval name = ""
| eval name = case(match(url,"^/ws/1/artist/","Artists",
match(url,"^/ws/1/track/"),"Tracks",
match(url,"^/ws/1/release-group/"),"Release-group",
match(url,"^/ws/1/release/"),"Releases",
match(url,"^/ws/1/label/"),"Labels")
| stats count by name
This should be more efficient. The remaining searches could follow this second pattern.
Try it and see what you think.
Okay, here is how I would do some of these
sourcetype="nginx_log" earliest=-1d@d latest=@d
| eval urlGroup = ""
| eval urlGroup = case(match(url,"^/ws/2/label/"),"/ws/2/label/",
match(url,"^/ws/2/artist/"),"/ws/2/artist/",
match(url,"^/ws/2/release-group/"),"/ws/2/release-group/",
match(url,"^/ws/2/release/"),"/ws/2/release/",
match(url,"^/ws/2/recording/"),"/ws/2/recording/",
match(url,"^/ws/2/work/"),"/ws/2/work/")
| top 50 inc by urlGroup
Would do the first 7 searches as a single search. The eighth search can be, in and of itself, dramatically simplified as follows:
sourcetype="nginx_log" query="*" earliest=-1d@d latest=@d
| fields url
| eval name = ""
| eval name = case(match(url,"^/ws/1/artist/","Artists",
match(url,"^/ws/1/track/"),"Tracks",
match(url,"^/ws/1/release-group/"),"Release-group",
match(url,"^/ws/1/release/"),"Releases",
match(url,"^/ws/1/label/"),"Labels")
| stats count by name
This should be more efficient. The remaining searches could follow this second pattern.
Try it and see what you think.
Thank you! Querying takes a significantly shorter amount of time now.
Thank you for your reply! Here is a sample of the queries I'm trying to run:
https://gist.github.com/3499469
The log is a basic nginx log. You can see that most of the queries contain 'top', or many 'count's.
Maybe, it depends on the searches. If you give us a sample (2 or 3) of the searches, and a few lines of the log... we might be able to come up with some ideas for you.
I find that it is often possible to reduce the number of searches, even when you can't bundle all of them together.