I have a dashboard that has a pull-down menu with a list of our hosts. By selecting a host, one can get a snapshot of that host's status- paging space use, web server hits, vmstat data, SAN disk stats, etc... (Note: because we're on AIX, we cannot run the *nix app, so we get our paging/vmstat data through commands sent to stdout and captured into Splunk).
Unfortunately, it takes a while to display all the results on the dashboard. When you have 20 hosts to check, it adds up fast.
I was thinking it would be a good idea to use summary indexing to speed things up a bit. I'm just not sure what the best way to structure this should be. Should I create a summary index "si-paging" (and "si-san", "si-webhits", etc...) that will capture all that information across all hosts every few minutes, then call that search when I select a host from the menu? Or should I do it from the other angle and create a summary index for each host, containing its paging/SAN/web server/etc data?
I'm thinking the latter would be better; from what I read, you can't add a search parameter when you call a saved search from a summary index. From the docs: "The search against the summary index cannot create or modify fields before the | stats command. That means I wouldn't be able to add a "host = 'xyz'". Do I have that right?
Is my logic sound here? Is it best to create a summary index for each host and generate my results in the dashboard that way?
asked 26 Oct '10, 13:49
Just use the default 'summary' index, and create 'markers' for each type of populating search. For example, a summary-index populating search could be:
When you are going through the saved search workflow, set earliest to -5m, latest to +0s, schedule the search to run every 5 minutes, and check the "enable summary indexing" checkbox, add the key "marker" and the value "cpu_by_host_5m" or something similar.
Then, to build your dashboard, use the following search or something similar (assuming $host$ is a replacement intention from your drop-down):
answered 26 Oct '10, 20:18