Splunk Realtime Dashboard Performance

[watsm10]

Hi,
We are having performance issues with Splunk. We haven't got the most powerful server (possibly Dual Core) and things keep coming to a standstill. Specifically around multiple users viewing the same dashboard, saved searches, and realtime searches (I’m wondering if a realtime dashboard can be set up that multiple people can log in to, but that the searches only run once…….. at the moment, the dashboard on our big screen uses real time searches (6 or 8 of them I think) and this happens for every user that opens that view…. Soon kills splunk!)

Would be great if anyone can offer their advice.

[Drainy · Accepted solution]

Well, each time you run a search that search will lock a core, splunkd will need at least one core itself to index and Splunk needs an absolute min of 6 cores for a linux setup and 8 for windows... so... your system is wildly under-spec'ed.

Presumably if it is a dual core then the IOPS available and RAM is also going to be limited. I would look at getting a better spec'ed system.

EDIT - Some links:
Module reference - http://docs.splunk.com/Documentation/Splunk/latest/Developer/ModuleReference
Post Processing - http://docs.splunk.com/Documentation/Splunk/latest/Developer/PostProcess

[watsm10]

We've converted the inline searches to saved searches and that's freed up a lot of resource and our offshore guys can have access to that dashboard at the same time. thanks for your advice 😄

[Drainy]

yes, they have to be savedsearches, it would be a massive resource drain if Splunk tried to match search strings so instead it can find jobs based on their name. Strip the searches out and make them into savedSearches, this will help a fair bit. Post processing is where you run one HiddenSearch at the top to pull in your results, you use a reporting command like fields or table to specify the fields you are interested in and then down your page for a timechart you just do a HiddenPostProcess search to do stats, timechart, reporting etc. Only search once though

[watsm10]

With the useHistory option, does the search have to be a saved search? Ours are embedded in the XML for the dashboard. How does postprocessing work? Thanks

[Drainy]

I've added some links to my answer related to what I included in the last comment

[Drainy]

there is a field called useHistory which by default should look for an existing search job of the same name and use it, you can define it specifically though. More importantly, the 6 or 8 real time searches on the one dashboard are going to be killing it. Sadly this is really just a case that the system isn't fit for purpose. At the very best you could try and cut down the number of searches and use postprocessing to handle filtering and reporting down the xml

[watsm10]

What if upgrading the system isn't an option? Is there a way of specifying that once a dashboard has been opened by User 1, all other users who wish to view the same dashboard will see the current real-time searches running for User 1. Almost like a screen capture tool works.