|
I am seeing an issue on our Splunk server where we seem to be hitting a performance bottleneck. When generating charts from searches it takes a long time to render the results. This is especially noticable on dashboards using where multiple graphs may be rendered. As our server is limited in resources we have restricted the concurrent searches to 5 (we patient enough to wait for the results) there it appears that the search itself completes relatively quickly and the jobs report indicates that 98% of these searches take between 0 and 3 seconds to complete. When loading a dashboard it can be seen that multiple splunkd processes are spawned, which I assume are carrying the concurrent searches. These splunkd processes disappear at times which would seem to correspond to the completion times shown in the job report. Around the same time the main splunkd process will rise up to as high at 300% and generate load average of up to 16. This increased CPU usage and corresponding load average remain high until all the the panels within the dashboard have been rendered and this rendering can take a long time. I would be prepared to accept this is delay is reasonable when the search is a long running and retrieves a lot of data, but this lag seems excessive for searches that only take between 0 and 3 seconds. So, it would appear that we are hiiting a bottleneck at the point at which the panels or data used to generate the panels are rendered, maybe queuing or backing up a process which is single threaded. Has anyone else observed this behaviour and is there any way to improve this performance other than increasing the servers CPU? Server specifications are: Example of the diffrence between normal and higher load: |
|
This feels more support-y (or consulting) as opposed to splunk-answers-y. Support may be able to help clarify if there is something glaringly wrong (especially if you have a clearly answerable question to work on). They're not very good at open ended performance goals, though we do have a consulting team who can work on such things. Generally speaking splunk searches are phased, with a period of IO, mixed possibly with high CPU, followed a period of usually high cpu, sometimes mixed with memory bandwidth limitations. It might be instructive to look at how your load changes over time to try to evaluate what's being exhausted. The jobs inspector interface (not trivially to map from a dashboard to a search job, sorry) can provide numbers on where the job spent its wallclock time to help identify as well. Macroscopically if you have several charts which are all essentially presenting information about the same serach concept, it's possible to rework your charts to all use a single search, which obviously reaps large benefits. There should be some examples of achieving this in, but I haven't grokked its secrets. http://splunkbase.splunk.com/apps/All/4.x/app:Splunk+UI+examples+app+for+4.1%2B Search performance can also be tuned by looking at the specific searches and evaluating if there's a faster way to do them, whether they should really be precalculated somehow, etc. More cpu may be wanted regardless, possibly in combination with the other approaches. We do offer professional services to take on these types of things as a whole. They sell in blocks of time and are more focused on ensuring customer success as opposed to some less satisfying incarnations of that term. We do have a support contract so I will take the question to Splunk support. Unfortunately the guy who handles the support is on holiday and impatiently, I asked here to try and get to the bottom of the issue and hoped that someone may have seen something similar before and may have had an easy solution.
(27 Jan '11, 07:54)
mcwomble
|