I have a dashboard with 2 saved searches loaded to it, when I try to load the dashboard it takes around 1-2 minutes for the graphs to render (only 1 is able to render each time - I have no idea why) and the CPU/MEM on the splunk machine spikes.
Here are the two saved searches:
This should show the avg of mem usage for the process 'splunkd' on two hosts
index="os" sourcetype="ps" host=stage.site OR host=testbuild.site | multikv fields pctMEM, COMMAND | where COMMAND="splunkd" | append [stats sum(pctMEM) as pctMEM by _time,COMMAND | eval host=split("stage.site;testbuild.site",";")
| mvexpand host ] | timechart avg(pctMEM) by host | eval count=count-1
This should show the avg of cpu usage for the process 'splunkd' on two hosts
index="os" sourcetype="ps" host=stage.site OR host=testbuild.site | multikv fields pctCPU, COMMAND | where COMMAND="splunkd" | append [stats sum(pctCPU) as pctCPU by _time,COMMAND | eval host=split("stage.site;testbuild.site",";")
| mvexpand host ] | timechart avg(pctCPU) by host | eval count=count-1
The time range for these two searches is defined to -24h@h.
I have another dashboard with the exact same saved searches but with a different time range of -1h@h and that dashboard loads correctly without spiking the mem/cpu usage and it does not hang.
This is how I load the 2 saved searches in my dashboard:
<module name="HiddenSavedSearch" layoutPanel="panel_row2_col1" group="splunkd CPU Usage last 24h" autoRun="True">
<param name="savedSearch">splunkd CPU Usage last 24h</param>
<param name="groupLabel">splunkd CPU Usage last hour (24h)</param>
<module name="HiddenChartFormatter">
<param name="charting.chart.stackMode">default</param>
<param name="charting.secondaryAxisTitle.text">% CPU</param>
<param name="charting.legend.placement">left</param>
<param name="charting.chart">line</param>
<param name="charting.chart.nullValueMode">connect</param>
<module name="JSChart">
<param name="width">100%</param>
<param name="height">280px</param>
<module name="ConvertToDrilldownSearch">
<module name="ViewRedirector">
<param name="viewTarget">flashtimeline</param>
</module>
</module>
</module>
<module name="ViewRedirectorLink">
<param name="viewTarget">flashtimeline</param>
</module>
</module>
</module>
<module name="HiddenSavedSearch" layoutPanel="panel_row2_col2" group="splunkd MEM Usage last 24h" autoRun="True">
<param name="savedSearch">splunkd MEM Usage last 24h</param>
<param name="groupLabel">splunkd MEM Usage last 24h</param>
<module name="HiddenChartFormatter">
<param name="charting.chart.stackMode">default</param>
<param name="charting.secondaryAxisTitle.text">MEM %</param>
<param name="charting.legend.placement">left</param>
<param name="charting.chart">line</param>
<param name="charting.chart.nullValueMode">connect</param>
<module name="JSChart">
<param name="width">100%</param>
<param name="height">280px</param>
<module name="ConvertToDrilldownSearch">
<module name="ViewRedirector">
<param name="viewTarget">flashtimeline</param>
</module>
</module>
</module>
<module name="ViewRedirectorLink">
<param name="viewTarget">flashtimeline</param>
</module>
</module>
</module>
Here are some ideas:
append
command will run by default for only 60 seconds. You might need to increase that by using the maxtime
option. Here is the link to the manual entry for append.HTH