How do I set up the Splunk on Splunk app to monitor Splunk's system resource consumption?

As of Splunk on Splunk 2.0, SoS ships with the ps_sos.sh scripted input which once enabled, allows you to track the CPU and memory usage of the main Splunk components :

• Splunkweb, the cherrypy-based web front-end.
• splunkd, the back-end daemon which manages the indexing of data.
• The searches fired by splunkd to retrieve events and build reports.

Note: Currently, this scripted input only exists on Linux and Unix. A similar functionality for Windows will be added in a future SoS release.

Also: You do not need to install the SoS technology add-on on an instance where SoS is already installed.

The ps_sos.sh data input ships both with the SoS app and with the SoS technology add-on for Unix and Linux.

1) On a search-head or stand-alone indexer:

• Install the SoS app.
• Enable the ps_sos.sh scripted input by one of the following methods:

a) Go to Manager > Data Inputs > Scripts. Enable the ps_sos.sh data input.

or

b) Run the following command from a terminal window:
$SPLUNK_HOME/bin/splunk _internal call '/servicesNS/nobody/sos/data/inputs/script/.%252Fbin%252Fps_sos.sh' -post:disabled 0

2) On search peers:

• Install the SoS technology add-on for Unix and Linux.
• Enable the ps_sos.sh scripted input by one of the following methods:

a) If Splunkweb is running, go to Manager > Data Inputs > Scripts. Enable the ps_sos.sh data input.

or

b) Run the following command from a terminal window:
$SPLUNK_HOME/bin/splunk _internal call '/servicesNS/nobody/TA-sos/data/inputs/script/.%252Fbin%252Fps_sos.sh' -post:disabled 0

Going forward, you will be able to track the CPU and memory usage of Splunk in the "Splunk CPU/Memory Usage" SoS view. Per-search memory usage for the biggest memory-consuming searches can be consulterd in the "Distributed Searches Memory Usage" view.

For further information on deployment best practices for SoS, please refer to this Splunk Answer.