How can I monitor the resource usage of my forwarder using the S.o.S app?

/!\ THE FOLLOWING INSTRUCTIONS ARE ONLY VALID FOR SOS 2.2 AND NEWER /!\

In SoS 2.2, the "Server to query" pulldown present in most views and that lists instances reachable by distributed search will load from a lookup file "splunk_servers_cache.csv", refreshed at 5-minute intervals.

This will considerably speed up the population of this omnipresent pulldown. It also means that you'll be able to scope S.o.S' views to events in the _internal index (or in other indexes, by updating the appropriate macros such as set_internal_index or set_sos_index) accessible by distributed search and belonging to any host, without the need for that host to be a search peer.

In order to do this, you will have to manually append an entry to the "splunk_servers_cache.csv" lookup. The fields required for an entry are:

• sos_server : This should be set to the value of the "host" field with which the events for the particular server are tagged.
• server_role : A description of the role of the server. Typically, this should be "search-head", "search-peer", "standalone indexer" or "forwarder".

Step by step instructions:

1 - Install the SoS technology add-on for Unix and Linux on the forwarder whose resource usage you care to monitor.
2 - Enable the ps_sos.sh scripted input by one of the following methods:

• Edit the `$SPLUNK_HOME/etc/apps/TA-sos/local/inputs.conf with the following content:

[script://./bin/ps_sos.sh]
disabled = 0

or

• Run the following command from a terminal window:

$SPLUNK_HOME/bin/splunk _internal call '/servicesNS/nobody/TA-sos/data/inputs/script/.%252Fbin%252Fps_sos.sh' -post:disabled 0

3 - Verify that the forwarder is now sending events generated by the ps_sos.sh scripted input to the indexer(s) by searching for:

index=sos sourcetype=ps | stats count by host

The hostname of your forwarder should be among the values returned.

4 - Append a new line to the $SPLUNK_HOME/etc/apps/sos/lookups/splunk_servers_cache.csv table composed of two, comma-separated values:

• The value of the "host" field for your forwarder's events in the sos index, as you gathered it in step #3.
• The string "forwarder", which will label that host as such in the "Server to query" pulldown.

Here's an example of the kind of line you would append to splunk_servers_cache.csv:
"forwarder.example.com","forwarder"

5 - The "Server to query" pulldown in the "Splunk CPU/Memory Resource Usage" view should now list your forwarder and enable you to view its system resource usage.