I have a custom search command that extracts a domain name from a url string field you specify into a new "domain" field. This works fine on a dev cluster we have setup (3 search heads, 2 indexers). For example this returns expected results:
index=main
| table _time url
| mycustomcommand field_in=url
but adding stats command at the end of the search causes the search to fail with the following error:
index=main
| table _time url
| mycustomcommmand field_in=url
| stats count by domain
2 errors occurred while the search was executing. Therefore, search results might be incomplete. Hide errors.
[ip-{indexer_1_ip}] Streamed search execute failed because: Error in 'mycustomcommmand' command: External search command exited unexpectedly with non-zero error code 1..
[ip-{indexer_2_ip}] Streamed search execute failed because: Error in 'mycustomcommmand' command: External search command exited unexpectedly with non-zero error code 1..
Running the search directly on the indexer returns 0 results, because we don't have the url field extraction there. But there are no errors.
My questions are
Thank you for the help! This led me to figure out I was only looking at the logs for the search head. The search was streamed to indexers when using the stats command, and those indexer search.log files can be found through the job inspector under the Search Job Properties link.
The script errors found there (ImportError: No module named {mylib}) led me to this answer which explains the problem nicely: https://community.splunk.com/t5/Developing-for-Splunk-Enterprise/Custom-streaming-search-command-err...
Try appending | noop log_DEBUG=* to the search. Then check the search log for debug messages that may help determine the cause of the error.
Thank you for the help! This led me to figure out I was only looking at the logs for the search head. The search was streamed to indexers when using the stats command, and those indexer search.log files can be found through the job inspector under the Search Job Properties link.
The script errors found there (ImportError: No module named {mylib}) led me to this answer which explains the problem nicely: https://community.splunk.com/t5/Developing-for-Splunk-Enterprise/Custom-streaming-search-command-err...