Issues Triggering Alert Scripts on Windows XP

This can be a complex problem and it's important to be thorough in checking that every step of the process (from scheduled search to alert script) is working as expected :

A) Is my scheduled search running?

• Check %SPLUNK_HOME%\var\log\splunk\scheduler.log or search for "index=_internal source=*scheduler.log savedsearch_name="my_saved_search_name" | stats count by status" to determine if your scheduled search is running and with a status of "success"? If you see failures here, drill down into those to see why the search is not running. Is it taking too long to execute? Are there too many concurrent searches running at that time?

B) Is my scheduled search generating the expected results?

• Again, check %SPLUNK_HOME%\var\log\splunk\scheduler.log or search for "index=_internal source=*scheduler.log savedsearch_name="my_saved_search_name" | stats count by result_count". Is the result event count as you would expect it?

• Provided you have configured Splunk to be able to send emails, add an email action to send yourself an email . Check the search results in the URL provided by the email, and see if they are as expected.

C) Is my alert action being triggered?

• Set up an additional alert action (typically an email) to see if alert actions are being triggered or not.

• Make sure that you don't have an issue with the condition of your alert. To verify this, change the alert condition to "always". If your script runs then, you know the problem is with the condition and you should study the results of your scheduled search to see why it isn't triggering the alert action as expected.

D) Is my alert script working?

• First, make sure that your script sits where it should : %SPLUNK_HOME%\bin\scripts\ is a good location, but you may also want to put it in %SPLUNK_HOME%\etc\apps\your_app_name\bin\ if your scheduled search is app-specific and not global.

• Let's check that the script itself runs outside of Splunk. As the user that splunkd runs as on your system, launch the script. Is it producing the expected output? If the script is dependent on variables passed by the Splunk scheduled search, you may want to temporarily set those to hard values in the script itself.

• Use the "runshellscript " command to run your script manually from the search bar. Does this work?

• If your own script is running manually and from the search bar but not when called as an alert action, it's time to check if Splunk is able to run a simple script in that manner. Splunk ships with a simple script located in %SPLUNK_HOME%\bin\scripts called echo.bat (or echo.sh for *nix systems). This script does a very simple thing when called : It attempts to write the 8 arguments passed by the Splunk scheduled search to the script (see http://www.splunk.com/base/Documentation/latest/Admin/Configurescriptedalerts#Script_options) to a text control file.

  This is a good script to test the basic functionality of triggering an alert script, and seeing how Splunk is passing variables to it. Note that you may want to change the script slightly, for example to write it's output to a different location or to add a time stamp to the output.

  I typically modify it in the following way in order to time stamp the output and also to specify an absolute path to a custom directory :

  @echo off
  echo %0, %1, %2, %3, %4, %5, %6, %7, %8 >> "c:\temp\echo_output.txt"
  date /T >> "c:\temp\echo_output.txt"
  time /t >> "c:\temp\echo_output.txt"
  echo ---------------------------------------- >> "c:\temp\echo_output.txt"

  To use this script, first run it manually as the user that runs splunkd and make sure that you get the expected result : A file called "echo_output.txt" should be created in %SPLUNK_HOME%\bin\scripts. If you have passed arguments to the script from the command line, they should be listed in the last line that was written to that file.

  Next, change your scheduled search to use echo.(bat|sh) instead of your own script, and observe the control file (echo_output.txt or any other file you may have pointed to script to output to).

  If none of the above steps allow you to solve your problem, it's probably time to contact Splunk Support!