My forehead is sore from banging it on my desk. Please help.
I cannot get scripts to run from an alert. The following is all the relevant info I have been using to trouble shoot this.
To make it easy, I configured the alert to run "always" so the search term is irrelevant. The alert should be triggered every minute regardless. (I have also ran this with a search that I was manually triggering events in)
The powershell script
#!C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe
"hello world" | Out-File -FilePath .\hello_ps.txt
The savedsearches.conf
stanzas
[script_test_ps]
action.script = 1
action.script.filename = test.ps1
alert.digest_mode = True
alert.suppress = 0
alert.track = 0
auto_summarize.dispatch.earliest_time = -1d@h
cron_schedule = * * * * *
enableSched = 1
search = sourcetype=nothing
We know the search is running, we can see the search run and we can watch the script being called but it never actually executes.
From python.log
2014-10-02 18:19:03,171 Central Daylight Time INFO runshellscript:188 - runshellscript: ['C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe', 'D:\\Splunk\\bin\\scripts\\test.ps1', '0', 'index=testing sourcetype=script_test', 'index=testing sourcetype=script_test', 'script_test_ps', 'Saved Search [script_test_ps] always(0)', 'https://splunkweb.domain.tld/app/search/@go?sid=scheduler_bmVpbC5wZXRlcnNvbg__search__RMD59d824a49b6b738b2_at_1412291940_27935', '', 'D:\\Splunk\\var\\run\\splunk\\dispatch\\scheduler_bmVpbC5wZXRlcnNvbg__search__RMD59d824a49b6b738b2_at_1412291940_27935\\results.csv.gz']
From scheduler.log
10-02-2014 18:19:03.889 -0500 INFO SavedSplunker - savedsearch_id="neil.peterson;search;script_test_ps", user="neil.peterson", app="search", savedsearch_name="script_test_ps", status=success, digest_mode=1, scheduled_time=1412291940, dispatch_time=1412291942, run_time=0.562, result_count=0, alert_actions="script", sid="scheduler_bmVpbC5wZXRlcnNvbg__search__RMD59d824a49b6b738b2_at_1412291940_27935", suppressed=0, thread_id="AlertNotifierWorker-1"
Powershell execution policy is unrestricted
PS D:\Splunk\bin\scripts> Get-ExecutionPolicy
Unrestricted
Splunkd is running as a service account
PS D:\Splunk\bin\scripts> Get-WmiObject win32_service | Where-Object {$_.name -like "splunk*"} | Select-Object name, startname
name startname
---- ---------
Splunkd domain\svc.splunk
splunkweb domain\svc.splunk
The service account is in the Administrators groups
The troubleshooting steps from this wiki article: http://wiki.splunk.com/Community:TroubleshootingAlertScripts
Is my scheduled search running? YES. I see it in scheduler.log
Is my scheduled search generating the expected results? YES. I can watch the results come in on a real time search. I have also scheduled the alert to run "always" to make the search part of it irrelevant.
Is my alert action being triggered? YES. I have added email actions and I get those emails, as well as watching it in https://splunkweb.domain.tld/en-US/app/launcher/job_management?savedSearch=script_test
Is my alert script working? YES. I can run it from the command line, as the svc.splunk user, but when the alert triggers it, nothing happens.
I have been using the following documentation. I have read it and reread it.
http://docs.splunk.com/Documentation/Splunk/6.1.3/alert/ConfiguringScriptedAlerts
http://docs.splunk.com/Documentation/Splunk/6.1.3/Alert/Setupalertactions#Run_a_script
http://docs.splunk.com/Documentation/Splunk/6.1.3/Admin/Savedsearchesconf
http://wiki.splunk.com/Community:TroubleshootingAlertScripts
I have been testing and troubleshooting this from lots of different angles. I tried to break everything down to the simplest example, but my copy pasta above may still have gotten mixed up. If so please point it out and I will verify my testing results.
EDIT: For posterity sake, I will explain the (very very very elementary and obvious) solution here. The script was writing out to $splunk_home\bin
not $splunk_home\bin\scripts
where the script was located. If you browse to the scripts location and run it, it obviously works, but when Splunk runs it, those relative paths do not exist from bin/
The other quirk I have noticed is that when Splunkd is running as LocalSystem, it can execute batch scripts but not powershell. When Splunk is running as a service account the powershell scripts work fine.
To wrap your ps1 up into a bat you should put the folllowing in the batch script.
C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -File .\location\ofyour\script.ps1
The above will look for $splunk_home\bin\location\ofyour\script.ps1
which is what we want.
where do you expect the file to be created? give the absolute file path rather than the one you have mentioned. I have not tries to give a ps1 file in the script trigger path ever. So could you try with a bat file? just mention complete path to script and output file everywhere. Make the cron schedule to */1 * * * *
. It's quite easy to test with the below in a .bat file
%SPLUNK_ARG_5% > e:\output.txt
Hey @neiljpeterson, if I'm understanding you correctly, the final solution uses a .bat file in action.script.filename, is that right? I just pinged docs team to add this to the official scripted alert docs, and wanted to make sure I had that detail right. Or were you only mentioning .bat because of the issues when running PowerShell as localsystem?
where do you expect the file to be created? give the absolute file path rather than the one you have mentioned. I have not tries to give a ps1 file in the script trigger path ever. So could you try with a bat file? just mention complete path to script and output file everywhere. Make the cron schedule to */1 * * * *
. It's quite easy to test with the below in a .bat file
%SPLUNK_ARG_5% > e:\output.txt
:hangs head in shame: The script was actually writing to $splunk_home\bin
, not to $splunk_home\bin\scripts
where the script was actually located. I found a bunch of text files from my testing in bin 😕
I had tested a hard path earlier, but I think I had other unresolved issues as well at that point so I errantly dismissed that as a solution.
Failure in basic troubleshooting I'm afraid 😐 I will blame it on a lack of sleep.
Thank you for your help! If you post it has an answer I can accept it.
Converted to an answer for ya.