Deployment client fails to restart splunkd - manual restart needed

Question

After deploying an updated app to a deployment client, instead of the client restarting splunkd (as requested), it shuts down compleetly. As a result, I have to connect to the boxes and manually restart the splunkd service. For a few months I only had this problem with a single forwarder (I tried a few upgrades, but that made no difference). Now more recently several additional servers are having this problem too.

Here are the splunkd.log message after a sample deployment:

01-06-2011 17:02:50.896 INFO  DeployedApplication - Checksum mismatch 1889753244061277121 <> 15808103211093779078 for app: co_windows_extras. It will be reloaded again from: splunk.co.com:8089/services/streams/deployment?name=default:windows-server:co_windows_extras  
01-06-2011 17:02:50.927 INFO  DeployedApplication - Remote repository has resolved to: splunk.co.com:8089/services/streams/deployment?name=default:windows-server:co_windows_extras
01-06-2011 17:02:50.974 INFO  DeployedApplication - Downloaded url: splunk.co.com:8089/services/streams/deployment?name=default:windows-server:co_windows_extras to file: H:\splunk\var\run\windows-server\co_windows_extras-1294350540.bundle
01-06-2011 17:02:51.396 WARN  DeployedApplication - Installing app: co_windows_extras to location: H:\splunk\etc\apps\co_windows_extras
01-06-2011 17:02:53.114 WARN  DeploymentClient - Restarting Splunk Web...
01-06-2011 17:03:04.145 WARN  DeploymentClient - Couldn't execute  H:\splunk\bin\splunk restart splunkweb
01-06-2011 17:03:05.755 ERROR DeploymentClient - Couldn't ensure that index dirs were created.
01-06-2011 17:03:05.755 WARN  DeploymentClient - Restarting Splunkd...
...  (other non-related activity) ...
01-06-2011 17:09:05.753 ERROR DeploymentClient - forcing shutdown since it didnt complete in 360 seconds

I'm not sure why it's trying to restart Splunk Web on a deployment client, since I have this disabled. Any ideas on what's hanging this up?

Answer 1

I just tried running the Splunk Web restart command manually (H:\splunk\bin\splunk restart splunkweb), and I see the issue now.

This appears to be an upgrade of Splunk.

--------------------------------------------------------------------------------

Splunk has detected an older version of Splunk installed on this machine. To
finish upgrading to the new version, Splunk's installer will automatically
update and alter your current configuration files. Deprecated configuration
files will be renamed with a .deprecated extension.

...

Perform migration and upgrade without previewing configuration changes? [y/n]

So it looks like it's waiting for an answer its never going to get. And even it you pass in --answer-yes, then you simply get the message: An error occurred: In order to migrate, Splunkd must not be running.

Fun. I guess I'll be updating all of these manually.

I thought this was taken care of automatically with the Windows install. I'm used to dealing with this manually after each splunk upgrade on Linux.

Deployment client fails to restart splunkd - manual restart needed

Follow this question

Splunk Resources

Related questions