Hi all. I am preparing for a production install of Splunk 6.1.1 (new install, 2 IDX, 2 SH w/ SHP, DS/License server VM, and a rSyslog VM with 105GB daily license) and am having an issue in a lab environment, that I have a less than desirable workaround for. I have opened a case with Splunk, but wanted to share my issue here as well to see if anyone has seen this before.
SUBJECT:
Running Splunk as non-root (RHEL 6.x) is not working properly
DESCRIPTION:
I am building out a dev environment of Splunk 6.1.1 to ensure my process is good prior to building out our production Splunk 6.1.1 environment. I am using the Splunk 64-bit Redhat RPM build, and when I tell Splunk to start as the 'splunk' user, it fails to start because it doesn't have permission to write to '$SPLUNK_HOME/var/logs/splunk/first_install.log'. first_install.log is create with 'root' as the owner, so 'splunk' does not have access to write to the file. If I chown that log file to 'splunk:splunk', splunk will now enable boot-start. However, upon rebooting the server, Splunk does not start. If I try to manually start Splunk using '/etc/init.d/splunk start', I get a similar error where permissions are denied when trying to write to splunkd-utility.log. Again, if I 'chown -R splunk;splunk /apps/splunk', then Splunk will successfully start.
Now it's my understanding that in Splunk 6.1, Splunk was changed to where it will start as root, and switch to a named account ('splunk', non-privileged), but it seems these log files are being created with root as owner. This is stopping me from successfully installing Splunk per the install guide, without running chown twice and I fear that this will cause other issues later on.
This issue is holding up our production deployment of Splunk.
STEPS REPRODUCE:
http://pastebin.com/pjijUpyb
In the pastebin you will see the full commands for my install, and my workaround. I am using a named account called 'splunker', which has sudoer rights. In my environment, once a *NIX based system goes production, only the *NIX admins have access to the root account so I have to use a named account with sudoer access.
Anyone run into this, or am I just being over-concerned about this?
... View more