Our shop has four indexers with limited storage. This is due to the fact that we wanted fast disk for quicker searching of the most recent data. All servers are RHEL 5.10 x64 running Splunk 6.0. I am planning on upgrading to 6.1.2 soon. Recently I noticed that we are getting the alert stating that there is only 5 GB of disk space left and indexing has been paused. This is happening on all four indexers from time to time. I have a volume configured on the indexers that when it reaches a max size to roll the warm buckets to cold (network storage). This has worked well for about 2 1/2 years until recently. I am guessing that there are other files that are outside of this volume cap that are not getting cleaned up.
I did a search for large files/directories and found the /searchpeers directory with bundles from all of the searchheads. Some of them seem somewhat old.
So enough of the back story. Here are my questions:
Any assistance with this issue would be greatly appreciated.
Thanks
If you are talking of the splunk logs, not the indexes.
The splunk logs are in $SPLUNK_HOME/var/log/splunk
This folder is also the location of the crashed and coredumps, and have to manually deleted the cores.
The splunk logs are controled by $SPLUNK_HOME/etc/log.cfg, and keep 5 copies of 25 MB each.
If you are talking of the splunk logs, not the indexes.
The splunk logs are in $SPLUNK_HOME/var/log/splunk
This folder is also the location of the crashed and coredumps, and have to manually deleted the cores.
The splunk logs are controled by $SPLUNK_HOME/etc/log.cfg, and keep 5 copies of 25 MB each.
I had to run this as a cron job every hour in order to rotate the log files before Splunk. Hopefully I will not have to increase the frequency.
@yannK, I made a logrotate config file to be run against all of the .log files in /opt/splunk/var/log/splunk. Should roll when a file is over 24M and only rotate 1 and compress. Went from 1.2GB to about 120MB. Nice disk size savings. I chose 24M so that my logrotate job would run against logs before the Splunk log rotate job. Do you know how often and at what times Splunk runs it's logrotate job?
So what you are saying is that I need to create my own logrotate config and drop it in the logrotate.d directory. I don't want anymore than 1 log file rolled. 5 is way too many and diskspace is an issue. I would rather indexed data fill that space instead of logs. Doesn't the SOS app pull in and index those logs anyway?
1 - no , dc only write in $SPLUNK_HOME/etc/apps not in $SPLUNK_HOME/etc/
2 - yes, the file is contained in the installer
3 - no, it seems to be per log file
@yannK a few followup questions:
is there a way to push this out with the deployment-server?
will an update of splunk erase changes made to this file?
is there a way to set a global setting for any log file and not have to update settings on each log file?
Thanks
Yep. I cleaned up a bunch of logs in $SPLUNK_HOME/var/log/splunk. I just needed to know where the log rotate config was that Splunk used to clean up the logs. Thanks!
I updated the title to reflect the new issue. I was able to reclaim a good amount of drive space by removing many of the redundant log files that have been rolled. I tried to look for a Splunk logrotate config file in /etc/logrotate.d/ but there is not one.
Does anyone know where the Splunk logrotate config file is located? I would like to update it to only roll a log file one, and not five times.
Thanks
Bump. This issue is still happening. I would really appreciate any thoughts.
Thanks
You maybe limited by Splunk's latest update about freespace being 5GB
http://docs.splunk.com/Documentation/Splunk/6.1.2/Installation/Systemrequirements#Recommended_hardwa...
Dave
Thanks Dave. I believe the 5 GB limit has been a requirement for a few major releases now.
Regardless, I need to find what files/directories are growing and are not being cleaned up.
Thanks