You don't need to stop Splunk to backup the configs.
As far as the configs go, create a git repo in $SPLUNK_BACKUP (off box, hopefully) then add this cron job however often you want the configs backed up (hourly, daily, decade-ly)
rsync -vaz $SPLUNK_HOME/etc/ $SPLUNK_BACKUP/;
cd $SPLUNK_BACKUP;
git commit -a -m "Configs as of $(date)";
Then you can revert back to whatever version of your configs you want by looking through git log for the date you want to revert to, running git checkout $COMMIT , copying it to your indexer(s), and bouncing splunkd
... View more