How to backup/restore Splunk db to new system

Just to add detail to these steps & after talking to Zach who explained they are doing a hardware switchover. Looks like they may not have the option to run concurrent systems with duplicate data, Although it is the ideal scenario especially for that much data!

If you must do a hard/switchover migration follow these steps
1) ensure the indexes are all created on your new server so the db dir structure is in place.
2) Backup your warm / cold buckets, and copy / move to new data store location (IMPORTANT do NOT backup your hot buckets)
warm buckets
$SPLUNK_HOME/var/lib/splunk/< indexname >/db/db_NN_NN_NN dirs
cold buckets
$SPLUNK_HOME/var/lib/splunk/< indexname >/coldb/db_NN_NN_N dirs
3) perform a splunk roll to warm command to get your hot buckets, rolled to warm. I would recommend running it on the CLI so you know when it's complete. From the CLI:

For all indexes
./splunk search "| debug cmd=roll"

or individual indexes
./splunk search "| debug cmd=roll index= < indexname > "

4) Immediately stop your production splunk so no new data is coming in, Although you will lose some data from the time the roll to warm completes and new data coming in to your hot buckets.
5) There will be new warm buckets created, from the roll to warm command backup these new warm buckets, and copy / move to new data store location warm buckets
$SPLUNK_HOME/var/lib/splunk/< indexname >/db/db_NN_NN_NN dirs

6) start up splunk and search on your old data