How do I reclaim disk space after a massive delete operation?

It is possible to reclaim disk space in this type of scenario by re-indexing the effected buckets.

Note: This may also be useful if you've deleted some sensitive information, such as a password, that really needs to be completely purged. This approach would prevent that indexed term from showing up in type-a-head, for example.

There are several steps to this process.

1. Identify all buckets for each index that were effected by your deletion. (This alone can be a complicated task, also keep in mind that the delete command forces a bucket roll for hot buckets.)
2. For each bucket, do the following:
   1. Export the bucket data to a .csv file
   2. Import the .csv file into a new empty bucket (with a temporary name/location)
   3. Optimize the new bucket.
   4. Replace the original bucket with the newly created bucket.

For users running on a unix platform, the following shell commands (script) may be of use: (Note that we are combining the export and import step into a single operation using a pipe)

#!/bin/bash
BUCKET=$1

# Be sure to compare the imported/exported event count.  They should be the same.
exporttool ${BUCKET} /dev/stdout -csv meta::all | importtool ${BUCKET}.new /dev/stdin

# Make sure that bucket .tsidx files are optimized (and merged_lexicon.lex is up to date)
splunk-optimize ${BUCKET}.new
splunk-optimize-lex ${BUCKET}.new

# Compress all rawdata files that were not gziped by importtool
find ${BUCKET}.new/rawdata -name '[0-9]*[0-9]' -size +1k -print0 | xargs -0 -r gzip -v9

# Swap buckets
mv ${BUCKET} ${BUCKET}.old
mv ${BUCKET}.new ${BUCKET}

# Uncomment next line if you really want to remove the original bucket automatically
# rm -rf ${BUCKET}.old

Note: If you plan on using this script, please be sure to add return-code checking. You wouldn't want to remove the original bucket if the export/import failed to complete, for example.

Other considerations:

• Keep in mind that importtool does not respect your segmentation settings. The default segmentation is used for all imported events. For many setups, this will not matter, but it is something to be aware of.
• It's possible to loose data using this approach. This is a use-at-your-own-risk kind of operation. It's possible that you may not even reclaim all that much disk space using this approach.