Hi,
My main index is stored in two locations depending on whether its a cold bucket or a hot/warm bucket. I set aside 400 GB of fast storage for the hot/warm buckets so with a maxDataSize of 10GB (auto_high_volume) plus 10 hot buckets I should set maxWarmDBCount to 30 right?
(30 warm buckets * 10GB) + (10 hot buckets * 10GB) = 400GB
The problem is that hot buckets are not reaching their 10GB limit so I end up having 30 warm buckets varying in size and consequently the 400GB are really never completely used up and is only half full with many 5GB buckets.
How can I make sure that buckets don't get rolled over until they reach the MaxDataSize limit?
Thanks!
I found that additional info regarding the rolling over of warm and hot buckets can be gleaned from splunkd.log after turning on debug mode.
Take a look maybe I can help you to verify your indexes.conf
http://wiki.splunk.com/Deploy:BucketRotationAndRetention
Hi,
The best approach is to specify a size for warm, and have a max bucket count in excess of what you would expect, so that the volume limit takes effect.
You can do this in two ways:
Set the homePath.maxDataSizeMB for the index
homePath.maxDataSizeMB =
* Limits the size of the hot/warm DB to the maximum specified size, in MB.
* If this size is exceeded, Splunk will move buckets with the oldest value of latest time (for a given bucket)
into the cold DB until the DB is below the maximum size.
* If this attribute is missing or set to 0, Splunk will not constrain size of the hot/warm DB.
* Defaults to 0.
Set up a volume, and use that for hot.
# volume definitions; prefixed with "volume:"
[volume:hot1]
path = /mnt/fast_disk
maxVolumeDataSizeMB = 100000
[volume:cold1]
path = /mnt/big_disk
# maxVolumeDataSizeMB not specified: no data size limitation on top of the existing ones
[volume:cold2]
path = /mnt/big_disk2
maxVolumeDataSizeMB = 1000000
# index definitions
[idx1]
homePath = volume:hot1/idx1
coldPath = volume:cold1/idx1
Buckets are not guaranteed to reach their maximum size; they can roll over 'early' for a number of reasons. I'd slightly prefer the second option, as it's easier if you have multiple indexes.
Duncan
for example a server reboot (indexer)
Yeah, its what I ended up doing. Do you know what reasons are the ones that make a bucket roll over early?
Thanks!
This may not apply to you at all, and perhaps you know it already, but all hot buckets roll to warm when Splunk is restarted. /k