index still growing above maxTotalDataSizeMB

Question

We are running out of disk space on one of our indexers on our SAN disk. We adjusted the maxTotalDataSizeMB and restarted but the index is still growing (we expected it to shrink to below maxTotalDataSizeMB). It has been up for about an hour.

10:00 1,376,072
10:15 1,376,137
10:30 1,376,416
10:45 1,377,537
11:00 1,378,630

main    
1,000,000 = max size
1,378,554 = current size
72,972,419,131
Jun 24, 1999 4:16:40 AM
Nov 30, 2010 11:59:00 P


[main]
homePath = /local/splunk/defaultdb/db
coldPath = /san/splunk/defaultdb/colddb
thawedPath = /san/splunk/defaultdb/thaweddb
maxTotalDataSizeMB = 1000000
disabled = false

Answer 1

2

Please log a support case if this has not been already done. Support would need to review the configuration settings for that indexer. In the meanwhile, the following link is a good reference

http://www.splunk.com/wiki/Deploy:BucketRotationAndRetention

link

answered 29 Nov '10, 16:41

Ellen ♦
2.3k●4●3●32
accept rate: 72%

Support case was logged and related to the existence of inflight-db buckets.
Inflight-db's are not counted towards the value for maxTotalDatasize of the index. With Splunk down, if inflight-db buckets remain, they are considered stale and can be deleted. Once those were removed, in the UI > Manager > Indexes > Current Size was reduced.

An enhancement has been requested for Splunk to look for the existence of these inflight-db buckets at start up and automatically delete these or identify (eg. log) for removal.

(07 Dec '10, 17:38) Ellen ♦

Answer 2

1

There are other indexes besides the default index - are you sure the current size is just of the default index or of the entire data store?

To verify:

du -ch /san/splunk/defaultdb/colddb
du -ch /local/splunk/defaultdb/db
du -ch /san/splunk/defauldb/thaweddb

Keep in mind that Splunk will never age anything out of the thawedPath.

Also, I don't see any indication that you are controlling the number of warm buckets. By default, Splunk will keep 300 warm buckets in the homePath.

Do you see any buckets with the suffix "inflight" attached to them?

link

answered 29 Nov '10, 16:39

araitz ♦
8.0k●3●10●25
accept rate: 45%

yes, there are other indexes besides default, this is just the one that we are trying to reduce the size of. Current size is received from GUI, so it is of default.

There is data in thaweddb. 182G /defaultdb/thaweddb/ 946G /defaultdb/colddb/ 215G /defaultdb/db/

The more indexs.conf maxTotalDataSizeMB = 2000000 frozenTimePeriodInSecs = 7776000 maxDataSize = auto_high_volume maxWarmDBCount = 20 maxHotBuckets=10

Yes, there are directories with inflight. 3.6G /san/splunk/defaultdb/colddb/inflight-db_1285372740_1285156778_3940/rawdata

(29 Nov '10, 16:53) tawollen

index still growing above maxTotalDataSizeMB

Follow this question

Splunk Resources

Related questions