How to aviod or figure out pool size in Splunk?

[prasadjvv522]

Hi All,

I am new to splunk, I have 14 node cluster in that splunk master and slaves.

Searching something in splaunk is getting error, Licence violation.

In licence violation having alerts like this

Sep 26, 2014 12:00:00 AM
(1 week ago) This pool has exceeded its configured poolsize=524288000 bytes. A warning has been recorded for all members splunk01.XXXXXXXXX.priv auto_generated_pool_free free license_window

This pool contains slave(s) with 18 warnings splunk01.XXXXXXXX.priv auto_generated_pool_free free pool_warning_count

Please help me how to avoid licence violation. and how to figure out the pool size (where to configured the pool size).

Note:- In the default DB having index size is like GB's ,this is the reason for licence violation.

Thanks,

Prasad

[jeremiahc4]

You can't change what that host has already logged in the past. You would only be able to change what it is logging going forward by modifying the inputs.conf on that host. You can determine which source on that host is consuming the most with a search like this (sub in the hostname of the offender);

index=_internal source=*license_usage.log h=yourhost | chart sum(b) as size by s | sort -size

[jrodman · Accepted solution]

The pool size is only configurable by allocating license quantity to pools. If you had multiple licenses, or wanted to allocate some amount of your license to one set of systems and another amount of your license to another set of systems, then this would make sense.

The documentation for this sort of thing is here:
http://docs.splunk.com/Documentation/Splunk/latest/Admin/HowSplunklicensingworks
http://docs.splunk.com/Documentation/Splunk/latest/Admin/Createalicensepool

However, the real issue here is you're trying to set up a 14-node cluster with a 500MB a day license. You're going to need a bigger license.

If you have gone over the license with a free license 3 times in 30 days, search will be locked out. You can clear this with splunk clean eventdata at the command line by dropping all your indexed data. With a paid (enterprise) license, the product will not lock out search unless you go over the license 5 times in 30 days.

[prasadjvv522]

Yeah jeremiahc,
Finally found that one host having more indexing ,size is 19547053198 (from the index search).

Please suggest best way to resolve this .

Thanks

[jeremiahc4]

unfortunately no, clearing eventdata will only remove your data, not your license violations. The answer below seems to suggest you can backup your data, reinstall, then restore your data to unlock your license;

http://answers.splunk.com/answers/68679/splunk-license-violation.html

[prasadjvv522]

Thanks jeremiahc ,

I will check the logs.and one more doubt is now it seems splunk index below 500MB.i think remove or clear the index data it will work or not(splunk clear eventdata/event ) ?

Please help me

[jeremiahc4]

you can search against your _internal index and find out which hosts (h) are using the most space using a search like this (searches are allowed against _internal even while searching is locked due to license violations).

index=_internal source=*license_usage.log | where isnotnull(h) | chart sum(b) as size by h | sort -size

[prasadjvv522]

Thanks Jeremiahc4 ,

But the data is important, please suggest other way means other than reinstall.

In licence usage report , it seems this week will use only below 500MB used ,previous two weeks more than 1GB for daily,
How to find which node having more indexing on last two weeks ?
Please help me.

Last week Index file:-
-rw------- 1 root root 1275742408 Sep 1 04:35 db_1410241271_1405830986_22/1409594401-1405830986-143948074789891008.tsidx
present :-
-rw------- 1 root root 58700 Apr 25 11:05 db_1389115156_1389115156_16/1389115156-1389115156-17861785314197610181.tsidx

[jeremiahc4]

I should add that you can see your license & pool settings by clicking on Settings in the upper right, then click on Licensing. There is a "Usage Report" button on the subsequent page that'll tell you how much you are using as well.

[jeremiahc4]

It looks like you are on the free license (524288000 bytes ~= 500 MB) and are exceeding 500 MB per day in data being sent to the index. The only way to avoid license problems is to reduce how much you are sending to your Splunk index. You would need to change some of your inputs (inputs.conf) to disable them or buy a larger license.

The free license has a maximum of 3 violations in a rolling 30 day window where you will be locked out until those violations roll off (>30 days). I'd heard that you can completely uninstall/reinstall to get around being locked out of searching. Seems a bit drastic, but if your data is critical enough to warrant that, you could try it.