Licensing warning issued within 24 hour. Daily indexing volume limit exceeded.

[jalen1331]

I had an issue with storage.  I was at another site for 2 weeks and we reached the max limit on our drive.  I had to reprovision in VMware and while it was out of storage we had issues, I can't remember the error message but it was related to storage.  Fixed the storage issue and rebooted and had to reset my certificate and everything looked fine.  A day later we started getting the license issue.  I read the articles in the community.  I didn't fully understand.  I think its polling the environment for the time that my storage limits were reached? It's been 4 days with us being over the licensing limit.  Looking back over the last year, we have never been close to our limits. Any help would be appreciated. 

[_JP]

My guess at what is driving these overages is something didn't get indexed during that time that you had your outage scenario.  For example, if an Indexer runs out of disk space, that will bubble out to your edge tiers.  So a Forwarder might pause reading a log file, for example.  When things come back on-line, everything tries to catch up, but you end up in a scenario that in one day you might index three days of data.

To see if this is what was happening to you, you can do a quick check of what your license usage was over time.  If it was really low during your outage, then pegged once you were back up, then this scenario is the most likely. 

Another way to look for this scenario is to search your indexes and compare the values of _time and _indextime.  The field _time is the timestamp of when the Event occured (often times the timestamp within the event data).  The field _indextime is the timestamp of when Splunk indexed the data.  Under normal conditions the variance between these fields should be small.

You could run a quick search that calculates the avg diff of _time and _indextime by hour over the time of your outage (and include some data before and after the actual outage to get a sense of your boundaries).  If you see a large avg difference during your outage period this also would tell you that Splunk was "catching up" and that's what caused your overages.

[isoutamo]

Hi

basically this means that your daily indexing amount is greater than you license allow. What will happen is totally dependent on your splunk version and size of your license.

With free version after you have gotten 5 violation in 30days you cannot do searches until there is 30d period when there are less than 5 violations. 

Enterprise with license less than 100GB in recent versions that limits is 45/60days. If you get more violations your searches will blocked, but you can ask reset license from your Splunk account manager.

If your license is 100GB+ then you will get those warnings, but your search are still working.

Some older version with paid license that was 5/30d and then you could ask reset license unless you have "non blocking" license. With it it act like current 100GB+ license.

r. Ismo