Refine your search:

Good morning,

I am suddenly receiving this error and not able to index:

skipped indexing of internal audit event will keep dropping events until indexer congestion is remedied. Check disk space and other issues that may cause indexer to block

The other day I received this error:

Applying indexing throttle for defaultdb\db because bucket has too many tsidx files, is your splunk-optimize working?

I have recently upgraded from 4.1.3 to 4.1.5. There was no immediate change but I did start using FSChange to monitor some directories.

I removed FSChange stanzas that I added from the inputs.conf and restarted and I am still having the issue, though the warning moved back to the second error.

In splunkd.log I see:

10-15-2010 08:20:18.131 ERROR DispatchCommand - Failed to start the search process.  
10-15-2010 08:20:18.162 WARN  DispatchCommand - The system is approaching the maximum number of historical searches that can be run concurrently. current=7 maximum=8  
10-15-2010 08:20:18.193 ERROR DispatchCommand - Failed to start the search process.  
10-15-2010 08:20:19.850 ERROR DispatchCommand - The maximum number of historical concurrent system-wide searches has been reached. current=8 maximum=8 Search not executed! SearchId=scheduler__nobody__windows_d2luX2V2ZW50bG9nX2NvdW50X3N1bV9pbmRleA_at_1287148800_967218771  
10-15-2010 08:20:19.896 ERROR SearchScheduler - The maximum number of historical concurrent system-wide searches has been reached. current=8 maximum=8 Search not executed!   SearchId=scheduler__nobody__windows_d2luX2V2ZW50bG9nX2NvdW50X3N1bV9pbmRleA_at_1287148800_967218771  
10-15-2010 08:20:23.193 WARN  timeinvertedIndex - splunk-optimize failed to start for index D:\Splunk_Data\var\defaultdb\db\hot_v1_16 : The session was canceled.  
10-15-2010 08:20:23.193 WARN  timeinvertedIndex - splunk-optimize failed to start for index D:\Splunk_Data\var\defaultdb\db\hot_v1_19 : The session was canceled.  

I am not sure if it is related. Perhaps with all my alerts that run at various intervals (10 min, 15 min, 20 min, 30 min) I am eclipsing 8. Would that cause the errors regarding not indexing?

I am currently not able to view any data for the last two days.

Thanks for any help!

Kevin

asked 15 Oct '10, 13:30

kholleran's gravatar image

kholleran
263120
accept rate: 33%

edited 15 Oct '10, 13:38

Lowell's gravatar image

Lowell ♦
9.6k637

What did it end up being?

(20 Oct '11, 09:12) chicodeme

One Answer:

For a "down" kind of scenario like this, it may be best to contact splunk support. Email them with a link to this page, run the "splunk diag" utility, upload the diag file to your case, then call the splunk support phone # to get in contact with someone quickly.


Things I would check:

  1. Disk space on all of your partitions. If your space is too low, then that will cause indexing and searching problems.
  2. Verify that "splunk-optimize" has been running. If you see a large number of *.tsidx files in your buckets, you can simply run splunk-optimize /path/to/your/bucket to force this process to run.
  3. Try disabling some non-essential scheduled saved searches and see if that helps relieve the problem. (You probably don't want to do this for summary-indexing saved searches, if it can be avoided.)
link

answered 15 Oct '10, 13:42

Lowell's gravatar image

Lowell ♦
9.6k637
accept rate: 40%

edited 15 Oct '10, 13:47

I have an open case and am working with them. Disk space is fine, I disabled a scheduled saved search, but just the one I added prior to this problem occurring.

I am looking into #2. Thanks very much for all your help.

(15 Oct '10, 15:50) kholleran

Unfortunately, those did not work and I have not heard back in a couple days regarding my case with Splunk. Any other thoughts as I am completely down. If this goes on much longer I may have to downgrade back to 4.1.3, even if that means ripping out and re-setting up the installation. Thanks.

(19 Oct '10, 13:36) kholleran
1

I suggest calling splunk support. Copied from the "Contact Us" web page: If you have purchased Enterprise Support, please call the Enterprise Support line at +1 415.848.8400 option 3.

(20 Oct '10, 14:44) Lowell ♦
Post your answer
toggle preview

Follow this question

Log In to enable email subscriptions

RSS:

Answers

Answers + Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "Title")
  • image?![alt text](/path/img.jpg "Title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Tags:

×1,090
×328
×300
×104

Asked: 15 Oct '10, 13:30

Seen: 1,754 times

Last updated: 20 Oct '11, 09:12

Copyright © 2005-2012 Splunk, Inc. All rights reserved.