I have very large number (over hundred) of scheduled searches done every minute. Some have alert actions to send an email.
I get thousands of events like this:
WARN SavedSplunker - AlertNotifier busy! Failed to enqueue job for search_id="scheduler_(...)". No actions will be executed. Consider improving action execution speed or increase action_execution_threads in limits.conf
some thousands per day.
I raised this limit to 6 (and to 10 after) - and now I get about 0-100 per day.
How to cope with that? Documentation says, that 10 is the maximum. I want to disable this limit at all.
There is no way to make it unlimited. Good practice is to use more search heads with SHC to distribute alerts and increase actions_queue_size (500 or so? ) Any unlimited settings need to be careful. It could use up all available resources.
There are 4 searchheads in cluster now.
Thank you for this setting - I will try it.