What should I do, if anything, when approaching the max number of searches?

If you have hardware to support allocation of more CPU resources to search execution, then it is possible to increase the max number of concurrent searches.

The number of concurrent searches is based on CPU and affected by 2 settings in limits.conf. From limits.conf.spec:

> [search]
> max_searches_per_cpu = <int>
> * the maximum number of concurrent searches per CPU. The system-wide number of searches
> * is computed as max_searches_per_cpu x number_of_cpus + 2
> * Defaults to 2

> [scheduler]
> max_searches_perc = <integer>
> * the maximum number of searches the scheduler can run, as a percentage
> * of the maximum number of concurrent searches, see [search] max_searches_per_cpu
> * for how to set the system wide maximum number of searches
> * Defaults to 25

For an 8 CPU box, the default number of max_searches_per_cpu is 18 (4x4cpu+2). It is likely, however, that such a server is capable of supporting more searches per CPU so it is safe to increment to max_searches_per_cpu=4.

Additionally, if you are running many scheduled searches for alerts or dashboards, you can find a more equitable division with max_searches_perc than the default 25%.

These settings will allow you to maximize your hardware. If you find this is not adequate, consider adding a server. In general, the recommended approach to scaling is to add more CPUs via additional Splunk servers so the workload of search execution can be shared.