When submitting queries in rapid succession to Splunk (via the REST API), I'm getting 503 errors from splunkd. This seems to occur after exactly 3 rapid-fire requests in most cases, although there have been times when I get much further than this before seeing the error.
The particular queries being executed don't seem to matter.
In etc/system/local/limits.conf I have:
Splunk appears to be ignoring limits.conf
You might hit the limits for individual role search quota(authorize.conf) before you hit the system search quota limitation(limits.conf). authorize.conf determines the limitation for concurrent search for roles.
The default quota for normal role is 3.
Based on your diag, there is no change in your $SPLUNK_HOME/etc/system/local/authorize.conf, so Splunk will use default srchJobsQuota = 3.
Please modify your $SPLUNK_HOME/etc/system/local/authorize.conf to increase your quota.
answered 08 Jun '10, 18:58