Splunk HTTP Event Collector intermittently sending "Server is busy"

[hrawat_splunk]

Example

ERROR HttpInputDataHandler [7000
HttpDedicatedIoThread-1] - Failed processing http input, token
name=hec-token, channel=n/a, source_IP=xxx.xxx.xxx.xxx, reply=9,
events_processed=nnn, http_input_body_size=yyyyyyy, parsing_err="Server is
busy"

[hrawat_splunk · Accepted solution]

If HEC server is continuously sending 503 reply with "Server is busy", you can discard this reply.

If HEC server is intermittently sending 503 reply with "Server is busy", then first understand following fields.

events_processed=number of events successfully inserted into pipeline queue

http_input_body_size=http POST payload size sent by HEC client

HEC server knows the http POST payload size, but it does not know how many events the payload has.
So after inserting events_processed events into the pipeline queue, HEC server receiver thread finds that now the pipeline queue is blocked. It still has some unknown number of events un-processed. Waits for 1 sec for the queue to have space to insert next event. If still not enough space in the queue, then HEC server drops remaining events of the payload and replies 503 "Server is busy".

To mitigate this problem, always ensure the parsingqueue is atleast 10 times more than the max POST payload.

In server.conf

[queue=parsingQueue] 
maxSize = <minimum 10 x (max expected POST payload size)>