I keep seeing this message in splunkd.log on my instance, what does it mean?
My instance is used primarily as a search and indexing instance, and it also distributes searches to other indexing instances.
asked 22 Jan '10, 18:35
It means that your search-head dispatched a search to the peer machines but didn't get a response from one or more peers within the specified timeout period.
Should have gotten at least 3 tokens in status line = I should have gotten a response from servers
Only got 0 = I didn't get a response
The error message is lacking in that it details a potential problem but it doesn't reveal a source machine. In general though, it's indicative of an over-worked search peer that doesn't have the cycles to spare to handle another search request, or there could be a network issue in between the 2 machines.
It's also possible that the search-head is itself oversubscribed in one way or another. For example, it is not recommended to run the Deployment Server on a search-head instance, as both this component and distributed search share the splunkd management port. Run the Deployment Server in its own dedicated instance instead.
First step to resolution will be to figure out which peer is causing the error message. Try turning your peers on one at a time to figure this out.
From my own experience I would rather think it means something else, or that there is a bug, or that something is slightly misconfigured.
I keep getting these messages from ALL BUT ONE of the hosts in my setup .
1 Indexer v4.2.1
I do not think it is network related, since the Search Head and Indexer are on the same VLAN and neither of them are under any considerable load (~5GB of daily log).
I do not believe that it has anything to do with HOW logs are transported to the indexer. Below is the outputs.conf that is delivered to all forwarders.
I would rather guess that this is along the lines of the bogus(?) error message:
That error also came in from all forwarders until we upgraded from 4.2.0 to 4.2.1. So currently there is one UF (4.2.0) still sending these latter error messages, but the logs keep coming in anyway.
If anyone knows a definite answer it'd be good to know, but I'm not too worried as everything seems to be working fine.
answered 14 Jul '11, 02:55