ERROR HTTPClient - Should have gotten at least 3 tokens in status line, while getting response code. Only got 0. - what does it mean?

Question

I keep seeing this message in splunkd.log on my instance, what does it mean?

My instance is used primarily as a search and indexing instance, and it also distributes searches to other indexing instances.

Answer 1

It means that your search-head dispatched a search to the peer machines but didn't get a response from one or more peers within the specified timeout period.

Should have gotten at least 3 tokens in status line = I should have gotten a response from servers

Only got 0 = I didn't get a response

The error message is lacking in that it details a potential problem but it doesn't reveal a source machine. In general though, it's indicative of an over-worked search peer that doesn't have the cycles to spare to handle another search request, or there could be a network issue in between the 2 machines.

First step to resolution will be to figure out which peer is causing the error message. Try turning your peers on one at a time to figure this out.

Answer 2

From my own experience I would rather think it means something else, or that there is a bug, or that something is slightly misconfigured.

I keep getting these messages from ALL BUT ONE of the hosts in my setup .

1 Indexer v4.2.1
1 Search Head v4.2.1
1 Heavy Forwarder v4.2.2 <-- only the 4.2.2 host is NOT generating these messages.
1 Universal Forwarder v4.2.0
50-odd Universal Forwarders v4.2.1

I do not think it is network related, since the Search Head and Indexer are on the same VLAN and neither of them are under any considerable load (~5GB of daily log).

I do not believe that it has anything to do with HOW logs are transported to the indexer. Below is the outputs.conf that is delivered to all forwarders.

[tcpout]
defaultGroup = splunkssl
disabled = false
compressed=true

[tcpout:splunkssl]
server = splunkindex.company.com:9997

[tcpout-server://splunkindex.company.com:9997]
sslCertPath = $SPLUNK_HOME/etc/apps/company-forwarding/local/company-splunk-forwarder.pem
sslCommonNameToCheck = company-splunk-indexer.company.com
sslPassword = XXXXXXXXX
sslRootCAPath = $SPLUNK_HOME/etc/apps/company-forwarding/local/company-ca.pem
sslVerifyServerCert = true

I would rather guess that this is along the lines of the bogus(?) error message:

Error encountered for connection from src=10.XXX.XXX.XXX:65278. Success

That error also came in from all forwarders until we upgraded from 4.2.0 to 4.2.1. So currently there is one UF (4.2.0) still sending these latter error messages, but the logs keep coming in anyway.

If anyone knows a definite answer it'd be good to know, but I'm not too worried as everything seems to be working fine.

Kristian

Answer 3

0

I see these errors from a Windows Server 2003 machine and my log machine is way under utilized?

link

answered 07 Apr '11, 20:25

mntbighker
1●1
accept rate: 0%

I am seeing the same thing... windows 2003 server used as indexer with distributed search head running on linux.

ERROR HTTPClient - Should have gotten at least 3 tokens in status line, while getting response code. Only got 0. ERROR TcpInputFd - SSL Error for fd from HOST:yy.xxx.55.10, IP:yy.xxx.55.10, PORT:45123 ERROR TcpInputFd - ACCEPT_RESULT=-1 VERIFY_RESULT=0 ERROR TcpInputFd - SSL Error = error:1407609C:SSL routines:SSL23_GET_CLIENT_HELLO:http request

working fine with no errors when linux search head to linux indexer.

(14 Oct '11, 17:59) EricPartington

ERROR HTTPClient - Should have gotten at least 3 tokens in status line, while getting response code. Only got 0. - what does it mean?

Follow this question

Splunk Resources

Related questions