splunkd.log: Denied session token for user: splunk-system-user

[Stefan]

My indexers started throwing this:

07-02-2011 04:00:58.307 -0300 WARN AdminHandler:AuthenticationHandler - Denied session token for user: splunk-system-user

07-02-2011 04:01:05.551 -0300 WARN AdminHandler:AuthenticationHandler - Denied session token for user: splunk-system-user

As a result, I get an error on the Search Heads stating that it has "skipped indexing of internal audit event will keep dropping events..."

The last time I got this warning at the Search Heads it was due to the optimization taking too long, but this one is new to me, any ideas?

[yoho]

It may be due to an edition of distsearch.conf without establishing trusts (indexers must trust searchhead's certificates) so you can fix this problem by following what's written in this paragraph.

[jbsplunk · Accepted solution]

Invalid authentication between the search peers and the search head distributing search to these peers would result in an 'Authenication Failed' type of message. This error indicates something is wrong with the token. That the search peer is sending back something in its header requesting access via an invalid token. It could also be related to time I suppose. Have you looked at splunkd_access.log to see if there are any errors in that file?

[anttih]

have you changed your admin passwords recently and is the search head using the correct admin user of the indexer ?
I believe this refers to invalid authentication between search peer and indexer and can be setup in the search peers part of the management console.

[Jason]

Incorrect. The search head only uses the admin password of the indexer once - to send over a certificate. The certificate is used in Distributed Search from there on, so it does not matter if any password is changed on the indexer.