Monitoring Splunk

Splunkd crash after upgrade to 4.3

crossroadsIT
Engager

After upgrading our splunk server to 4.3 from 4.2.5 splunkd crashes with the following errors in splunkd_stderr.log:

2012-01-11 11:01:27.098 +0800 splunkd started (build 115073)
terminate called after throwing an instance of 'PropertyPagesException'
what():  Cannot get user to act as: No user info provider registered (user: xxxx, app: user-prefs, root: /opt/splunk/etc)

Running on CentOS 5.7 64-bit, dual Quad Core Xeon with 8GB RAM.

Tags (3)
1 Solution

hexx
Splunk Employee
Splunk Employee

I believe you are experiencing a crash that was discovered just today and filed as a bug with reference SPL-47232.

Symptoms:

The signature of this crash is as follows :

  • The crashing thread is always DispatchReaper, the thread which curates the search artifacts in the dispatch directory ($SPLUNK_HOME/var/run/splunk/dispatch). The crashing thread is indicated at the very beginning of the crash log file that can be found in $SPLUNK_HOME/var/log/splunk :

[build 115073] 2012-01-12 14:54:12
Received fatal signal 6 (Aborted).
Cause:
Signal sent by PID 28468 running under UID 0.
Crashing thread: DispatchReaperCrashing thread: DispatchReaper
<==

  • The file logging splunkd's stderr output ($SPLUNK_HOME/var/log/splunk/splunkd_stderr.log) will contain the following error:

terminate called after throwing an instance of 'PropertyPagesException'
what(): Cannot get user to act as: No user info provider registered (user: splunk-system-user, app: user-prefs, root: /opt/splunk/etc)

  • The crash almost always occurs on splunkd start-up, sometimes on the first start-up attempt after the 4.3 upgrade. It has also been known to occur during normal operation of the Splunk instance.

Work-around:

Until this crash is fixed in an upcoming release, you'll have to take the following steps to allow splunkd to start again:

  • Delete the contents of the search dispatch directory:

rm -rf $SPLUNK_HOME/var/run/splunk/dispatch/*

  • Start up Splunk:

$SPLUNK_HOME/bin/splunk start

Note: If you are unfamiliar with the search dispatch directory, it is the location where Splunk stores search artifacts for past and currently running searches. That data is volatile by nature and can be regenerated by re-running the searches that generated it.

View solution in original post

hexx
Splunk Employee
Splunk Employee

I believe you are experiencing a crash that was discovered just today and filed as a bug with reference SPL-47232.

Symptoms:

The signature of this crash is as follows :

  • The crashing thread is always DispatchReaper, the thread which curates the search artifacts in the dispatch directory ($SPLUNK_HOME/var/run/splunk/dispatch). The crashing thread is indicated at the very beginning of the crash log file that can be found in $SPLUNK_HOME/var/log/splunk :

[build 115073] 2012-01-12 14:54:12
Received fatal signal 6 (Aborted).
Cause:
Signal sent by PID 28468 running under UID 0.
Crashing thread: DispatchReaperCrashing thread: DispatchReaper
<==

  • The file logging splunkd's stderr output ($SPLUNK_HOME/var/log/splunk/splunkd_stderr.log) will contain the following error:

terminate called after throwing an instance of 'PropertyPagesException'
what(): Cannot get user to act as: No user info provider registered (user: splunk-system-user, app: user-prefs, root: /opt/splunk/etc)

  • The crash almost always occurs on splunkd start-up, sometimes on the first start-up attempt after the 4.3 upgrade. It has also been known to occur during normal operation of the Splunk instance.

Work-around:

Until this crash is fixed in an upcoming release, you'll have to take the following steps to allow splunkd to start again:

  • Delete the contents of the search dispatch directory:

rm -rf $SPLUNK_HOME/var/run/splunk/dispatch/*

  • Start up Splunk:

$SPLUNK_HOME/bin/splunk start

Note: If you are unfamiliar with the search dispatch directory, it is the location where Splunk stores search artifacts for past and currently running searches. That data is volatile by nature and can be regenerated by re-running the searches that generated it.

crossroadsIT
Engager

This is the exact error that we had faced. Interestingly our Splunk instance started working of its' own accord after a couple of hours.

0 Karma

hexx
Splunk Employee
Splunk Employee

We will at least need to look at the corresponding crash log and at what is logged at the time of the crash in splunkd.log in order to comment. I would strongly recommend that you log a support case and attach a Splunk diag to get this crash analyzed. It doesn't seem likely that we'll be able to determine the cause of the problem just from the information provided so far.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...