The management interface ran fine until I restarted to install Universal forwarder, now splunkd will not start. Universal forwarder installed directly from the GUI. Fortunately this is a vm, so I’ve restored snapshot to just before splunk install. Unfortunately this happens each time – here’s the sequence.
Install splunk as root using dpkg –i splunk-4.2-96430-linux-2.6-amd64.deb
Start splunk - /opt/splunk/bin/splunk start, get the typical successful start dialogue
Login to the management console, configure collecting data for the splunk server
Go to manage apps, and enable universal forwarder, it then prompts to restart the server
Click the link in management console to restart
Restart splunk in CLI on server – splunk restart,
root@deb-splunk:~# /opt/splunk/bin/splunk start splunkd
Splunk> All batbelt. No tights.
Checking prerequisites...
Checking mgmt port [8089]: open
Checking configuration... Done.
Checking index directory...
Validated databases: _audit _blocksignature _internal _thefishbucket history main summary
Done
Success
Checking conf files for typos...
All preliminary checks passed.
Starting splunk server daemon (splunkd)...
Done.
root@deb-splunk:~#
No restart of splunkweb appears, and doing splunk status shows:
root@deb-splunk:~# /opt/splunk/bin/splunk status
splunkd 1968 was not running.
Removing stale pid file... done.
splunkweb is not running.
Here's the crashlog:
Received fatal signal 6 (Aborted).
Cause:
Signal sent by PID 2119 running under UID 0.
Crashing thread: MainTailingThread
Registers:
RIP: [0x00007FFB77221165] gsignal + 53 (/lib/libc.so.6)
RDI: [0x0000000000000847]
RSI: [0x000000000000085A]
RBP: [0x0000000002909A68]
RSP: [0x00007FFB75DE38C8]
RAX: [0x0000000000000000]
RBX: [0x00000000014FE8B0]
RCX: [0xFFFFFFFFFFFFFFFF]
RDX: [0x0000000000000006]
R8: [0x00007FFB78C93037]
R9: [0x2C7472617473206F]
R10: [0x0000000000000008]
R11: [0x0000000000000206]
R12: [0x000000000290C040]
R13: [0x00007FFB75DE3A60]
R14: [0x0000000002891B40]
R15: [0x0000000001547E80]
EFL: [0x0000000000000206]
TRAPNO: [0x0000000000000000]
ERR: [0x0000000000000000]
CSGSFS: [0x0000000000000033]
OLDMASK: [0x0000000000000000]
OS: Linux
Arch: x86-64
Backtrace:
[0x00007FFB77223F70] abort + 384 (/lib/libc.so.6)
[0x0000000000F7D068] ZN9_gnu_cxx27__verbose_terminate_handlerEv + 200 (splunkd)
[0x0000000000F7CE16] ZN10_cxxabiv111__terminateEPFvvE + 6 (splunkd)
[0x0000000000F7CE43] ? (splunkd)
[0x0000000000F7CF43] ? (splunkd)
[0x0000000000957C66] _ZN19InputProcessorKindaC2ER6Logger + 230 (splunkd)
[0x0000000000669D9A] _ZN11TailWatcherC1ERK3StrP11InputStatus + 90 (splunkd)
[0x000000000066A2E4] _ZN13TailingThread4mainEv + 244 (splunkd)
[0x0000000000BB03B2] _ZN6Thread8callMainEPv + 66 (splunkd)
[0x00007FFB788638BA] ? (/lib/libpthread.so.0)
Linux / deb-splunk / 2.6.32-5-amd64 / #1 SMP Wed Jan 12 03:40:32 UTC 2011 / x86_64
Last few lines of stderr (may contain info on assertion failure, but also could be old):
2011-03-17 11:59:18.937 -0700 Interrupt signal received
2011-03-17 11:59:30.099 -0700 splunkd started (build 96430)
terminate called after throwing an instance of 'PluginException'
what(): Indexer failed to start, will not continue.
2011-03-17 12:03:22.591 -0700 splunkd started (build 96430)
terminate called after throwing an instance of 'PluginException'
what(): Indexer failed to start, will not continue.
2011-03-17 12:11:49.390 -0700 splunkd started (build 96430)
terminate called after throwing an instance of 'PluginException'
what(): Indexer failed to start, will not continue.
2011-03-17 12:17:50.449 -0700 splunkd started (build 96430)
terminate called after throwing an instance of 'PluginException'
what(): Indexer failed to start, will not continue.
/etc/debian_version: 6.0
glibc version: 2.11.2
glibc release: stable
Threads running: 13
argv: [splunkd -p 8089 start splunkd]
terminating...
... View more