SplunkWeb Service occasionally hangs...

Hey guys,

Doesn't seem like many people have had problems with the Splunk Web Service hanging on them, but this is somewhat similar to:

http://answers.splunk.com/questions/1468/splunkweb-crashes-although-splunkd-is-still-running-is-this-a-known-problem

Basically, I have been testing my Foundstone Vulnerability Scanner on my splunk indexers and search head and it looks like at least one of servers has its Splunk Web Service die. Port is reachable via telnet, and splunk status states that the service is up, but the site itself is unreachable.

I simulated the situation and tailed the logs and see these types of errors:

splunkd.log

06-22-2010 20:42:42.816 ERROR NetUtils - SSL_ERROR_SSL in SSL_write. nbytes=-1, Error = error:140D00CF:SSL routines:SSL_write:protocol is shutdown

06-22-2010 20:42:43.210 ERROR TcpInputFd - SSL Error = error:1407609C:SSL routines:SSL23_GET_CLIENT_HELLO:http request
06-22-2010 20:42:43.210 ERROR TcpInputFd - ACCEPT_RESULT=-1 VERIFY_RESULT=0
06-22-2010 20:42:43.210 ERROR TcpInputFd - SSL Error for fd from HOST:$foundstone_scanner$, IP:$foundstone_IP$, PORT:5927
06-22-2010 20:42:43.365 ERROR TcpInputFd - SSL Error = error:1407609C:SSL routines:SSL23_GET_CLIENT_HELLO:http request
06-22-2010 20:42:43.365 ERROR TcpInputFd - ACCEPT_RESULT=-1 VERIFY_RESULT=0
06-22-2010 20:42:43.365 ERROR TcpInputFd - SSL Error for fd from HOST:$foundstone_scanner$, IP:$foundstone_IP$, PORT:5939
06-22-2010 20:42:44.116 ERROR TcpInputFd - SSL Error = error:1407609C:SSL routines:SSL23_GET_CLIENT_HELLO:http request
06-22-2010 20:42:44.116 ERROR TcpInputFd - ACCEPT_RESULT=-1 VERIFY_RESULT=0

06-22-2010 20:45:11.376 INFO TailingProcessor - File descriptor cache is full (64), trimming...
06-22-2010 20:45:17.397 INFO TailingProcessor - File descriptor cache is full (64), trimming...
06-22-2010 20:45:19.516 INFO TailingProcessor - File descriptor cache is full (64), trimming...
06-22-2010 20:46:22.830 INFO TailingProcessor - File descriptor cache is full (64), trimming...

web_service.log

This is the error message that appears right before it hangs:

2010-06-18 19:05:05,842 ERROR [4c1798223163010d0] root:120 - ENGINE: Error in HTTP server: shutting down
Traceback (most recent call last):
File "/opt/splunk/lib/python2.6/site-packages/cherrypy/process/servers.py", line 73, in _start_http_thread
File "/opt/splunk/lib/python2.6/site-packages/cherrypy/wsgiserver/init.py", line 1662, in start
self.tick()
File "/opt/splunk/lib/python2.6/site-packages/cherrypy/wsgiserver/init.py", line 717, in tick
s, addr = self.socket.accept()
File "/opt/splunk/lib/python2.6/ssl.py", line 317, in accept
newsock, addr = socket.accept(self)
File "/opt/splunk/lib/python2.6/socket.py", line 195, in accept
error: [Errno 24] Too many open files

Anyone ever have the same issue?

I also ran splunk diag so that I can open support ticket but just wanted to see if anyone else has ever had this problem.

Let me know.

Thanks Guys!

Brian