I have been encountering a problem with some of my file monitoring inputs. Specifically I am trying to monitor a few jboss server logs that tend to be very busy during periods of heavy load for this application (several thousand messages written every minute). I have setup the splunk installation on these jboss servers as lightweight forwarders forwarding the jboss server log events to the central splunk indexer.
During periods of heavy load on this application, I have observed that the data from these server logs stops getting forwarded to the central indexer for several minutes (not just lags in indexing but complete gaps in the data for several minutes). I followed the troubleshooting tips specified in the following URL - http://www.splunk.com/wiki/Community:Troubleshooting_Monitor_Inputs and noticed the following pattern as noted in this URL.
"Files that cease to be indexed but don't close
If you see many entries of the form:
But none of the form:
Then your file is being kept open, but no new data is arriving. The second type of message shows Splunk is advancing through the file, while the first shows that Splunk believes the file is 'active' via the modification time. "
Although this section identifies the possible cause for these data gaps, as
I am not sure what I can do to fix this issue, since the time stamps on the forwarder and the indexer are in sync and I am not sure how to identify and tackle the bugs in log truncation.
Any help would be greatly appreaciated!!!
asked 08 Jul '10, 21:33