Data Gaps in Indexing

Hello,

I have been encountering a problem with some of my file monitoring inputs. Specifically I am trying to monitor a few jboss server logs that tend to be very busy during periods of heavy load for this application (several thousand messages written every minute). I have setup the splunk installation on these jboss servers as lightweight forwarders forwarding the jboss server log events to the central splunk indexer.

During periods of heavy load on this application, I have observed that the data from these server logs stops getting forwarded to the central indexer for several minutes (not just lags in indexing but complete gaps in the data for several minutes). I followed the troubleshooting tips specified in the following URL - http://www.splunk.com/wiki/Community:Troubleshooting_Monitor_Inputs and noticed the following pattern as noted in this URL.

"Files that cease to be indexed but don't close

If you see many entries of the form:

    06-04-2009 09:31:25.003 DEBUG selectProcessor - EOF '<path_to_file>' found. File has been written to in the last '5' seconds. Will keep open.

But none of the form:

    06-04-2009 09:31:25.003 INFO  FileInputTracker - Computing CRC for seekPtr=55de367d filename=<path_to_file>

Then your file is being kept open, but no new data is arriving. The second type of message shows Splunk is advancing through the file, while the first shows that Splunk believes the file is 'active' via the modification time. "

Although this section identifies the possible cause for these data gaps, as

*  Time skew between servers, where the files are timestamped in the future
* Bugs where programs truncate their logs (log4j will do this on windows in some cases) 

I am not sure what I can do to fix this issue, since the time stamps on the forwarder and the indexer are in sync and I am not sure how to identify and tackle the bugs in log truncation.

Any help would be greatly appreaciated!!!

Thanks,

Deepak