I am trying to configure Splunk to monitor a service that has some unusual (to me) logging behaviour (unfortunately re-writing the logging code is not an option as it is 3rd party)
In folder X it creates several files
and appends to them.
After 24 hours it
and truncates/creates the original file again
If I just monitor the original files i.e. A.log, ..., would it be possible that splunk may miss a logging event due to the event being written and the file being copied/truncated?
If I try to monitor all the log files (so I never loose a message) can I wildcard it such that all A*.log events have a common sourcetype?
If I can end up watching all the log files it is possible I'll have duplicate entries 1st from the orginal and the 2nd from the copy into the old folder and if so is there a common/known strategy to handle that within splunk?
asked 04 Mar '12, 20:32
You probably want to monitor via a wildcard, unless your process pauses before rotating logs.
Why? If you use a wildcard to monitor a path, Splunk will index duplicates. Splunk calculates a CRC of the file and uses that when keeping track of how far into a file it has read. If the log is rotated, but still included in them monitor stanza (i.e. via a wildcard) Splunk will know it has already read most of that file and only read anything it hasn't already indexed.
NB: Caveats - This is not true if files are compressed as part of rotation. If a wildcard includes compressed files, they should be blacklisted. A generic blacklist might look something like this:
Using crcSalt=<source> would also break this model. For a more complete explanation of how Splunk handles log file rotation, check out said topic in the Getting Things in Manual.
answered 04 Mar '12, 20:44