We plan to use Splunk to keep log for several java application including web server like Tomcat. Those application are using log4j with org.apache.log4j.RollingFileAppender. The partial config will be like below:
log4j.appender.R.MaxFileSize=10MB
log4j.appender.R.MaxBackupIndex=20
That is when the server.log reaches 10MB, it will be renamed/rollover to server.log.1 and server.log.1 will be server.log.2 and so forth...
My questions:
You should monitor the whole set of server.log*
files. There are some circumstances where you will lose events if you just monitor server.log. First is if Splunk happens to be down while a file rolls. There is also some possibility of losing a few events from the end of the file depending on exactly how the roll is done (via a rename and then creation of a new file, or a copy and truncate, or what), particularly if the file is not updated often enough for Splunk to keep an open file descriptor on it at the time it rolls, e.g., maybe it's only update every 60 seconds.
The cost is that the monitoring processor has to track more files, but this is not a very big problem in version 4.1+, and with only 20 files, this is fine in older versions too (though you might want to raise the max_fd in those versions).
Let me throw in one more scenario, say server.log
rotates to server.log.n+1
at the same time as when splunkd is down due to a configuration change.
In general, Splunk does a really good job a keeping up with log files but there are a few scenarios that just aren't covered. And since splunk automatically recognizes rotated log files (and therefore will only indexed the previously unread portions of the log file), to me it can make sense to monitor all of the log files at once. Especially, in scenarios where you can't afford to drop and events.
Also, I don't like seeing multiple sources in this kind of scenario (e.g. server.log.1
, server.log.2
, ... server.log.N
) when they are all originally from a single log file, so I use a transformer to renaming them back to the original server.log
name.
Here is an example set of configs:
Example inputs.conf
:
[monitor:///opt/java_app/server.log*] sourcetype = java_app_server
Example props.conf
:
[java_app_server] TRANSFORMS-rename_source = drop_trailing_digit
Example transforms.conf
:
[drop_trailing_digit] DEST_KEY = MetaData:Source SOURCE_KEY = MetaData:Source REGEX = source::(.*)[-._]\d+$ FORMAT = source::$1
I've found this kind of setup to work well. Anyone know of a reason why not to monitor all the rotated log files too? I know there are some advantages to monitoring a single file vs a directory like this, but I've never observed any performance impact from it.
For anyone interested, there are some additional source renaming examples here: http://answers.splunk.com/questions/3470/consolidate-similarly-named-log-files-into-a-single-source/...
Hi Alan,
I may be able to answer question 1 but question 2 is definitely worth investigating.
IMHO, you only need to monitor server.log
This is because when log4j detects that server.log reaches 10MB it will rollover and write its content to a new file, namely server.log.n, and then clears itself to append new output stream (which the other log file would become server.log.n+1).
Nicholas
You should monitor all the logs, server.log*
. There are only 20 files, so it's not much more work. Please see my answer for why.