It seems that Splunk is only indexing the sourcetype defined last (in this example, it's WebSphere:SystemOutErrLog) when I'm monitoring the same folder but with different whitelists. I'm only seeing WebSphere:SystemOutErrLog being indexed. Is there a workaround to this issue?
[monitor://C:\Program Files (x86)\IBM\WebSphere\AppServer\profiles\AppSrv01\logs\server1\]
sourcetype = WebSphere:NativeStdOutErrLog
whitelist = native(.*)\.log$
crcSalt = <SOURCE>
disabled = 0
[monitor://C:\Program Files (x86)\IBM\WebSphere\AppServer\profiles\AppSrv01\logs\server1\]
sourcetype = WebSphere:StartStopServerLog
whitelist = startServer\.log$|stopServer\.log$
crcSalt = <SOURCE>
disabled = 0
[monitor://C:\Program Files (x86)\IBM\WebSphere\AppServer\profiles\AppSrv01\logs\server1\]
sourcetype = WebSphere:SystemOutErrLog
whitelist = SystemOut\.log$|SystemErr\.log$
crcSalt = <SOURCE>
disabled = 0
Out of curiosity, any reason why you aren't monitoring these with a single [monitor://]
and then setting up sourcetype rules in props.conf
. It seems that Splunk 4.1 is more flexibal than previous versions, but I'm still wondering which approach is better.
To be clear, here is what I'm suggesting:
inputs.conf:
[monitor://C:\Program Files (x86)\IBM\WebSphere\AppServer\profiles\AppSrv01\logs\server1]
whitelist = (native.*|(start|stop)Server|System(Out|Err))\.log$
Side Note: BTW, I saw your question where someone recommended using the "crcSalt = <SOURCE>
" entry, but you really don't want this setting when you are monitoring regular log files because using this setting will disable splunk's automatic detection and proper handling of rotated log files. Of course, if you change it now, your files may be indexed again, it's tricky. Which is also why it should be generally avoided.)
props.conf:
[source::...[/\\]WebSphere[/\\]...[/\\]native*.log]
sourcetype = WebSphere:NativeStdOutErrLog
[source::...[/\\]WebSphere[/\\]...[/\\](start|stop)Server.log]
sourcetype = WebSphere:StartStopServerLog
[source::...[/\\]WebSphere[/\\]...[/\\]System(Out|Error).log]
sourcetype = WebSphere:SystemOutErrLog
Note: I use the ackward looking expression [/\\]
to match both windows and unix style paths (that way I can use the same rule on either systems). You may want to match more than just WebSpehere
and the log name in your path, but this is just to give you a general idea. You probably don't want to match just the file name (without any other portion of the path) then could could have collisions with non-WebSphere logs files.
Of course, if you are not running 4.1 yet, then this would probably be your only option. I would be curious if anyone has any pros/cons to the multiple monitor approach vs the traditional props-based (source matching) approach?
BTW: I've used this an approach like this to get certain log files to get the correct sourcetypes for quite a number of configurations. When I first started using splunk I really wanted to use the multiple monitor approach (like what is now available in 4.1), however I now see the props-based approach as much more flexible. Think about it this way, when you add a new set of log files, do you want to have to remember the exact name of every sourcetype (especially if your doing this a few months down the road), or would you rather have existing rules setup that do the work for you.
Another advantage I see to the props-based approach is that you can interactively probe log files to determine there properties. This is great for new setups and for double-checking yourself after a config change. This feature has saved me TONS of time (and it works against the current config so you can test your props settings without restarting splunk.) To run the tool, open a command window, and cd
to splunk's bin
directory. Then run the following command:
splunk test sourcetype "<path to logfile>"
Ok, that was way too long. Sorry. Hopefully this was helpful.
Out of curiosity, any reason why you aren't monitoring these with a single [monitor://]
and then setting up sourcetype rules in props.conf
. It seems that Splunk 4.1 is more flexibal than previous versions, but I'm still wondering which approach is better.
To be clear, here is what I'm suggesting:
inputs.conf:
[monitor://C:\Program Files (x86)\IBM\WebSphere\AppServer\profiles\AppSrv01\logs\server1]
whitelist = (native.*|(start|stop)Server|System(Out|Err))\.log$
Side Note: BTW, I saw your question where someone recommended using the "crcSalt = <SOURCE>
" entry, but you really don't want this setting when you are monitoring regular log files because using this setting will disable splunk's automatic detection and proper handling of rotated log files. Of course, if you change it now, your files may be indexed again, it's tricky. Which is also why it should be generally avoided.)
props.conf:
[source::...[/\\]WebSphere[/\\]...[/\\]native*.log]
sourcetype = WebSphere:NativeStdOutErrLog
[source::...[/\\]WebSphere[/\\]...[/\\](start|stop)Server.log]
sourcetype = WebSphere:StartStopServerLog
[source::...[/\\]WebSphere[/\\]...[/\\]System(Out|Error).log]
sourcetype = WebSphere:SystemOutErrLog
Note: I use the ackward looking expression [/\\]
to match both windows and unix style paths (that way I can use the same rule on either systems). You may want to match more than just WebSpehere
and the log name in your path, but this is just to give you a general idea. You probably don't want to match just the file name (without any other portion of the path) then could could have collisions with non-WebSphere logs files.
Of course, if you are not running 4.1 yet, then this would probably be your only option. I would be curious if anyone has any pros/cons to the multiple monitor approach vs the traditional props-based (source matching) approach?
BTW: I've used this an approach like this to get certain log files to get the correct sourcetypes for quite a number of configurations. When I first started using splunk I really wanted to use the multiple monitor approach (like what is now available in 4.1), however I now see the props-based approach as much more flexible. Think about it this way, when you add a new set of log files, do you want to have to remember the exact name of every sourcetype (especially if your doing this a few months down the road), or would you rather have existing rules setup that do the work for you.
Another advantage I see to the props-based approach is that you can interactively probe log files to determine there properties. This is great for new setups and for double-checking yourself after a config change. This feature has saved me TONS of time (and it works against the current config so you can test your props settings without restarting splunk.) To run the tool, open a command window, and cd
to splunk's bin
directory. Then run the following command:
splunk test sourcetype "<path to logfile>"
Ok, that was way too long. Sorry. Hopefully this was helpful.
lengthy answer --> yes
helpful answer --> yes
I am impressed that you suggested the regex [/\] to match Windows and *NIX path delimiters.
I'll give this a try.
Try setting your inputs to debug mode, we will need to figure out why splunk's complaining on those files in your etc/log.cfg set these or input them if not already there.
category.TailingProcessor=DEBUG
category.WatchedFile=DEBUG
category.BatchReader=DEBUG
category.FileTracker=DEBUG
restart splunk, let it run a few minutes and check log/splunkd.log for any of those files like native*.log, splunk should be pretty verbose about what's wrong
it seems that i have to configure props.conf to monitor the log files
To splunk, you're defining the same input three times. We fold together stanzas with the same names, intentionally, to allow redefinitions from default to apps, to local settings.
Create independent stanzas by using wildcards as part of the stanza name (note, this is specific to 4.1).
What is the recommended solution on 4.3.2? Wildcards don't seem to work anymore.
Pretty unreadable, but yes that should work fine, on 4.1.
Lowell's approach will work on 4.0 also.
Are you suggesting:
[monitor://C:\Program Files (x86)\IBM\WebSphere\AppServer\profiles\AppSrv01\logs\server1\native*.log]
sourcetype = WebSphere:NativeStdOutErrLog
crcSalt =
[monitor://C:\Program Files (x86)\IBM\WebSphere\AppServer\profiles\AppSrv01\logs\server1\st*Server.log]
sourcetype = WebSphere:StartStopServerLog
crcSalt =
[monitor://C:\Program Files (x86)\IBM\WebSphere\AppServer\profiles\AppSrv01\logs\server1\System*.log]
sourcetype = WebSphere:SystemOutErrLog
crcSalt =