I am having issues setting up a UNIX universal forwarder to monitor IBM IHS http log files -- it does not appear to be setting up the monitor. I can see that the stanza in the inputs.conf is being parsed by looking at the splunkd.log, but I don't see a corresponding entry "Adding watch on path...".
Am I missing something in my configuration? What would prevent a watch from being added? I've checked the UNIX file permissions and the 'splunk' user has read-access to all files and directories.
All other monitored entities in my inputs.conf have a log entry indicating that a watch has been started. There are no errors in the splunkd.log file.
splunkd.log on forwarder host
TailingProcessor - Parsing configuration stanza: monitor:///apps/logs/http/IBMIHS.
...
09-11-2014 08:53:48.274 -0500 INFO TailingProcessor - Adding watch on path: /apps/logs/WebSphere/AS_WAS_01.
09-11-2014 08:53:48.274 -0500 INFO TailingProcessor - Adding watch on path: /apps/logs/WebSphere/AS_WAS_02
09-11-2014 08:53:48.274 -0500 INFO TailingProcessor - Adding watch on path: /apps/splunkforwarder/etc/splunk.version.
09-11-2014 08:53:48.274 -0500 INFO TailingProcessor - Adding watch on path: /apps/splunkforwarder/var/log/splunk.
09-11-2014 08:53:48.274 -0500 INFO TailingProcessor - Adding watch on path: /apps/splunkforwarder/var/spool/splunk.
--> no corresponding entry saying that a watch has been added for IHS -- see config below.
inputs.conf on forwarder host
[monitor:///apps/logs/http/IBMIHS]
disabled = false
recursive = false
index = myindex
blacklist = \.(gz)$
sourcetype = access_combined
inputs.conf on indexer host
[access_combined]
pulldown_type = true
maxDist = 28
MAX_TIMESTAMP_LOOKAHEAD = 128
REPORT-access = access-extractions
SHOULD_LINEMERGE = False
TIME_PREFIX = \[
transforms.conf on indexer host
[access-extractions]
\# matches access-common or access-combined apache logging formats
\# Extracts: clientip, clientport, ident, user, req_time, method, uri, root, file, uri_domain, uri_query, version, status, bytes, referer_url, referer_domain, referer_proto, useragent, cookie, other (remaining chars)
\# Note: referer is misspelled in purpose because that is the "official" spelling for "HTTP referer"
REGEX = ^[[nspaces:clientip]]\s++[[nspaces:ident]]\s++[[nspaces:user]]\s++[[sbstring:req_time]]\s++[[access-request]]\s++[[nspaces:status]]\s++[[nspaces:bytes]](?:\s++"(?<referer>[[bc_domain:referer_]]?+[^"]*+)"(?:\s++[[qstring:useragent]](?:\s++[[qstring:cookie]])?+)?+)?[[all:other]]
Used workaround outlined in here
Everything looks good. The blacklist is strange though. Do you really need the parentheses around gz? I have always done it:
blacklist=\.gz$
I've adjusted the blacklist parameters & re-started the universal forwarder.
Sill no go...no log entry that indicates that a watch has been added, nor is the indexer picking anything up.