LightForwarder, Not sending updated log entries

Hi,
Have a lightforwarder configured to send updated entries from /mnt/nagios/nagios.log on 10.1.1.1. It looks like there was an initial load into the search app (42k events) and it hasn't updated in 5 days. Also interesting is that on stop/start it shows parsing configuration for the file, but never states "Will begin reading". The log itself is being updated every couple minutes and shows an updated timestamp on 10.1.1.1. Permissions are open to 755. Syslog is being sent and properly updated to our splunk instance. I also have nagios events logged to syslog and those are appearing (just in-case this sorta thing were to happen). but, I would really like to disable that and have the log with the sep. index and sourcetype be logged from the proper log.

FORWARDER:
./splunk list monitor
Monitored Files:
/mnt/nagios/nagios.log

inputs.conf in search/local:
[monitor:///mnt/nagios/nagios.log]
disabled = false
host = nagios.blah.blah.com
sourcetype = nagios
index = nagios

outputs.conf in search/local:
[tcpout]
defaultGroup = 10.1.1.1_514
disabled = false

[tcpout:10.1.1.1_514]
server = 10.1.1.1:514

[tcpout-server://10.1.1.1:514]

stop/start log:
9-14-2010 17:50:32.263 INFO loader - Server supporting SSL v2/v3
09-14-2010 17:50:32.263 INFO loader - Using cipher suite ALL:!aNULL:!eNULL:!LOW:!EXP:RC4+RSA:+HIGH:+MEDIUM
09-14-2010 17:50:32.272 INFO TPool - initializing BatchReaderTPool with 1 workers
09-14-2010 17:50:32.272 INFO TcpOutputProc - attempting to connect to 10.1.1.1:514...
09-14-2010 17:50:32.273 INFO TcpOutputProc - Connected to 10.1.1.1:514
09-14-2010 17:50:33.513 INFO TailingProcessor - TailWatcher initializing...
09-14-2010 17:50:33.543 INFO TailingProcessor - Parsing configuration stanza: monitor:///mnt/nagios/nagios.log.
09-14-2010 17:50:33.544 INFO WatchedFile - Will begin reading at offset=7600309 for file='/mnt/nagios/nagios.log'.
09-14-2010 17:50:53.056 INFO timeinvertedIndex - starting loggerPipe eloop
09-14-2010 17:50:53.056 INFO timeinvertedIndex - running loggerPipe eloop

INDEXER:
inputs.conf in search/local:
[splunktcp://514]

inputs.conf in system/local:
[default] host = splunk.blah.blah.com

asked 14 Sep '10, 17:57

drewbfl
43●7
accept rate: 0%

edited 14 Sep '10, 22:23

One Answer:

oldest newest most voted

Did you happen to enable LWF in the last 5 days/since setting up the forwarder? The index parameter in inputs.conf on a LWF is not honored. It needs to be a regular forwarding if you want to perform routing to an index other than the default.

Thanks for updating your description. Can you try adding this to the inputs.conf on the indexer?

[monitor:///mnt/nagios/nagios.log]
index = nagios

Also, did you try enabling "index and forward" on the forwarder to ensure that data is indeed getting indexed and to the correct index? Then we can rule out any input config issues.

link

answered 14 Sep '10, 18:08

hulahoop ♦
2.5k●3●2●40
accept rate: 40%

edited 15 Sep '10, 16:13

enabled SplunkForwarder. stoppped. started.

still no luck.

(14 Sep '10, 18:52) drewbfl

is index=nagios created on the indexer?

(14 Sep '10, 19:12) hulahoop ♦

if it is, then try enabling local indexing on the forwarder to ensure there is nothing wrong with the input config. you'll probably have to create the nagios index temporarily on the forwarder.

(14 Sep '10, 19:22) hulahoop ♦

i should also note, if you want to use the LWF, then i believe you can put the index=nagios setting on the indexer.

(14 Sep '10, 19:26) hulahoop ♦

it is on the indexer. interestingly, the latest event in the nagios index is accurate. it must be pulling that from the syslog source. the source and sourcetype on the main search app still have the stale numbers.

(14 Sep '10, 20:40) drewbfl

would you please update your question with inputs.conf from forwarder and indexer?

(14 Sep '10, 21:41) hulahoop ♦

i added it above. thanks

(14 Sep '10, 22:24) drewbfl

Didn't help. I tried adding it to both system/local and search/local inputs.confs and it didn't help.

(15 Sep '10, 21:37) drewbfl

I'm sorry these steps haven't produced any different results for you. Have you tried enabling "index and forward" on the forwarder? If that does not produce the correct result, then I would recommend opening a ticket with the Splunk support team to have your configuration files reviewed in detail.

(16 Sep '10, 00:17) hulahoop ♦

I really don't want the forwarder to do any indexing, it doesn't have the cycles nor should it need to. Isn't this a common thing everyone does with the product?

(17 Sep '10, 14:15) drewbfl

I just mean to enable it for debugging purposes.

(20 Sep '10, 17:04) hulahoop ♦

using index= in inputs.conf on LWF does work, and should work, and is the preferred way to set an index when using a LWF. What does not work is routing to an index via transforms.

(22 Dec '10, 23:29) gkanapathy ♦

showing 5 of 12 show 7 more comments ▼

Post your answer

toggle preview

Follow this question

Email:

RSS:

Answers

Answers + Comments

Markdown Basics

*italic* or _italic_
**bold** or __bold__
link:[text](http://url.com/ "Title")
image?![alt text](/path/img.jpg "Title")
numbered list: 1. Foo 2. Bar
to add a line break simply add two spaces to where you would like the new line to be.
basic HTML tags are also supported

learn more about Markdown

Tags:

syslog ×185
lightforwarder ×120
nagios ×18

Asked: 14 Sep '10, 17:57

Seen: 1,024 times

Last updated: 03 Apr '11, 19:22

LightForwarder, Not sending updated log entries

Follow this question

Splunk Resources

Related questions