How to fix my props.conf configuration to extract headers from a log file with garbage data on top?

[normangoh]

Hi guys,

I need to extract headers from a log file, so that when it is pushed to the Indexer, those headers will be displayed.

The log file looks like this.

#Software: IIS Advanced Logging Module
#Version: 1.0
#Start-Date: 2014-11-11 00:00:00.210
#Fields:  date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) cs(Referer) sc-status sc-substatus sc-win32-status sc-bytes cs-bytes TimeTakenMS
2014-11-11 00:00:03.283 172.18.10.88 GET / - 80 - - - - 200 0 0 0 7 46
2014-11-11 00:00:03.736 172.18.10.88 GET / - 80 - - - - 200 0 0 0 7 46
2014-11-11 00:00:08.291 172.18.10.88 GET / - 80 - - - - 200 0 0 0 7 46
2014-11-11 00:00:08.728 172.18.10.88 GET / - 80 - - - - 200 0 0 0 7 46
2014-11-11 00:00:13.299 172.18.10.88 GET / - 80 - - - - 200 0 0 0 7 46
2014-11-11 00:00:13.751 172.18.10.88 GET / - 80 - - - - 200 0 0 0 7 46
2014-11-11 00:00:18.306 172.18.10.88 GET / - 80 - - - - 200 0 0 0 7 46
2014-11-11 00:00:18.759 172.18.10.88 GET / - 80 - - - - 200 0 0 0 7 62
2014-11-11 00:00:23.064 172.18.10.88 GET /announce/6mBill-result.html - 80 - "69.191.211.202" "BLP_bbot/0.1" - 301 0 0 257 270 0
2014-11-11 00:00:23.314 172.18.10.88 GET / - 80 - - - - 200 0 0 0 7 46
2014-11-11 00:00:23.579 172.18.10.88 GET /News/T-Bill-Announcements.aspx - 80 - "69.191.211.202" "BLP_bbot/0.1" "http://www.sgs.gov.sg/announce/6mBill-result.html" 200 0 0 24830 493 62
2014-11-11 00:00:23.766 172.18.10.88 GET / - 80 - - - - 200 0 0 0 7 62
2014-11-11 00:00:28.337 172.18.10.88 GET / - 80 - - - - 200 0 0 0 7 62
2014-11-11 00:00:28.665 172.18.10.88 GET / - 80 - - - - 200 0 0 0 7 46
2014-11-11 00:00:33.329 172.18.10.88 GET / - 80 - - - - 200 0 0 0 7 46
2014-11-11 00:00:33.673 172.18.10.88 GET / - 80 - - - - 200 0 0 0 7 46
2014-11-11 00:00:38.384 172.18.10.88 GET / - 80 - - - - 200 0 0 0 7 62

As you see, the header line starts at line 4 and line 1-3 contains garbage with some time stamp. I tried putting the props.conf file in my universal forwarder's app, but it does not seem to be extracting the header. My props.conf looks like this.

[demozxc]
FIELD_DELIMITER = \s
FIELD_HEADER_REGEX = #Fields:\s+(.*)

May I know if there is anything I have done incorrectly? I tried putting "HEADER_FIELD_LINE_NUMBER = 4" in the props.conf, but it did not work as well.

[woodcock]

Skip the FIELD_HEADER_REGEX and use only HEADER_FIELD_LINE_NUMBER like this:

[demozxc]
FIELD_DELIMITER = \s
HEADER_FIELD_LINE_NUMBER = 4

Also, because this is an index-time configuration, you need to make sure that you deploy it to your Indexers (or to your Forwarders if you are using a Heavy Forwarder configuration). If it still doesn't work, then the only thing it can be is a mismatch of the sourcetype so triple-check that and be aware that there may be precedence problems if you are overriding it.

[normangoh]

Thank you for your reply!

I was also wondering if I should put this props.conf on my universal forwarder or my indexer? Currently I am placing this props.conf at my universal forwarder and it is not working.

[woodcock]

That is why; it has to go on your Indexers and the Splunk instance will have to be restarted on each Indexer for it to take effect.

[normangoh]

I tried putting the props.conf on my indexer but it still did not work. When I run the search there are no fields as specified in the log file header.

My code in props.conf looks like this.

[sourcetypename]
FIELD_DELIMITER = \s
HEADER_FIELD_LINE_NUMBER = 4

I have double checked and the sourcetype name is correct but still the fields are not extracted at index time. I shifted the props.conf to my indexer as well. Did I miss out anything?

[woodcock]

Did you bounce Splunk on each Indexer like I said like rhis?

$SPLUNK_HOME/bin/splunk restart