IIS sourcetype switching by time

Question

Hello,

We're seeing kind of a strange issue with IIS sourcetypes for two IIS servers that are forwarding logs to the same Splunk indexer. From midnight until between 2 and 6 hours later (local server time; GMT/log time is +5), the logs are showing up as sourcetype "iis". Then at some given point (2:41 a.m. on host 1, 5:40 a.m. on host 2) the sourcetype is switching to "iis-3". There's no overlap between the sourcetypes and the log entries have the same number of fields. See below:

Host 1

12/8/10 2:41:07.000 AM   2010-12-08 07:41:07 HOST1 [IP_ADDRESS_1] GET / - 80 - [IP_ADDRESS_2] - - 403 14 5 218 7 0

host=HOST1, sourcetype=iis, source=D:\webserver\logs\W3SVC1\ex101208.log

12/8/10 2:41:12.000 AM   2010-12-08 07:41:12 HOST1 [IP_ADDRESS_1] GET / - 80 - [IP_ADDRESS_2] - - 403 14 5 218 7 0

host=HOST1, sourcetype=iis-3, source=D:\webserver\logs\W3SVC1\ex101208.log

Host 2

12/8/10 5:40:24.000 AM   2010-12-08 10:40:24 HOST2 [IP_ADDRESS_1] GET / - 80 - [IP_ADDRESS_2] - - 403 14 5

host=HOST2, sourcetype=iis, source=D:\webserver\logs\W3SVC1\ex101208.log

12/8/10 5:40:29.000 AM   2010-12-08 10:40:29 HOST2 [IP_ADDRESS_1] GET / - 80 - [IP_ADDRESS_2] - - 403 14 5

host=HOST2, sourcetype=iis-3, source=D:\webserver\logs\W3SVC1\ex101208.log

Any idea what might be causing the sourcetypes to flip back and forth like that?

Accepted Answer

While Splunk's automatic sourcetyping is a convenient feature, trusting the software to make that decision can cause problems sometimes.

It sounds like Splunk's automatic sourcetyping is active, and it has decided to create different sourcetypes (perhaps something to do with seeing or not seeing the IIS header rows, especially since they can appear in the middle of the file when IIS restarts).

Your best bet is to do two things:

Explicitly assign the sourcetype as iis, via inputs.conf or props.conf.

Rename the sourcetype iis-3 to iis to make the already-indexed logs consistent.

Edit:

Since you mentioned CHECK_FOR_HEADER, definitely be aware that that CHECK_FOR_HEADER can occasionally get confused about the format. Also, by design it will create new sourcetypes if the field list does not match across all of your IIS sources.

In this case, it's safer to use a fixed field list or a regex-based field extraction, instead of using CHECK_FOR_HEADER. Here's one approach...

props.conf:

[rule::sourcetype_iis]
sourcetype=iis
MORE_THAN_75 = W3SVC

[iis]
CHECK_FOR_HEADER = False
TIME_PREFIX = :\s
MAX_TIMESTAMP_LOOKAHEAD = 128
TIME_FORMAT = %Y-%m-%d %H:%M:%S
TZ = GMT
REPORT-iisfields = iis-fields

transforms.conf:

[iis-fields]
REGEX = \d\d\d\d-\d\d-\d\d [\d:]+ (W3\S+) (\S+) (\S+) (\S+) (\S+) (\S+) (\S+) (\S+) (\S+) (\S+)? (\S+)? (\S+)?
FORMAT = cs_sitename::$1 cs_ip::$2 method::$3 uri_stem::$4 uri_query::$5 s_port::$6 user::$7 src_ip::$8 useragent::$9 statuscode::$10 substatuscode::$11 sc_win32_status::$12

Answer 2

Recommendations :

when using CHECK_FOR_HEADER = False, add it on the forwarder (UF and LWF) too, because this props is applied for the sourcetype learning (otherwise you will have iis-2 iis-3 etc...)

 #on the forwarder in props.conf
[iis]
CHECK_FOR_HEADER = False

# on the indexer in props.conf
[iis]
CHECK_FOR_HEADER = False
TIME_PREFIX = :\\s
MAX_TIMESTAMP_LOOKAHEAD = 128
TIME_FORMAT = %Y-%m-%d %H:%M:%S
TZ = GMT
 # add field extraction if needed, see bellow

Because the IIS logs have many format for the fields, instead of using the models that you find on answers, it's always good to define yourself the field extraction based on your real logs.

example :

 # my iis log header
 # date : sssss
 # FIELDS: date time c-ip cs-username s-sitename etc ....

use

props.conf
[iis]
REPORT-iisfields = iis-fields
transforms.conf
[iis_w3c]
FIELDS="date","time","c-ip","cs-username","s-sitename" etc....
DELIMS = " "

IIS sourcetype switching by time

Follow this question

Splunk Resources

Related questions