|
In Splunk 4.0, how can I extract fields from W3C Extended Log format files? (W3C Extended Log format is the standard log file format used by Windows' IIS web server). Sure, I can manually create fields in props.conf or inside my search query using regexes, but that's fragile if I start logging different fields on each site or server. The field names are sitting on top of the IIS log file. Can Splunk detect those fields like it does with CSV files? |
|
Yeah, just put these lines in etc\apps\search\local\props.conf That is one of many possible locations, but I'm keeping it simple. It is extremely useful, and probably one of the first things about configuring Splunk that should be learned, to really read and understand the configuration file priority and merging and rules: http://www.splunk.com/base/Documentation/latest/Admin/Aboutconfigurationfiles and also I strongly recommend: http://www.splunk.com/base/Documentation/latest/Admin/Troubleshootingconfigurations This is pretty much the foundation of how to manage Splunk, how to deploy it, and how to develop apps in a structured way. |
|
Minor tweak: do not edit a file in a 'default' directory, as it will be overwritten on upgrade. Instead, modify Splunk/etc/system/local/props.conf to set this value: [iis] CHECK_FOR_HEADER = true Of course, if iis always has headers, this setting should be pushed upstream to be there by default. |
|
As I found out via some experimentation, Splunk can automatically extract fields from IIS log files but you'll need to: First, edit \Program Files\SplunkBeta\etc\system\default\props.conf to change this: to this: (Note the last line) Second, when you import the log files or set up the directory monitor, manually specify the "iis" sourcetype (you can choose this from the dropdown list or type it in). If you've already imported the files without doing both of those steps, I've not figured out how to get this to work. So be sure to do the config change before you try to imoprt the files! Another caveat: you're not really supposed to be editing the system props.conf directly. If someone more knowledgeable about splunk wants to comment with the proper placed to edit, I'll revise this answer. |
|
we do not recommend you edit files in the default directory because any changes you make there would be undone during an upgrade. The recommended way of doing this is to edit the files in the local directory as follows: SPLUNK_HOME/etc/system/local/props.conf |
|
FYI : the best guide to fix IIS sourcetype and have the correct field extractions http://splunk-base.splunk.com/answers/1902/iis-and-exchange-log-header-extraction |
|
@stjack, The lightweight forwarder will simply be attaching a few pieces of metadata along to the indexer(s). That said, I got this to work when it was a single machine doing the indexing and searching, but for a distributed instance the header field extraction only seemed to work once and a while, and on smaller subsets of data. I know CHECK_FOR_HEADER is kind of an older, less focused on feature. Does it not always work right with distributed search? |
