Hex Time Stamp Extraction issues with DATETIME_CONFIG

Question

I have a log file from EMC where the timestamp is in hex:  timeStamp="0x4E665CB6000269D7".  I have provided a sample log snippet and all the .conf files below.  I can't get it to recognize the hex time stamp.  What am I doing wrong?

log file:

CheckEvent: Request  : <CheckEventRequest>
  <EventList count="1">
    <Event event="0x100000" path="\\omega22.w2k8r2.vee.com\CHECK$\server2fs1\test" flag="0x2" protocol="0" server="OMEGA22" share="server2fs1" clientIP="10.245.72.115" serverIP="10.245.72.209" timeStamp="0x4E665CB6000269D7" userSid="S-1-5-21-3594340890-534397530-1661201549-500" ownerSid="S-1-5-21-3594340890-534397530-1661201549-500" fileSize="0x0" desiredAccess="0x100081" createDispo="0x1" ntStatus="0x0" relativePath="\\OMEGA22\server2fs1\test"/>
  </EventList>
</CheckEventRequest>

inputs.conf

[monitor:///Users/dmaislin/Desktop/EMC/fsc.txt]
disabled = false
followTail = 0
sourcetype = emc

props.conf

[source::/Users/dmaislin/Desktop/EMC/fsc.txt]
sourcetype=emc

[emc]
TIME_PREFIX = timeStamp
MAX_TIMESTAMP_LOOKAHEAD = 9999
BREAK_ONLY_BEFORE = CheckEvent:
MUST_NOT_BREAK_BEFORE = </CheckEventRequest>
SHOULD_LINEMERGE = TRUE
DATETIME_CONFIG = /etc/system/local/emc-epoch.xml
REPORT-xmlkv = xmlkv-alternative

transforms.conf

[xmlkv-alternative]
REGEX = <([^\s\>]*)[^\>]*\>([^<]*)\<\/\1\>
FORMAT = $1::$2
MV_ADD = True

emc-epoch.xml

<datetime>
         <define name="_hexepoch" extract="hexepoch">
           <text><![CDATA[timeStamp="0x([A-Fa-f0-9]{8})]]></text>
         </define>
         <timePatterns>
           <use name="_hexepoch"/>
         </timePatterns>
         <datePatterns>
         </datePatterns>
</datetime>

Answer 1

The 1st problem I can see here is that the source-based stanza [source:///Users/dmaislin/Desktop/EMC/fsc.txt] will take precedence over the sourcetype-based [emc] stanza.

As per props.conf.spec :

**[<spec>] stanza precedence:**

For settings that are specified in multiple categories of matching [<spec>] stanzas,
[host::<host>] settings override [<sourcetype>] settings. Additionally,
[source::<source>] settings override both [host::<host>] and
[<sourcetype>] settings.

You should not have two stanzas here.

Also :

Your TIME_PREFIX should be :

TIME_PREFIX = timeStamp="

...and to reflect this, I would suggest that you change your time-stamp extraction regex in emc-epoch.xml with :

<text><![CDATA[0x([A-Fa-f0-9]{8})]]></text>

Your MAX_TIMESTAMP_LOOKAHEAD should be anchored to the end of TIME_PREFIX, not to the beginning of the event :

MAX_TIMESTAMP_LOOKAHEAD = 16

Answer 2

Here are the fixed files...

inputs.conf

[monitor:///Users/dmaislin/Desktop/EMC/fsc.txt]
disabled = false
followTail = 0
sourcetype = emc

props.conf

[source::/Users/dmaislin/Desktop/EMC/fsc.txt]
TIME_PREFIX = timeStamp="
MAX_TIMESTAMP_LOOKAHEAD = 16
BREAK_ONLY_BEFORE = CheckEvent:
MUST_NOT_BREAK_BEFORE = </CheckEventRequest>
SHOULD_LINEMERGE = TRUE
DATETIME_CONFIG = /etc/system/local/emc-epoch.xml
REPORT-xmlkv = xmlkv-alternative
sourcetype = emc

transforms.conf

[xmlkv-alternative]
REGEX = <([^\s\>]*)[^\>]*\>([^<]*)\<\/\1\>
FORMAT = $1::$2
MV_ADD = True

emc-epoch.xml

<datetime>
         <define name="_hexepoch" extract="hexepoch">
           <text><![CDATA[0x([A-Fa-f0-9]{8})]]></text>
         </define>
         <timePatterns>
           <use name="_hexepoch"/>
         </timePatterns>
         <datePatterns>
         </datePatterns>
</datetime>

Hex Time Stamp Extraction issues with DATETIME_CONFIG

Follow this question

Splunk Resources

Related questions