Pad thousandths of a second in timestamp with leading zeros

You can't fix the existing entries without deleting and re-indexing your events. (And this is true for any change to your event data; once splunk indexes and event, it's unchangeable)

Moving forward, you should be able get splunk to accept the correct timestamp by using an explicit TIME_FORMAT entry in your props.conf file. I believe that using the "%nN" format entry will force splunk to assume a certain number of digits, in this case 3. So you would want something like:

[your_sourcetype]
TIME_FORMAT = %b %d, %Y\t%H:%M:%S:%3N

I'm not 100% sure about the tab thing, you could try putting in a literal tab, or a single space, like so:

TIME_FORMAT = %b %d, %Y %H:%M:%S:%3N

Note: Using an explicit TIME_FORMAT setting should make indexing faster that making splunk guess as to where you timestamp is and what format its in. I use an explicit entry like this for nearly all my sourcetype. (It also helps out if you have random timestamps scattered throughout your logs that you don't want splunk to use.)

Documentation:

• Configure timestamp recognition

This is not an "answer" but more of a list of things I've tried on my system with no workable solutions, so I'm calling on the help of those more knowledgeable gurus, like Gerald.

I confirmed that both the following timestamp format strings to do not work, just as Jeffa has reported:

TIME_FORMAT = %b %d, %Y\t%H:%M:%S:%3N
TIME_FORMAT = %b %d, %Y\t%H:%M:%S:%Q

Knowing that the sub-seconds is actually stored in the indexed _subsecond field, I decided to try a more exotic workaround by populating the field explicitly:

So I used a TIME_FORMAT with no sub-seconds, then used a transformer to modify the _raw event to add in one or two leading 0s (so it is properly padded to 3 digits), and then used a final transformer to extract out the value of _subsecond, however doing this seems to result in splunk completely dropping the event all together. I think this may be because splunk is still extracting a subseconds value even when no %3N or %Q is actually used in the TIME_FORMAT string. So therefore, I suspect that my attempt to set _subsecond manually is infact leading to two _subsecond values which I'm guessing is what splunk doesn't like and is therefore dropping it (This is just a theory)

Here is the config I tried:

props.conf:

[source::.../timestamp_test.log]
sourcetype = timestamp_test

[timestamp_test]
TIME_FORMAT = %b %d, %Y\t%H:%M:%S
SHOULD_LINEMERGE = False
TRANSFORMS-subseconds = fixup_subsecond, fixup_subsecond, extract_subsecond

transforms.conf:

[fixup_subsecond]
REGEX = ^(\S+\s+\w{3} \d+,\s+\d{4}\s+\d\d:\d\d:\d\d):(\d\d?\s.*)$
FORMAT = $1:0$2
DEST_KEY = _raw

[extract_subsecond]
REGEX = ^\S+\s+\w{3} \d+,\s+\d{4}\s+\d\d:\d\d:\d\d:(\d\d\d)\s
FORMAT = _subsecond::$
WRITE_META = True

This config actually prevents the events from being indexed at all. But if you change, the format line to FORMAT = the_sub_second::$1 (for example) then the event does get indexed but obviously it still doesn't have the correct sub-second timestamp.

On a further test, I determined that if you modify the example data and replace the ":" in the timestamp between seconds and subseconds with a space (" "), and then adjust the all the regexes and formats appropriately, then the above extract-your-own-subseconds-hack acutally does work. Of course, this still doesn't work, because if reformatting the input were an option, the obvious thing to do would be to pad the sub-seconds properly. So that's it, I'm out of ideas.