I have two sourcetypes where the thousandth of a second portion of the timestamp is not padded w/ leading zeros if the time is less than 100 thousandths of a second.
A log event is created at eight AM and eight thousandths of a second. The timestamp for this event would be... 08:00:00:8 Splunk interprets that time as... 08:00:00:800 instead of 08:00:00:008
A log event is created at nine AM and ninety thousandths of a second. The timestamp for this event would be... 09:00:00:90 Splunk interprets that time as... 09:00:00:900 instead of 09:00:00:090
This causes log entries in Splunk to be out of order when viewing a sequence of logs.
My question is how I should go about fixing the timestamp for future logs (w/o a huge burden on the indexer). And can I fix the timestamps for events that already exist in my index?
I will include some example logs for additional clarification. Note that all items (including date and time) are tab delimited.
Per Lowell's suggestions, I have tried the following permutations for the time format of my sourcetype in the local props.conf file with no improvement...
It's hard to see, but the third and seventh lines have a literal tab between the year and hour, and the fourth and eighth line have a space between the year and hour.
You can't fix the existing entries without deleting and re-indexing your events. (And this is true for any change to your event data; once splunk indexes and event, it's unchangeable)
Moving forward, you should be able get splunk to accept the correct timestamp by using an explicit
I'm not 100% sure about the tab thing, you could try putting in a literal tab, or a single space, like so:
Note: Using an explicit
This is not an "answer" but more of a list of things I've tried on my system with no workable solutions, so I'm calling on the help of those more knowledgeable gurus, like Gerald.
I confirmed that both the following timestamp format strings to do not work, just as Jeffa has reported:
Knowing that the sub-seconds is actually stored in the indexed
So I used a
Here is the config I tried:
This config actually prevents the events from being indexed at all. But if you change, the format line to
On a further test, I determined that if you modify the example data and replace the "