Why is log time is showing 5 hours ahead of system time?

[sagar06]

The server we are monitoring log files is in EDT timezone, the indexers are in utc time zone.

the problem is logs are printing timestamp in log files which are 5 hours ahead of actual system time. (Example: system time is 6PM EDT then in log file time it is showing 11PM)

now when these events are getting indexed the event time is showing 5 hours ahead of index time as it is taking system time (EDT)and converting it to UTC.

what can be done to get the correct event time? 

please help. 

[sagar06]

I have tried configuring TZ=UTC on props of universal forwarder as well as on Heavy forwarder, still it is showing event time as 5 hours ahead of index time.

[PickleRick]

Wait a second. You said that raw events contain 11:10:18 or 11:11:18 but the event's timestamp shows 6:11:40. Is it so? That would mean that the timestamp is actually not parsed fully (or not at all) from the raw event.

[sagar06]

Hi richgalloway,

We have already tried time zones, still it is showing the event time as +5 hours.

The timestamp format in log is below.

27 Jun 2023 11:10:18 abcdgdgdggdgdgdgdgdgd

27 Jun 2023 11:11:18 ababcabcbdbcbadcsadsa

The system time is showing as Tue Jun 27 06:11:40 CDT 2023

There are 5 hours difference between log time and system time due to which in splunk the event time is showing as 5 hours ahead of index time.

Please suggest how we can fix this.

[richgalloway]

Time zones can be confusing so let's make sure neither of us is confused.

The OP said files are timestamped in EDT, yet the most recent example used CDT.

If the time in the log is 11:10:18 and the system time is 06:11:40 then the events will appear to be 5 hours in the future.  That can be corrected using the TZ=UTC setting in props.conf.

Please share the props.conf settings for the sourcetype.

[sagar06]

Hi richgalloway,

In first post, there is typo, it is not EDT, it should be CDT.

Below is the props

[xx:xxxxxxxx]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)[^:]+\:\d+\:\d+
NO_BINARY_CHECK=true
TIME_FORMAT=%d %b %Y %H:%M:%S
TIME_PREFIX=^
TZ=UTC
MAX_EVENTS=10000
MAX_TIMESTAMP_LOOKAHEAD=25
TRUNCATE=99999

[richgalloway]

Thanks for sharing the props settings.  The TZ setting needs to be on the local instance that reads the source file rather than on an indexer.

[PickleRick]

Nope. Timestamp recognition is performed in the merging pipeline on the parsing component.

It's used on UF it indexed extractions are used.

[richgalloway]

Timestamp recognition is done by the parsing process, but the TZ setting comes from the UF.  It's how an indexer in one time zone can correctly process events coming from forwarders in many other time zones.  The UF is saying "hey! these times are in my time zone (PDT) not yours".

[PickleRick]

I keep forgetting about this part. But generally it's the last step in the TZ recognition:

TZ = <timezone identifier>
* The algorithm for determining the time zone for a particular event is as
  follows:
  * If the event has a timezone in its raw text (for example, UTC, -08:00),
  use that.
  * If TZ is set to a valid timezone string, use that.
  * If the event was forwarded, and the forwarder-indexer connection uses
  the version 6.0 and higher forwarding protocol, use the timezone provided
  by the forwarder.
  * Otherwise, use the timezone of the system that is running splunkd.
* Default: empty string

 So I understand it that if your HF/idx has a TZ definition for a given source, it will ignore UF-supplied one. (And that's I think the only way I used it so I keep forgetting about the last possibility I admit).

[dural_yyz]

You have the timezone specified in the indexer PROPS as UTC but you have stated the logs are actually CDT.  If you change that timezone value you should notice your log timestamp issues resolve itself.

PROPS.conf acts independent of the indexer local time, unless the setting is not explicitly set then the indexer will assume the timezones are the same at the source and the indexer.

[sagar06]

Hi Dural_yyz,

Let me explain the issue again.

UF server time is in cdt. I am using date command to check,  output of date command is Tue Jun 27 06:11:40 CDT 2023

Now on same server in logs, time is showing 5 hours ahead of CDT time. below time it is showing.

27 Jun 2023 11:10:18 abcdgdgdggdgdgdgdgdgd

27 Jun 2023 11:11:18 ababcabcbdbcbadcsadsa

Because of this time differences, event time in splunk is showing 5 hours ahead of index_time

[dural_yyz]

I believe what is occurring is that the TZ value is set via the default stanza on the UF(local server timezone) and the Indexer props value is being ignored.  Update the props.conf on the UF to include the "TZ = UTC" or "TZ = GMT" key value pair and validate results.

[sagar06]

Hi,

We have not set TZ value in default/props.conf on UF.

[dural_yyz]

Understood, the UF may be applying local logic from datetime.xml to auto interpret.  By adding the TZ key value under props for your source/sourcetype on the UF local then you interrupt the auto interpret process.

Try and let me know - I have looked over my own environment and found instances of TZ set to overcome similar issues.

[sagar06]

Hi,

The props.conf we have been using is on HF. I can see one app with datetime.xml and its props on UF, can I add the TZ properties into the props of the same app on UF? Or we need to remove DATETIME_CONFIG from UF's props?

Thanks.

[richgalloway]

DATETIME_CONFIG and TZ are two separate settings and can co-exist.  They can be in same app.

[dural_yyz]

Leave the DATETIME_CONFIG xml value as is.  Set the TZ on the PROPS.CONF that is distributed to the UF for your sourcetype where you do event breaking.

[richgalloway]

There are two common causes for this problem:

1) The time zone specifier in the event timestamp is being ignored.  To fix it, check the TIME_FORMAT settings in the relevant props.conf file.  Ensure it properly ingests the time zone.  Usually, this means having %Z on the end of the format string.  Share a sample event and we can be more specific.

2) There is no time zone specified in the event timestamp so Splunk assumes the local time zone.  In this case, the fix is to add TZ=America/New_York to the props.conf file on the monitoring system.