syslog messages showing the exact same _time for one network only, regardless when sent in

[loganseth]

Hi.

For about a month, Splunk was receiving syslog messages and indexing the time sent to it into the _time field correctly (the time received 'just worked'), but then on Jan 11, all syslog messages from one network (172.16.2.x) had the timestamp ALWAYS BE EXACTLY Jan 11, 2022, 09:13.

To be clear, all syslog messages from network b (172.18.100.x) continued working fine, with _time being updated to the timestamp received, but ever since that date, just syslog messages from that one network are 'stacking up' on that exact time stamp.

I started with the linux servers themselves and found they are all running ntp and have the correct time and then ran tcpdump -vvi on Splunk to see the times coming in (which manually sending syslog messages) and looks good.

I cannot figure out what is going on or where to even look next!!

[loganseth]

i added a screenshot of the data input and a working example using 'janebo' as the text.

the manual syslog command i'm using to generate from either server is:

echo "{\"test:\":\"janebo\"}" > /dev/udp/172.16.2.34/514

[PickleRick]

If you send your events like this, they don't contain any timestamp at all - just payload.

In such case the date should get generated locally by one of the components along the way (most probably the receiving forwarder) according to the mechanism described here: https://docs.splunk.com/Documentation/Splunk/latest/Data/HowSplunkextractstimestamps

But apparently this doesn't work very good in your case. There are some possibilities for this - for example, assignment of a time outside of allowed range (way too far into the past or the future). In such case the event's time would get overwritten with the time from the previous event which could result in all events ending up with the same timestamp.

You should look into _internal for errors/warnings regarding timestamps.

[loganseth]

Hi.  Thx or the _internal  logs check.  I did find some entries related to exactly what you said, but, again, to push the point, the exact same command (and actual syslog script we have running) is working on hosts on the 172.18 network.

This is the warning I see:

WARN DateParserVerbose [160796 merging] - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (128) characters of event. Defaulting to timestamp of previous event

which 100% is exactly what you said, but thoughts on why this would work for for one network and not the other? this is the frustrating part to me.

i added two new screenshots that start with 'internal logs' to show what i'm seeing from both networks (to the dropbox link)

really appreciate your leads and support, sir.

[PickleRick]

If indeed those events are received on the same indexer/forwarder's input and they are the same, it's hard to say without going into details of your config and maybe enabling debug on inputs.

[loganramirez]

i'm all ears on what configs to look at and how to debug!

[PickleRick]

Hard to say without seeing at least some of the config (inputs, props/transforms related to your syslog sourcetypes). And some samples of the events.

[loganseth]

upfront, the splunk admin left (though I have some familiarity) and my understanding is no changes were made to any config files.

there is the data input and the specific index it assigns it, too.

I cannot see how to upload any files, but here are two screenshots from splunk:
https://www.dropbox.com/sh/csd51vxt18943wk/AADCBLBWRVu68HiXUZTmvHTMa?dl=0

in the 01 file you can see the large spike on the 01/11 from the network of hosts.  this is NOT a lot of calls on that day, but is all of the calls since that day 'stacking' on that particular date.

in the 02 file you can see two different manual tests i can from the cli, one yesterday (1/25) and one today (1/26), but you can visibly see the timestamp is identical (and incorrect).

i can show this working from the other network if that helps and i'm willing to chase down any config files if you can offer some guidance on where to look!

we are not using any addons that i am aware of, just a generic_single_line udp data input.

THANK YOU!