Why is _time in my JSON payload parsed correctly with milliseconds via manual upload but not via HEC?

[andrewtrobec]

Hello!

I have a JSON payload whose _time field gets parsed no issue when I perform a manual upload, but when that same payload comes in through a HEC with the same sourcetype then it doesn't parse the milliseconds.

Sample payload:

{"flowid":"dc59cf7376370faadfb89764e1896a1b","id":23431,"action":"upload","request":"","response":"{\"success\":false,\"correlation_id\":\"00-dc59cf7376370faadfb89764e1896a1b-d23cff8d675709d1-01\",\"status_code\":\"401\",\"message\":\"Request unsuccessful. The following errors were found.\",\"errors\":[{\"code\":\"E_TECHNICAL\",\"value\":\"A technical error prevented the success of the request.\"}]}","midid":"","dest":"","type":"GET","requesttime":"2023-07-12T10:17:32.4327504Z","externaltime": null,"externalresponsetime": null,"middlewaretime":"2023-07-12T10:17:32.4327504Z","logtime":"2023-07-12T10:17:32.6039773","globaltime":"2023-07-12T10:17:32.4333085","responsetime":"2023-07-12T10:17:32.4364843"}

Sourcetype:

[json_sourcetype]
SHOULD_LINEMERGE = false
TIME_PREFIX = \"logtime\"\:\"
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%6N
TZ = UTC
TRUNCATE = 0
MAX_TIMESTAMP_LOOKAHEAD = 0
KV_MODE = json

Has anybody faced this issue before? What could the problem be?

Thank you and best regards,

Andrew

[PickleRick]

Splunk skips several stages of event processing for HEC inputs. Most notably, line breaking but also (by default) timestamp recognition.

https://docs.splunk.com/Documentation/Splunk/9.1.0/RESTREF/RESTinput#services.2Fcollector.2Fevent

[andrewtrobec]

Thank you for the response!  I have this same setup with a very similar payload and it works no issues.  Also, when the data comes into the HEC it parses the whole timestamp with the exception of the milliseconds for 

the timestamp.  So for example the _time for the example above is:

manual upload: 2023-07-12 10:17:32.432
HEC: 2023-07-12 10:17:32.000

One thing about the timestamp format is that it has 7 decimal places and not 6, but I am parsing %6N.  I'm not sure if this would have any bearing.

I hope this helps!

[PickleRick]

OK. Which endpoint are you sending your data to:

/services/collector/raw

/services/collector/event

/services/collector/event?auto_extract_timestamp=true

[andrewtrobec]

I need to find that out, but thank you for bringing this up!  Am I to assume that it would be best to send to /services/collector/raw?  Thanks!

[PickleRick · Accepted solution]

Not necessarily. The /raw endpoint works as if you sent the events over plain TCP connection with the difference that you can send events to specific channels tied to different sourcetypes and destination indexes.

The /event endpoint skips some parts of event processing (which are assumed to have been done earlier on the sending side) and thus is lighter on the receiver's performance.

So there are several possible approaches:

1. Send to the /raw endpoint and do all the parsing yourself in Splunk

2. Send to the /event?auto_extract_timestamp=true and do the timestamp parsing yourself in Splunk

3. Parse the time on your source before sending to HEC and include the properly formatted time field along with your event contents.

I'd opt for the last option but your circumstances might - for example - not allow reformatting the event on send.

[andrewtrobec]

Apologies for the delayed response, I needed time to explore and test.  In the end the issue was that the source was sending to /events endpoint and not /raw.  After updating everything starting working correctly!  Thank you so much for the support!

I am accepting this as the answer because it provides multiple approaches, of which redirecting to /raw worked.