For the following example JSON message (formated to make it easier to read), how can I configure props.conf to inform Splunk that it should use data.timestamp for its event timestamp?
{
"publish_time": 1580824871.446,
"data": {
"textPayload": "DEBUG | 2020-02-04T14:01:05,760 | A very long string here...<snip>",
"logName": "blah0",
"receiveTimestamp": "2020-02-04T14:01:07.707699223Z",
"labels": {
"k8s-pod/version": "blah2",
"k8s-pod/track": "blah3",
"k8s-pod/app": "blah4",
"k8s-pod/pod-template-hash": "blah5"
},
"insertId": "blah6",
"resource": {
"type": "k8s_container",
"labels": {
"project_id": "blah7",
"pod_name": "blah8",
"cluster_name": "blah9",
"location": "blah10",
"container_name": "blah11",
"namespace_name": "blah12"
}
},
"severity": "INFO",
"timestamp": "2020-02-04T14:01:05.760888513Z"
},
"attributes": {
"logging.googleapis.com/timestamp": "2020-02-04T14:01:05.760888513Z"
}
}
Would the following be correct & performant ?
File: props.conf
---snip---
[google:gcp:pubsub:message]
INDEXED_EXTRACTIONS = json
KV_MODE = none
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TIMESTAMP_FIELDS = data.timestamp
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%9N%Z
---snip---
Yes configuration you posted works. Also please set KV_MODE = none and AUTO_KV_JSON = false for this source/sourcetype on search head(s) to avoid multi values in fields.
File: props.conf
[google:gcp:pubsub:message]
KV_MODE = none
AUTO_KV_JSON = false
Thanks for your response.
I tried using the following:
File: props.conf
---snip---
[google:gcp:pubsub:message]
INDEXED_EXTRACTIONS = json
KV_MODE = none
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
AUTO_KV_JSON = false
TIMESTAMP_FIELDS = data.timestamp
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%9N%Z
---snip---
and found that Splunk's _time field was not being set to the value of the data.timestamp field.
Maybe Splunk doesn't support nested JSON fields in TIMESTAMP_FIELDS ?
Currently, I've configured the props.conf file to use:
[google:gcp:pubsub:message]
INDEXED_EXTRACTIONS = json
KV_MODE = none
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TIME_PREFIX = \"timestamp\": \"
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%9N%Z
which correctly sets Splunk's _time field to the value of data.timestamp
Splunk does support nested json parsing.Please remove attribute TIME_FORMAT from your configurations and try. I am able to parse above json with below configurations.
[google:gcp:pubsub:message]
INDEXED_EXTRACTIONS = json
KV_MODE = none
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
AUTO_KV_JSON = false
TIMESTAMP_FIELDS = data.timestamp