I see some useful info in _internal under the fwdinfo
sourcetype, fwd
source. However, I can't figure out where this data is coming from. It's not in any of the conf files, on forwarder or indexer.
Edit: Annoyingly, this data doesn't use the hostname set by the Splunk instance - but always the reverse DNS - which is not always the same!
It turns out this is a bug, and should not be ending up in _internal.
fwdinfo is some data generated by 4.2+ forwarders describing the state of their forwarding and communication with the indexers they are connected to. The forwarding code itself inserts these descriptive data items into the datastream bound for the receiving indexer.
The use of rdns to define the hostname is actually what happens for any events which arrive at a tcp input without a hostname provided. This applies for tcp input as well as splunktcp. However for splunktcp, the arriving data nearly always provides a hostname to use, so this behavior is not apparent.
It seems, currently (4.2.3 is current as of this writing), that this will happen if the "v3" protocol, which is 4.2+ is in use, without any available configurability.
Edit: the fwdinfo events were not ever intended to reach the index, but instead intended to provide data so that metrics.log would contain some information about connected forwarders.
Changes:
fwdinfo is some data generated by 4.2+ forwarders describing the state of their forwarding and communication with the indexers they are connected to. The forwarding code itself inserts these descriptive data items into the datastream bound for the receiving indexer.
The use of rdns to define the hostname is actually what happens for any events which arrive at a tcp input without a hostname provided. This applies for tcp input as well as splunktcp. However for splunktcp, the arriving data nearly always provides a hostname to use, so this behavior is not apparent.
It seems, currently (4.2.3 is current as of this writing), that this will happen if the "v3" protocol, which is 4.2+ is in use, without any available configurability.
Edit: the fwdinfo events were not ever intended to reach the index, but instead intended to provide data so that metrics.log would contain some information about connected forwarders.
Changes:
This internal sourcetype is based on the forwarders host/hostname.