How do I add metadata to events coming from a Splunk forwarder?

Question

We have Splunk running on a server at each of our locations. The NIC on the server has two IPs, one is a unique IP for communications back to our headquarters, the second IP is the same on all servers and is used to communicate with the cash registers running Snare agents to forward their windows events.

The way that the application that manages the terminal works, every location has a Terminal1, Terminal2, and Terminal3. The hostname and IP addresses are the same at every location. So, when a terminal generates a Windows security log message, we can't tell which location it originated at since the hostname/IP are the same everywhere.

How can I add some metadata to the UDP input so that we can uniquely identify each terminal?

Accepted Answer

If I am understanding correctly, each Splunk instance gets data that looks the same, and when you use distributed search, the results from each place can't be distinguished? If that's so, all you would need to do is display (or search on) the field splunk_server to distinguish the events at each location.

Update:

So in fact the Splunk instances are forwarders, not indexers. Are they light or heavy? This can be done, but how will be different depending. With a light forwarder, you can just add a key to the input stanza and use a transform on the indexer to write the key to a field. With a heavy forwarder, the transform occurs on the forwarder. You will not not be able to use environment variables in the key value field, unfortunately. This ability is targets for version 4.0.11 however.

inputs:

[udp://1030]
location = mylocation
# in some future release, you should be able to specify location = $HOSTNAME

props:

[source::udp:1030]
TRANSFORMS-location = addlocation

transforms:

[addlocation]
SOURCE_KEY = location
REGEX = (.*)
FORMAT = location::$1
WRITE_META = true

Will add an indexed field location to your data.

Accepted Answer

3

On the forwarder, you can specify a new metadata field in inputs.conf using the '_meta' setting. For example -

[WinEventLog:Security]
disabled = 0
_meta = Terminal::1

This will added to all events coming in from this input source, and will appear as an indexed field on your indexing instance

link

answered 15 Apr '10, 21:33

Mick ♦
4.0k●1●3●27
accept rate: 52%

edited 16 Jun '11, 17:41

Genti ♦
3.5k●1●22

I believe he's not using a forwarder, but sending via Snare UDP syslog

(15 Apr '10, 23:34) gkanapathy ♦

This works like a charm!!!!

(16 Jun '11, 17:53) Genti ♦

Answer 3

The servers in the stores are forwarding to a central splunk server. They listen on UDP 1030 for incoming syslog messages from snare agents running on the POS terminals. The terminals in each store all use the same names (term1, term2, term3).

The forwarders aren't indexing events. They are just forwarding. So, when we look at the splunk_server field, they all appear as the central splunk server.

The documentation says you can't add metadata to UDP ports. Since we need to push this change out to 500+ stores, I'd like to use the deployment server. If we can add metadata to UDP ports, can we use a variable like:

_meta = $HOSTNAME

Thx.

Answer 4

0

They are heavy forwarders, but we could turn them into light forwarders if this will solve the problem.

We're not using distributed search, just a single central splunk server.

How do I implement this on a light/heavy forwarder?

link

answered 16 Apr '10, 17:57

jambajuice
461●14
accept rate: 9%

Answer 5

0	Is the syntax for the props.conf file correct: [source::udp:1030] rather than [source:udp:1030] or [source::udp::1030] link answered 10 May '10, 18:38 jambajuice 461●14 accept rate: 9%

How do I add metadata to events coming from a Splunk forwarder?

Follow this question

Splunk Resources

Related questions