Sending certain logs from UDP port 514 to specific indexes

Unfortunately, as of Splunk 4.3 you cannot have more than one functional stanza/input per UDP port. The devil is in the details of inputs.conf.spec where this limitation is made implicit by the following statement :

[udp://<remote server>:<port>] (...) * If <remote server> is specified, the specified port will only accept data from that server.

It's important to understand that this refers to the specified port, not to the specified data input.

This will be further clarified in our next maintenance release by the addition of the following statement to inputs.conf.spec :

* Only one stanza per port number is currently supported.

If you have configured more than one UDP input stanza using the same port but specifying different hosts of origin, pushing the UDPInputProcessor log channel to DEBUG level will typically yield messages like this one :

02-03-2011 11:19:07.167 DEBUG UDPInputProcessor - The UDP packet data from '10.1.5.139' was refused only accepting data from '10.1.8.120'

This means that in your case, you have to use transformations to route your UDP data to different indexes, as per Lisa's 2nd suggestion.

A few considerations I would add :

A) If you are using a [host::] stanza in props.conf to conditionally apply your routing, I would suggest to use something like this to add resilience to RDNS failures:

[host::(my.hostname.com|10.1.8.120)]

B) I personally am not too crazy about host-based filtering in the stanza name of props.conf. I would rather do something like this:

B.1) In props.conf, use a sourcetype-based stanza to apply a generic routing transformation to the data coming from the UDP input:

[syslog] TRANSFORMS-idx_routing = generic_idx_routing

B.2) In transforms.conf, use a regular expression on SOURCE_KEY = MetaData:Host to route the events to a different index, whose name will be dynamically based on the value of "host":

[generic_idx_routing] SOURCE_KEY = MetaData:Host REGEX = (device1|device2|device3)\.domain\.com DEST_KEY = _MetaData:Index FORMAT = $1

In this example, events with a value of "device1.domain.com" for the "host" field will be sent to the "device1" index, while "host=device2.domain.com" and "host=device3.domain.com" will respectively go to indexes "device2" and "device3".
Note : This implies that you will have created the indexes necessary for the routing before-hand, based on the hostnames of the devices sending data to UDP:514. If you configure a new device to send data to that port, make sure that you create the corresponding index on the indexers first.

C) Finally, an alternative to dynamic index-routing would be to use several transformations with hard-coded REGEX and FORMAT values, like so:

C.1) In props.conf, use a sourcetype-based stanza to apply several routing transformation to the data coming from the UDP input:

[syslog] TRANSFORMS-idx_routing = device1_idx_routing, device2_idx_routing

C.2) In transforms.conf, use a regular expression on SOURCE_KEY = MetaData:Host to route the events to a different index, whose name will be dynamically based on the value of "host":

[device1_idx_routing] SOURCE_KEY = MetaData:Host REGEX = device1\.domain\.com DEST_KEY = _MetaData:Index FORMAT = device1
[device2_idx_routing] SOURCE_KEY = MetaData:Host REGEX = device2\.domain\.com DEST_KEY = _MetaData:Index FORMAT = device2

This would be particularly useful if you want to use index names that are not included in the sending device's host name or in the event's raw data.

Note: In both scenarios, you might prefer to use the event raw data as the field on which to apply the REGEX rather than "host", in which case you would set SOURCE_KEY = _raw and modify your REGEX accordingly to dynamically extract the name of the target index from the event data.