I use several SplunkLightFirwarders on Suslog-ng servers to have a "buffer" to relatively large amounts of syslog that are then forwarded to the main indexer.
Several different corporate units need separate acces restrictions and different retention time. so i would like to put data from different host:: in separate indexes.
as far as I found in the Docunentation the Light Forwarder do not take any routing desicions. and i therefore put following config on the main indexer:
[host::192.168.9*] TRANSFORMS-routing = route_to_groupA [host::192.168.13.*] TRANSFORMS-routing = route_to_groupB
[route_to_groupA] DEST_KEY = _MetaData:Index FORMAT = index_groupA [route_to_groupB] DEST_KEY = _MetaData:Index FORMAT = index_groupB
Put the same config on the SplunkLightForwarder which made no change to my result.
Not working ...
Light Forwarders do not perform transforms. That's part of what makes them light forwarders.
Consequently, sophisticated routing cannot be accomplished on a light forwarder.
answered 17 Apr '10, 00:16
You mentioned syslog-ng. Assuming your host is the path for monitor input, you should be able to do this by setting the following on your LWF:
[monitor:///syslog-ng/path/.../logs] host_segment = 3
So if you have a path like /var/log/192.168.9.1/ events will be set to host=192.168.9.1 Without this setting, you can't use your current props configuration.
This will set the host field for your syslog-ng events and allow the props/transforms on the indexer to do their job.
answered 21 Apr '10, 04:33
Try removing the transforms stanza and modify your props with the below settings, restart afterwards:
props.conf [host::192.168.9*] index = index_GroupA [host::192.168.13.*] index = index_GroupB
answered 16 Apr '10, 12:48
Thanks for all the help here. There are obviously several ways to solve this problem . I wanted to separate log into separate indexes based on which host generated the event.
Since it was indicated to me that the host:: tag was already put in there by the LWF , i used some time to try to make that work. to no avail. ( i really miss some "sniffing" tools to see what are actually see what king of data and tags are communicated between Indexer, FW and LWF )
Since the source tag are available from the LWF My solution became this.: