Central Syslog-NG server, extra headers, and field extraction

[edgustaf]

We run a central Syslog-NG server, which all the logs for the servers and devices we care about get sent to. We use this to send a copy of this log stream to various different log analyzers, including Splunk. When Splunk gets this data however, it appends the time it got the log and the host it came from, the syslog server, to each log. This seems to screw up Splunk's syslog field extraction.

Searching around, I found this built-in transform, which I added to my props.conf like this: (we're using port 515 here, since 514 is being used for something else at the moment)

[source::udp:515]
TRANSFORMS-stripheader=syslog-header-stripper-ts-host

That works, except the fields seem to have been extracted before the transform gets applied in the process. That is, the host= still shows the syslog server as the host, even though all mention of it was removed from the event. What's the best way to correct this?

To make the matter even more complicated, the incoming stream has things other than syslog data in it. Our windows domain controllers, web proxy, etc log there as well. Do I need to have these log to different inputs, or will Splunk handle this on its own? I imagine this is easier to do once Splunk recognizes different hosts, though.

Thanks,
- Eric

[Stephen_Sorkin · Accepted solution]

This is because of the order of operations for extractions. The default configuration in props.conf for syslog looks like:

[syslog]
pulldown_type = true 
maxDist = 3
TIME_FORMAT = %b %d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 32
TRANSFORMS = syslog-host
REPORT-syslog = syslog-extractions
SHOULD_LINEMERGE = False

Since TRANSFORMS are applied alphabetically, syslog-host is applied before your stripheader transform. You can modify yours as follows:

[source::udp:515]
TRANSFORMS = syslog-header-stripper-ts-host syslog-host 

This will override the default TRANSFORMS and perform the stripping before the host extraction.

[edgustaf]

I got it to work by adding a label (if that's the right term for it) to the transforms like this:

[source::udp:515]
TRANSFORMS-syslog = syslog-header-stripper-ts-host syslog-host
Not sure why it didn't work the other way, but it's all fixed now! Thanks!

[Stephen_Sorkin]

You could try to set TRANSSFORMS to be the empty string and set the regexes individually. It should, however, work to have them all set there.

[edgustaf]

Hmm now with my input entry set like you said, it runs neither transform.

[source::udp:515]
TRANSFORMS = syslog-header-stripper-ts-host syslog-host

Ideas?
Thanks again,
- Eric