forward data to a syslog server

[acalvo]

We're trying to forward data to a syslog server from a splunk server. However, seems that the hostname and process id tags are missing.

We've set up a central splunk server (10.10.40.9), and another splunk server to forward some data to a syslog server (10.10.40.10).

Configurations used: outputs.conf:

[tcpout]
defaultGroup = 10.10.40.9_9997

[tcpout:10.10.40.9_9997]
server = 10.10.40.9:9997

[tcpout-server://10.10.40.9:9997]

[syslog:localhost_10514]
server = 127.0.0.1:10514
type = tcp

inputs.conf:

[monitor:///var/log]
disabled = false

props.conf:

[host::*]
TRANSFORMS-routing=SYSLOG_FWD

transforms.conf:

[SYSLOG_FWD]
REGEX = .
DEST_KEY = _SYSLOG_ROUTING
FORMAT = localhost_10514

example output messages (forwarded splunk output to syslog):

2010-10-29T17:13:05.557347+02:00 10-29-2010 17: 13:04.924 INFO  Metrics - group=per_source_thruput, series="/opt/splunk/var/log/splunk/splunkd.log", kbps=2.094271, eps=20.466667, kb=62.828125
2010-10-29T17:13:05.557347+02:00 10-29-2010 17: 13:04.924 INFO  Metrics - group=per_source_thruput, series="/opt/splunk/var/log/splunk/splunkd_stderr.log", kbps=0.000684, eps=0.033333, kb=0.020508

example desired output:

2010-10-23T08:29:08.087021+02:00 VCS02 glassfish-log: [#|2010-10-23T08:29:00.584+0200|INFO|glassfish3.0.1|null|_ThreadID=29;_ThreadName=Thread-1;|Total number of available updates : 0|#]
2010-10-24T01:38:48.570726+02:00 VCS02 ntpd[14162]: synchronized to 77.226.252.14, stratum 2
2010-10-24T04:02:08.609446+02:00 VCS02 rsyslogd: [origin software="rsyslogd" swVersion="5.6.0" x-pid="26020" x-info="http://www.rsyslog.com"] rsyslogd was HUPed

[hexx]

In order to receive RFC3164-compliant syslog output from Splunk (and namely, to include a time stamp in the sent event), you need to make sure to adequately set the timestampformat configuration key in the [syslog:] stanza in outputs.conf, as per outputs.conf.spec:

timestampformat = <format>
* If specified, the formatted timestamps are added to the start of events forwarded to syslog.
* As above, this logic is only applied when the data is not syslog, or the syslogSourceType.
* The format is a strftime-style timestamp formatting string. This is the same implementation used in the 'eval' search command, splunk logging, and other places in splunkd.
    *  For example: %b %e %H:%M:%S
    * %b - Abbreviated month name (Jan, Feb, ...)
    * %e - Day of month
    * %H - Hour
    * %M - Minute
    * %s - Second
* For a more exhaustive list of the formatting specifiers, refer to the online documentation.
* Note that the string is not quoted.
* Defaults to unset, which means that no timestamp will be inserted into the front of events.

Test configuration (tested with Splunk 4.3 only😞

• outputs.conf:

[syslog:syslog_out]
server = syslog.splunk.com:514
type = tcp
timestampformat = %b %e %H:%M:%S

• props.conf:

[syslog_test]
TRANSFORMS-routing = syslog_routing

• transforms.conf:

[syslog_routing]
REGEX = .
DEST_KEY = _SYSLOG_ROUTING
FORMAT = syslog_out

With this configuration, all events with sourcetype "syslog_test" will be routed to syslog host "syslog.splunk.com" on port TCP/514. The default priority code of "13" will be used, which is equivalent to "user.info", and a time stamp will be added as a prefix.

Example:

• On the server hosting the Splunk instance:

[root@beefysup01 43]# cat ../sources/test.log
This is a sample syslog event
[root@beefysup01 43]# $SPLUNK_HOME/bin/splunk add oneshot ../sources/test.log -sourcetype syslog_test
Oneshot '/home/octavio/sources/test.log' added

• On the server recipient of the syslog forwarded messages, using netcat to receive the events:

[root@syslog.splunk.com:/]# nc -kl 514
<13>Jan 25 19:52:07 beefysup01 This is a sample syslog event

Note: I was unable to make this configuration work in versions earlier to Splunk 4.3. There are reports that this specific feature can fail in certain scenarios in 4.2.x. If you need this feature to work, please upgrade to Splunk 4.3.

[nisse]

See also similar: http://splunk-base.splunk.com/answers/28438/no-time-or-host-in-forwarded-syslog-messages

[calyope7]

Hello acalvo,

Did you find a resolution to this issue?

Thanks, Chris

[acalvo]

The message should look like a standard syslog message (while I was testing I've used the same log that Splunk produces to send it to the syslog server):

$DATE $HOSTNAME $PROCESS[$PID] $MESSAGE

[thepocketwade]

Have you considered putting the syslog server in front of the Splunk server?

[ftk]

What do you want the output to look like?