Refine your search:

3
2

Conventional wisdom for collecting syslog data from external sources (network equipment, etc) was to put a couple of dedicated syslog-ng servers behind a load balancer, write the logs to a file, and have Splunk monitor the files. With Splunk 4.2 and the Universal Forwarders, does this still hold true?

We want to add some resiliency in our log collection... in most cases we can use the Splunk Universal Forwarder, but in cases where we can't, we rely on syslog. I was considering deploying a couple of dedicated VMs running Splunk Universal Forwarders behind a load balancer to grab the syslog data from this equipment. I am considering:

  1. rsyslog, write to a file, have Splunk Universal Forwarders monitor the files and send to Splunk indexers.
  2. Splunk Universal Forwarders listening on port 514/udp, forwarding to Splunk indexer (no dedicated syslog listener)

What are the pros & cons of each approach? My gut tells me that with the proper monitoring and load balancing, the Splunk Universal Forwarder could handle this job by itself.

Thanks.

asked 08 Aug '11, 13:39

jeff's gravatar image

jeff
7964421
accept rate: 27%


One Answer:

Yes, writing to files (split out by host, with at least one rotated file) is still the recommendation, and the reasons have not changed between 4.1 and 4.2. Three reasons are:

  • Performance. For several reasons, you will get better performance on the indexing side, better performance and less resource consumption on the forwarder side, and lower network utilization, if you capture and write to a file with rsyslog instead of capturing UDP packets.
  • Reliability. The files provide a buffer of data so that if there are short network or server disruptions, or even extended ones, you don't lose any data.
  • Flexibility. If you use rsyslog to split the data by host, you will be able to use [host::] stanzas to do index-time processing on the new (resolved) host names.
link

answered 08 Aug '11, 15:29

gkanapathy's gravatar image

gkanapathy ♦
32.4k4827
accept rate: 41%

As a beginner here, I have to ask about that statement on flexibility: can you give an example of such processing? Or, put in a different way, what is it you can't do, if you go with Splunk UF instead?

(12 Jun '12, 07:22) reedmohn
Post your answer
toggle preview

Follow this question

Log In to enable email subscriptions

RSS:

Answers

Answers + Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "Title")
  • image?![alt text](/path/img.jpg "Title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Tags:

×294
×278
×25

Asked: 08 Aug '11, 13:39

Seen: 2,695 times

Last updated: 12 Jun '12, 07:22

Copyright © 2005-2012 Splunk Inc. All rights reserved.