Refine your search:

I have noticed that universal forwarders receiving data from a high-traffic input will fail to distribute events evenly across all indexers in the autoLB group defined in outputs.conf.

What is the reason for this? Can anything be done to prevent it?

asked 30 Nov '11, 19:11

hexx's gravatar image

hexx ♦
7.6k1941
accept rate: 51%


One Answer:

In order to address this issue and spread the data evenly, use a regular (heavy) forwarder to collect the data and parse it before sending it to the indexer.

With the Universal Forwarder, minimal parsing is performed on the forwarder side before sending the data onwards. This means that the UF has no idea where line-breaks occur between events, so in order to use auto-LB, it has to wait until there's a break in the data-stream before switching the output connection to a new indexer. The same behaviour would be observed if it was monitoring a file, and the logging application never stopped writing to that file. As long as data from a specific source keeps appearing fast enough, the UF will continue to send that data to a single indexer in order to avoid corruption of the index.

A regular forwarder will parse the data fully parsed before being sending it to the indexers, making it easy to identify points where the connection can be switched. Note that using this instance will increase the resource usage on the host server, so if that box is running critical applications, we should advise using a separate, dedicated box for this purpose.

link

answered 30 Nov '11, 21:57

Mick's gravatar image

Mick ♦
4.0k1327
accept rate: 52%

Hi Mick,

Thanks for this answer we are having this exact problem, that about just under 100gigs a day. Here is one of our daily reports.

host sum(mb) splunk-w1-inf53 7209.3280268010 splunk1-d1-inf 24543.7608717865 splunk2-d2-inf 17171.5732630553 splunk2-w2-inf5 9376.1935319420 splunk3-d1-inf 16665.1996564898 splunk4-d2-inf 24236.0354538779

As you can see splunk1-d1-inf and splunk4-d2-inf got 24 gigs compared to the others which got 7-16 gigs

Seems rather randomly we get big bursts of data on some indexers compared to others.

(20 Jan, 14:03) sonicZ
Post your answer
toggle preview

Follow this question

Log In to enable email subscriptions

RSS:

Answers

Answers + Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "Title")
  • image?![alt text](/path/img.jpg "Title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Tags:

×156
×77
×6
×4

Asked: 30 Nov '11, 19:11

Seen: 455 times

Last updated: 20 Jan, 14:03

Copyright © 2005-2012 Splunk, Inc. All rights reserved.