Splunk 5.0.1 Clustered Indexes and Duplicate Data

[dturner83]

I have the following Splunk build below.

I have a replication factor of 3 and search factor of 2.
Just using 1 search head at the moment, splunksearch1, which is the master node. It distributes appropriately to splunkindex1, 2, and 3 but I get duplicate data back.

So I have a forwarder there at the bottom, it forwards data to splunkforward1 and splunkforward2, which in turn send to splunkindex1-3. When searching I get the results from all 3 with the same timestamp and exact same data so I'm assuming it's returning all the data. According to the documentation Clustering is supposed to only return the primary data, but I'm unsure how to check/troubleshoot farther than that.

Anyone got any ideas?

Splunk Environment

Update: Instead of having both forwarders forward to all 3 indexers I made them point at just 1. This has fixed the issue of seeing the data duplicated through the searches. But this seems less than ideal. If the indexer which is receiving the data goes down a change needs to be made to change the destination indexer.

[dturner83]

I modified both heavy forwarders configs to this:
[tcpout:autolbgroup1]
server = 192.168.101.22:9997,192.168.101.23:9997,192.168.101.33:9997
autoLB = true
useACK = true

[tcpout]
defaultGroup = autolbgroup1
disabled = 0

the key appears to be autoLB = true. I previously understood that it was always true but didn't appear so. Anyway setting this to true fixes the entire problem. I'm assuming it was sending all indexers all copies of the data and they all thought they were new primary copies and then returning those results. Now it is all working properly.