How to send same logs to multiple indexes in different indexers?

[jsingh03]

Hi Everyone,

The Requirement is to send same logs to Multiple indexers.

Index name at both the indexes should be different.

Scenario is to send Logs from UF to Indx1 (logs indexed in index1) and same logs from UF has to be ingested in an intermediate forwarder (HF) and then indexed in an indexer (in an index named index2).

I am able to send the logs to both the indexers if the name of the index is same but when I am changing the index name Splunk is not able to send the logs to one of the indexes.

Thanks for the help!!

[gcusello]

Hi @jsingh03,

I suppose that you know that in this way, you pay twice the license for the duplicated logs and that you don't want to use an Indexer Cluster for data replication.

Anyway, you can use a fork in your UFs following the instructions at https://docs.splunk.com/Documentation/Splunk/9.1.0/Forwarding/Routeandfilterdatad#Perform_selective_...

This should be applieds on UFs' outputs.conf file.

Then to rename indexes (it isn't clear why you want to rename indexes!), you should put in the first Splunk full instance (an Indexer or an Heavy Forwarder if present) the props.conf and transforms.conf that you can find at https://community.splunk.com/t5/Getting-Data-In/How-can-I-override-an-index-name-based-on-sourcetype...

My hint is to stop for a while and redesign your data architecture, especially index structure and names.

Ciao.

Giuseppe

[jsingh03]

Hi @gcusello ,

Thanks for answering.

So, the requirement is that same index should not be present in both the indexers by that I meant that if I want to index the data in index1 in first indexer then the data must be in index2 in another indexer. 

We don't want same index name on both indexers. So, I guess (I am not sure) to override index we need to have the index first where the data would be ingested and then we override index so that it takes the metadata of the previous index.

Requirement is to send logs to first indexer with index name index1 and send same logs to another indexer in index2 and we don't want index1 at all on the second indexer.

Thanks

[PickleRick]

Wait a second.

Back up a little.

The main question here is - what problem are you trying to solve here? Because you're trying to do something technically but what is the business need here?

[jsingh03]

Hi @PickleRick ,

Thanks for Replying.

Organization is currently sending logs to say to Indexer1 which has indexes with name say abc_index. Now they have a completely separate Splunk environment as well where they have indexer2 which has index name say xyz_index. They want to send the logs to xyz_index as well and that by having an intermediate forwarder (HF) between UF and indexer2 (So from UF>HF>Indx2).

Now they want to dual feed the logs to indexer1 (in index abc_index) and indexer2 (in index xyz_index).

Eventually they going to get rid of indexer1 but for now they want dual feed.

I guess it is more close to this problem.

Can I override an index name at indexer? - Splunk Community

The only difference is in that case logs are splitting from HF and in my case it is from UF.

you have replied there as well but finding it hard to implement. 

Thanks

[gcusello]

Hi @jsingh03,

as I said, Splunk index isn't a database table, is a container for data.

You can recognize if an index belongs to an indexer or to another using the splunk_server field.

In addition, as I said, think that duplicating logs you pay twice the Splunk License.

Maybe it could be better to have an Indexer Cluster that automatically duplicates logs and you don pay twice them.

For this reason the correct approach is the one from @PickleRick : what are your business requirement? in other words, what's the process that you have in place?

Ciao.

Giuseppe

[PickleRick]

You're still describing technicalities while I'm asking about the business itch you're trying to scratch. Business need is - for example - "we need data repl9cated in multiple datacenters" or "we'll be phasing out equipment which is EoL-ed" and so on. That's _why_ you're trying to do something, not _what_ you're doing.

[gcusello]

Hi @jsingh03,

as I said, there's no reason to do this because a Splunk index isn't a database table that you need to rename.

This is a frequent error for all of us that arrive to Splunk from databases (I did this error some years ago!).

Usually indexes are created when there are data with different retention time and different access grants, in other words, you could put in the same index firewall logs and application logs if they must be accessed by the same groups of users and they have the same retention time.

The most important field is the sourcetype that usually define the tipe of the logs and all the parsing rules.

On the contrary, we usually try to use the same index name as much as possible just to simplify the management operations.

Anyway, I described how to do the log duplication (that requires a double payment of the license!) and the index override for one of te Indexers.

Ciao.

Giuseppe