How to connect data to CIM when the user has changed the default index and sourcetype for an Input?

[shai]

Hi,

I have a modular input that is connected to CIM through eventtypes and tags as follows:

default/eventtypes.conf

[my_event_type]
search = sourcetype=my_default_source_type

default/tags.conf

[eventtype=my_event_type]
alert = enabled

This generally works up until a user decides to reconfigure the default sourcetype and index.
When the sourcetype is altered the eventtype stanza breaks.
How to go about this?
What is the best practive to allow a user to reconfigure sourcetype while ensuring the CIM integration works. 
Should I relay on other fields that I create? 
Can this damage the efficiency of the query? 

Thanks

[richgalloway]

No matter how foolproof your configuration some clever fool will find a way to break it.

Educate your users about the importance of letting you know about changes to sourcetypes so you can make the necessary changes to configurations.

IMO, sourcetypes should not change unless the structure/format of the data changes.

As for the efficiency question, it depends.  If sourcetypes and indexes change often enough that searches must look for multiples of each then, yes, efficiency is affected.  Searching two indexes is less efficient than searching one index.

[isoutamo]

Hi
I always teach my users that when they change log format they must also rename its sourcetype. e.g. adding a number after it.
My standard is name source types like "vendor:system:log-file:incremental number starting from 0". That way it' s easy just to add 1 to this #.
r. Ismo

[shai]

Thanks for the reply! This raises up one more question.
I wonder about a specific foolproof solutionif you will:
If I define a field which always has the same unique value, e.g.

my_unique_modular_input_identifier="A(*#IKF)ApdSAODF)SIEKSD"

And  If  I use this fields in all my reports, my tags, etc, instead of using sourcetype or index vaules,
then - should this bypass the issue correctly?

besides the efficiency problem of querying multiple indexes, is there any other obvious disadvantage to this solution?

Is this a common solution for this types of problem?

Thanks!

[richgalloway]

The index and sourcetype fields have the advantage of being baked into Splunk - they're present in every event with little to no effort.  The same cannot be said for user-defined fields.  You would need to come up with a way to ensure the my_modular_input_identifier field gets unique values at index time across indexers/HFs while surviving restarts.

At the risk of redundancy, I believe sourcetypes should be permanently associated with shapes of data.  If the data shape doesn't change then the sourcetype should not change, either.  Users should not be changing the sourcetype of data without just cause.

Similarly, an index is a storage location rather than an attribute of data.  Since users likely are not familiar with the Splunk storage environment or with the access and permissions of indexes, they should not be changing the index names in inputs.  Tell them where their data goes and don't allow changes.