Hello fellow Splunkers.
I am trying to set the sourcetype name using a part of the source path. I've read the answers from the same question on the community, but i just cant get it working, so ill give it a shot and ask here.
The goal:
Set sourcetype name from the third folder in the source path.
For example, automaticly set sourcetype to "foobar" from logs collected from source C:\data\Logs\foobar\logfile.log
What i have tried:
(on the universal forwarder)
inputs.conf:
[monitor://C:\data\Logs\...\*]
index = main
(on the indexer receiving from the universal forwarder)
props.conf:
[source::C:\data\Logs\*]
TRANSFORMS-changesourcetype = changesourcetype
transforms.conf:
[changesourcetype]
SOURCE_KEY = MetaData:Source
REGEX = C:\\data\\[^\\]+\\([^\\]+)\\
#REGEX = C:\\data\\Logs\\(\w+)\\ (Commented out: for debugging purposes)
FORMAT = sourcetype::$1
DEST_KEY = MetaData:Sourcetype
I've also tried (for debugging-reasons) solving it differently, by tagging a temporary sourcetype at the UFW:
inputs.conf:
[monitor://C:\data\Logs\...\*]
index = main
sourcetype = changemeplease
props.conf:
[changemeplease]
TRANSFORMS-changesourcetype = changesourcetype
transforms.conf:
[changesourcetype]
SOURCE_KEY = MetaData:Source
REGEX = C:\\data\\[^\\]+\\([^\\]+)\\
#REGEX = C:\\data\\Logs\\(\w+)\\ (Commented out: for debugging purposes)
FORMAT = sourcetype::$1
DEST_KEY = MetaData:Sourcetype
The data gets forwarded and indexed, but the transforms seem to not hit, what so ever.
Any suggestions on what i am doing wrong here?
Ok, so i found the problem. I took a look at the transforms from the web gui afterwards, and the error was clear:
So the config files is not correctly readed by Splunk. Its like it doesnt register the "new line" afte the regex key value. Does anyone have any idea what causes this?
(editing it from the web gui, made a simular file in the apps local folder, and fixed the problem).
Config file:
Thanks for the reply.
I am pretty sure it has nothing to do with the regex.
For debugging purposes I tried using a static value instead of an regex, but the transforms still doesnt rewrite the sourcetype:
transforms.conf
[changesourcetype]
SOURCE_KEY = MetaData:Source
FORMAT = sourcetype::test
DEST_KEY = MetaData:Sourcetype
One thing i am unsure of:
Does this only work if the data passes trough an heavy forwarder before it reaches the indexer?
I am trying to achieve this on a single node Splunk Enterprise instance.
Data must go trough that HF to apply these props and transforms. If you have these on separate HF and source data is on separate node then it didn't work. You must send events to that HF to apply these configurations to it.
Also you must remember that
[<spec>] * This stanza enables properties for a given <spec>. * A props.conf file can contain multiple stanzas for any number of different <spec>. * Follow this stanza name with any number of the following setting/value pairs, as appropriate for what you want to do. * If you do not set a setting for a given <spec>, the default is used. <spec> can be: 1. <sourcetype>, the source type of an event. 2. host::<host>, where <host> is the host, or host-matching pattern, for an event. 3. source::<source>, where <source> is the source, or source-matching pattern, for an event. 4. rule::<rulename>, where <rulename> is a unique name of a source type classification rule. 5. delayedrule::<rulename>, where <rulename> is a unique name of a delayed source type classification rule. These are only considered as a last resort before generating a new source type based on the source seen. **[<spec>] stanza precedence:** For settings that are specified in multiple categories of matching [<spec>] stanzas, [host::<host>] settings override [<sourcetype>] settings. Additionally, [source::<source>] settings override both [host::<host>] and [<sourcetype>] settings.
This means that precedence for those configurations are above.
Uhm.. I do not have a separate HF.
I only have a single Splunk Enterprise instance:
Splunk Universal Forwarder -> Splunk Enterprise
Are you saying that i need another instance of Splunk Enterprise, configured as a HF, to achieve this?
Splunk Universal Forwarder -> Splunk Enterprise (HF) -> Splunk Enterprise (SH and Indexer)
UF -> Indexer / all in one instance is enough, no need for separate HF. Usually you should send all events directly from UF to indexers. Just install those configuration on indexer (Your splunk enter. server).
Are you sure that you haven't both source, sourcetype and/or host configuration on props.conf? Only one of those can be effective and the selection between those are done by precedence (see my previous post).
Yes, i am sure i do not have two configurations at the same time in props.conf.
But i've tried with both [source::<source>] and [sourcetype] at different times, just to try to figure out why this props.conf does not "hit", because that seems to be the problem.
To summary what ive tried regarding props.conf:
* Set a temporary sourcetype on the UniversalForwarder, and target that sourcetype in props.conf
* Do not set any sourcetype on the UniversalForwarder, and instead target the source in props.conf
[changemeplease]
TRANSFORMS-changesourcetype = changesourcetype
[source::C:\data\Logs\*]
TRANSFORMS-changesourcetype = changesourcetype
On props.conf is example
When you specify Windows-based file paths as part of a [source::<source>] stanza, you must escape any backslashes contained within the specified file path. Example: [source::c:\\path_to\\file.txt]
and you have
[source::C:\data\Logs\*]
I suppose that you must use \\ instead of \ on source:: stanza.
I suppose that the issue is wrong amount of escape characters?
You should try it with
REGEX = C:\\\data\\\[^\\\]+\\\([^\\\]+)\\\\
Or if not needed drop last 4 \ away.
You could test it with
| makeresults
| eval source="C:\data\Logs\foobar\logfile.log"
| rex field=source "C:\\\data\\\[^\\\]+\\\(?<BAR>[^\\\]+)\\\\"
| table source BAR
The rule of thumb with rex and \ is "just add one more \ and check if it work" 😉
r. Ismo
Ok, so i found the problem. I took a look at the transforms from the web gui afterwards, and the error was clear:
So the config files is not correctly readed by Splunk. Its like it doesnt register the "new line" afte the regex key value. Does anyone have any idea what causes this?
(editing it from the web gui, made a simular file in the apps local folder, and fixed the problem).
Config file: