My Indexer is not processing "Keep specific events and discard the rest"

[GersonGarcia]

Hello all,
I have one app that generates a lot of data and it is killing my license. We need this data for sensitive customer only.
So, I followed
https://docs.splunk.com/Documentation/Splunk/6.5.2/Forwarding/Routeandfilterdatad
Keep specific events and discard the rest:
My props.conf is:

[metrics-app-gateway]
TRANSFORMS-set= setnull,setparsing

And trasnsforms.conf:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX=lasconpr01vmw08.las.ssnsgs.net
DEST_KEY = queue
FORMAT = indexQueue

But for some reason it is not working. When I comment out #TRANSFORMS-set= in props.conf, I ingested the file properly.
How can I troubleshoot this? How come it is not passing by REGEX?

Thank you
Gerson Garcia.

[chrisyounger · Accepted solution]

Hi @GersonGarcia

Does the string lasconpr01vmw08.las.ssnsgs.net exist in the _raw message? This won't match the MetaData fields

Secondly, this props and transforms are parse time (unless using indexed extractions) that means they need to go on the Indexer, or heavy forwarder if the data goes through a heavy forwarder.

Hope this helps.

[GersonGarcia]

Ah, that may be the reason. Is there any way I can use MetaData field something like:
REGEX=host::lasconpr01vmw08.las.ssnsgs.net

The props.conf and transforms.conf are in the Indexer

Thank you.

[chrisyounger]

This should work

[host::lasconpr01vmw08.las.ssnsgs.net]
TRANSFORMS-set= setparsing

[host::<other hosts>]
TRANSFORMS-set= setnull

Note you will have to set the other_hosts yourself.

[GersonGarcia]

But hold on, if I do that, I will exclude everything else (all apps and logs):

[host::<other hosts>]
TRANSFORMS-set= setnull

Right?

[chrisyounger]

Yes that's right. So you should set it very carefully. Sorry I should have made that clearer.

[GersonGarcia]

I can't do that, I just need to stop index this particular stanza [metrics-app-gateway] for all hosts except lasconpr01vmw08.las.ssnsgs.net.

[chrisyounger]

OK give this a try. Use the configs from your original message but add SOURCE_KEY like so

[setparsing]
REGEX=lasconpr01vmw08.las.ssnsgs.net
DEST_KEY = queue
SOURCE_KEY = MetaData:Host
FORMAT = indexQueue

[GersonGarcia]

It didn't work. It is not finding the REGEX. In the documentation I found:

Data:Host : The host associated with the event.
The value must be prefixed by "host::"

What does it mean?

In my inputs.conf I have:

[monitor:///usr/ssn/gateway/logs/metrics.log]
host = lasconpr01vmw08.las.ssnsgs.net
sourcetype = metrics-app-gateway
_meta = ssnservice::CONED-PROD01
index = ssn

Should I add:

[monitor:///usr/ssn/gateway/logs/metrics.log]
host = lasconpr01vmw08.las.ssnsgs.net
sourcetype = metrics-app-gateway
_meta = ssnservice::CONED-PROD01 host::lasconpr01vmw08.las.ssnsgs.net
index = ssn