Post search filtering

Question

Is there a way to apply a SED like filter after a search. The plumbing is there to filter and sanitize data going into the indexer. You could achieve this with a custom search command but is there another way? Would be handy if there was a props config.

Accepted Answer

You have two options. Either use the | rex mode=sed "s/find/replace/g" at runtime, or use a SEDCMD at index time, like so:

In props.conf:

[my_source_type]
SEDCMD = s/find/replace/g

Now, if if you are asking if you case setup something like a SEDCMD that gets run at search time, then the answer is no.

Here is a possible workaround: If your sed command is really long and ugly and you simply don't want to see it or don't want to repeat it in multiple searches, then I would suggest that you create a macro with your rex. I think that's the best you can do.

Answer 2

There are some tricks, but I would just stick with expressing them in the search query language (with either rex or eval for your case), using macros if that makes things clearer or easier to manage.

Note that the existing "automatic" extractions and lookups at search time are not merely syntactic replacements for rex and lookup commands. They also allow you to perform efficient searches on the new fields by reverse-mapping the requested fields to original indexed values. This would not be possible with a general sed command.

But you can make things automatic. Note that doing this will also probably make it incredibly confusing to search for items, even if you're looking right at them:

props.conf:

[mysourcetype]
REPORT-raw = changeraw
FIELDALIAS-raw = _mychangedraw AS _raw

transforms.conf:

[changeraw]
SOURCE_KEY = _raw
REGEX = ^(?<_mychangedraw>.{1,64})
CLEAN_KEYS = false

This only works however if the change you're trying to make to a field is simply slicing out one part of it. If you need to do something more complicated, you do need a custom script, but you can use a lookup to get it to run automatically. A lookup, like an extraction, won't overwrite an initially returned field, so you'd do similar to the above:

[mysourcetype]
LOOKUP-raw = mycustomlookupscript _raw OUTPUT _mychangedraw
FIELDALIAS-raw = _mychangedraw AS _raw

[mycustomlookupscript]
external_cmd = mylookupscript.py
fields_list = _raw,_mychangedraw

Well, you have to have an external script instead of using Splunk to set regexes here.

Post search filtering

Follow this question

Splunk Resources

Related questions