How do i exclude some events from being indexed by Splunk?

Question

i have a data source that is very noisy, and i only want to index specific events from it, not all of them. for example, i only want to index logins and logouts, or login failures. how do i do this?

Accepted Answer

12

This is done by defining a regex to match the necessary event(s) and send everything else to nullqueue

Here is a basic example that will drop everything except events that contain the string login In props.conf:

[source::/var/log/foo]
# Transforms must be applied in this order
# to make sure events are dropped on the
# floor prior to making their way to the
# index processor
TRANSFORMS-set= setnull,setparsing

In transforms.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = login
DEST_KEY = queue
FORMAT = indexQueue

link

answered 15 Jan '10, 00:01

matt ♦♦
3.5k●12●11●40
accept rate: 81%

edited 16 Apr '10, 18:41

2

I prefer to use the regex (?=) to match anything. The regex . won't match if the field you're checking against happens to be empty, for example. Though this doesn't happen with the default _raw field, other fields can be empty.

(15 Jan '10, 01:10) gkanapathy ♦

Answer 2

5

This example only includes things that contain 'login' and drops everthing else. Another use case would be to take in everything and make an exception for nosie you want filtered out.

The inverse to accept all except anything with the word 'info' would require just one stanza in transforms.conf:

[setnull]
REGEX = info
DEST_KEY = queue
FORMAT = nullQueue

link

answered 08 Apr '10, 19:56

dskillman ♦
537●1●4●13
accept rate: 29%

Answer 3

See this post:

http://answers.splunk.com/questions/96/how-do-i-exclude-some-events-from-being-indexed-by-splunk

=========

This is done by defining a regex to match the necessary event(s) and send everything else to nullqueue

Here is a basic example that will drop everything except events that contain the string login

In props.conf:

[source::/var/log/foo]
# Transforms must be applied in this order
# to make sure events are dropped on the
# floor prior to making their way to the
# index processor
TRANSFORMS-set= setnull,setparsing

In transforms.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = login
DEST_KEY = queue
FORMAT = indexQueue

Answer 4

I ran into the same pitfall but I don't know any way out of it. I need to discard certain noisy syslog events and keep the rest.

C:Program FilesSplunketcsystemlocalprops.conf

[cisco_asa]
TRANSFORMS-null= setnull

I also tried :

[source::udp:2000]
TRANSFORMS-null= setnull

I use the Cisco Security Suite App, thus the syslog port moved to 2000 for coexistence with the standard syslog service.

C:Program FilesSplunketcsystemlocaltransforms.conf

[setnull]
REGEX = (ASA-6-302014|ASA-6-302013|ASA-6-302016|ASA-6-302015)
DEST_KEY = queue
FORMAT = nullQueue

Do you have an idea for further troubleshooting?

Cheers, Philipp

How do i exclude some events from being indexed by Splunk?

Follow this question

Splunk Resources

Related questions