How do I configure Splunk to filter out events I don’t want to index?

Question

A lot of the Windows Security Events we see in Splunk, come from system-users that we're not interested in. I know there's a way to configure Splunk to filter out events based on the event content, but I can't get it to work.

In the example event below, I dont want to index events produced bt the

04/27/10 03:15:38 PM  
LogName=Security  
SourceName=Microsoft Windows security auditing.  
EventCode=4634  
EventType=0  
Type=Information  
ComputerName=my.testmachine.env  
TaskCategory=Logoff  
OpCode=Info  
RecordNumber=12345678  
Keywords=Audit Success  
Message=An account was logged off.

Subject:  
Security ID:            L-O-O-K-Y  
Account Name:           MAN-PAGE-H3LP$  
Account Domain:         TEST  
Logon ID:               0x123456789  
Logon Type:                     7

My current configuration is -

inputs.conf:  
[WinEventLog:Security]  
disabled = 0  

props.conf:  
[source::WinEventLog:Security]  
TRANSFORMS-Event=MachineNamePurge  
blacklist = Account\sName:\s+\w+\$

Accepted Answer

I think you're just a bit unclear around the terminology & concepts in play here, we only implement 'blacklists' at the file level, not the event level. So you can blacklist entire files or directories using an entry in inputs.conf, but for individual events it's a bit more involved.

This would be more clear if you were familiar with the 3.x version of Splunk, back then it took both a props.conf entry AND a transforms.conf entry to extract a field from an event. With the current version you only need to do this if you are configuring an index-time field-extraction, which is basically what you're trying to achieve here.

In the config below, we have your regular inputs.conf entry -

Inputs.conf 
[WinEventLog:Security] 
disabled = 0

As you can see, nothing relating to blacklisting here. Next comes the props.conf entry -

Props.conf 
[source::WinEventLog:Security] 
TRANSFORMS-nullQ= nullFilter

and then the transforms.conf entry is where the actual REGEX and actions are defined -

Transforms.conf 
[nullFilter] 
REGEX = Account\sName:[\s|\w|-]+\$ 
DEST_KEY=queue 
FORMAT = nullQueue

So in the 3.x world, if you had a setting in props.conf that started with 'REPORT-'. that meant it was a search-time field-extraction. If it started with 'TRANSFORMS-', that meant it was an index-time extraction. In this case, we want the action to run at index time, so we call it using TRANSFORMS.

In the transforms.conf entry, we apply the REGEX to every single event covered by the props.conf spec - WinEventLog:Security - so this is an expensive filtering method from a CPU perspective. If the REGEX matches, then the event is routed to the 'nullQueue', i.e. it's thrown away and never makes it into the index. This only affects events that match the REGEX, if it's not a match, then the event continues to the index as normal.

All of the configuration steps around this is documented here

How do I configure Splunk to filter out events I don’t want to index?

Follow this question

Splunk Resources

Related questions