[udp://:portnumber] Event Blacklist- How do I prevent unwanted data from being indexed?

[LinghGroove]

Hello,

I am currently receiving firewall data on my heavy forwarder on a specific port number. On the HF there is an simple inputs.conf with

[udp://:portnumber]
sourcetype=fgt_log
index=fw_data

and an outputs.conf that sends all to the indexers. The problem is that i am receiving a lot of garbage traffic (like DNS traffic to 8.8.8.8 or 8.8.4.4). I don't want to index this data. I don't have access to the firewall so i can't just stop it there. 

I thought that a blacklist would stop the events from coming. (I tried a simple blacklist like the ones used under the [monitor] stanza. Something like this: blacklist=(dstip=8\.8\.8\.8|dstip=8\.8\.4\.4|service="DNS"))

Unfortunately it didn't work... I made some research but i only found the "acceptFrom" that in this situation i don't think it's useful. Came across this post but wasn't useful https://community.splunk.com/t5/Getting-Data-In/Blacklist-a-host-hosts-is-sending-logs-to-Splunk-via...

Any tips? 

[maciep]

There used to be a really good .conf presentation on regex, but I guess Splunk decided to remove all of the old .conf content from their site....such a shame.  Glad you were able to make it work!

[maciep]

I am not sure about the udp input stanza and whether you can blacklist there or not.  But a common approach is to use props/transforms to filter matching events to the "nullQueue" so that they are not ingested.  That is something that can be done on the heavy forwarder.

You can find details in the docs about it but also probably quite a few posts out here detailing the steps...
Route and filter data - Splunk Documentation

[LinghGroove]

Hello @maciep thanks for your reply,

I have tried the solution you are suggesting... unfortunatly i am not able to make it work... I just now tried the solution from https://community.splunk.com/t5/Getting-Data-In/How-could-i-filter-network-firewall-data-using-a-fil...

The problem presented in this question is nearly identical. I tried to apply it just as they answered but i doesn't work. Maybe i'm setting something wrong or maybe i am putting a wrong regex. If i'll make it work i will write you back here.

Thanks again

[LinghGroove · Accepted solution]

@maciep Ok after some test i figured out how to make it work. 

PROPS

[fgt_log]                          #here you have to put the sourcetype of your data, not the source like the documentation tells you
TRANSFORMS-null = transnull        #reference to the transform stanza for the nullqueue   

here the syntax is TRANSFORMS-<class> and for what i have understood the <class>name can be anything you want (except some special characters, if I'm not mistaken), same for the stanza name in the transform.

TRANSFORMS

[transnull]
REGEX = (dstip=(?:8\.8\.8\.8|8\.8\.4\.4)[^D]+D(?:NS)?(?:[^D\n]+DNS)?)
DEST_KEY = queue
FORMAT = nullQueue

 Here i had a bit of trouble with the regex syntax but in the end i made it work. DEST_KEY like the documentation says, same for FORMAT. The name of the stanza is the same called in the props.conf.

INPUTS

[udp://:myport]
sourcetype = fgt_log
source = ***
index = ***

 With this it works all perfectly. Thaks again for the support @maciep 🙂