Refine your search:

I need to reduce our licensing usage by filtering common, valid, no-news-is-good-news domains out of our Barracuda Web Filter logs. I am trying to do this by sending such log messages to the nullQueue, but I clearly am not doing it correctly.

The set_bwf transform works, but the bwf_discard transform does not.

Given that I am a regex newbie, I suspect that I have crafted the regular expression under "[bwf_discard]" incorrectly.

Here's what I have so far:

props.conf:

[source::udp:514]
TRANSFORMS-discard=bwf_discard
TRANSFORMS-changesourcetype=set_bwf

[syslog:bwf]
REPORT-extract=bwf_extract,user

transforms.conf:

[bwf_discard]
REGEX = (?m)(commondomain1.com|commondomain2.com|commondomain3.com|commondomain4.com|commondomain5.com)
DEST_KEY = queue
FORMAT = nullQueue

[set_bwf]
REGEX = barracuda-hostname.localdomain
FORMAT = sourcetype::syslog:bwf
DEST_KEY = MetaData:Sourcetype

[bwf_extract]
DELIMS = " " 
FIELDS = bwf_month, bwf_day, bwf_time, bwf_hostname, bwf_daemon_info, bwf_timestamp, bwf_number_1, bwf_src_ip, bwf_dest_ip, bwf_content_type, bwf_src_ip2, bwf_dest_url, bwf_data_size, bwf_md5_anchor, bwf_action, bwf_reason, bwf_format_version, bwf_match_flag, bwf_TQ_flag, bwf_action_type, bwf_src_type, bwf_src_detail, bwf_dest_type, bwf_dest_detail, bwf_spy_type, bwf_spy_id, bwf_infection_score, bwf_matched_part, bwf_matched_category, bwf_user

[user]
REGEX = ([^\s]+)\s\[([\w\:]+)\]\s+$
FORMAT = category::$1 user::$2

asked 15 Oct '10, 19:47

mileserickson's gravatar image

mileserickson
21214
accept rate: 0%


2 Answers:

It looks like your regex should work, you don't need the (?m) for multi-line matching, and I would suggest making the regex slight more specific to the host field (rather than matching anywhere in the event), but that shouldn't stop it from working.

You can try this, but I suspect your issue is elsewhere.

[bwf_discard]
REGEX = ^[A-Z][a-z]{2} .\d \d\d:\d\d:\d\d (commondomain1\.com|commondomain2\.com|commondomain3\.com|commondomain4\.com|commondomain5\.com)\s
DEST_KEY = queue
FORMAT = nullQueue

If you don't have a regex testing tool, I'd highly suggested getting familiar with one. There are a number of free options out there, and it will save you time in the long run.

link

answered 16 Oct '10, 04:11

Lowell's gravatar image

Lowell ♦
11.2k91291
accept rate: 41%

-4

just have a look thomas sabo

link

answered 07 Mar '11, 05:26

fineman's gravatar image

fineman
0
accept rate: 0%

Post your answer
toggle preview

Follow this question

Log In to enable email subscriptions

RSS:

Answers

Answers + Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "Title")
  • image?![alt text](/path/img.jpg "Title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Tags:

×471
×285
×142

Asked: 15 Oct '10, 19:47

Seen: 2,155 times

Last updated: 04 Apr '11, 12:22

Copyright © 2005-2012 Splunk Inc. All rights reserved.