Could someone provide some tips and tricks to configure Splunk for BlueCoat application?

How to properly display time, add reports, modify field extractions?

These are the customizations, which I did. I am answering my own question. Hopefully, this will help someone to configure Splunk for BlueCoat plug-in and start a larger discussion on tips and tricks for this application.

1. Configured time to be diplayed properly.

BlueCoat by default logs time in GMT format. In order for GMT time to be properly processed by Splunk configure the following:

-- Added clause:

TZ=US/Eastern

to [default] section in <splunkroot>/etc/system/local/props.conf

-- Then marked that each proxy logs in GMT format by modifying <splunkroot>/etc/apps/SplunkforBlueCoat/local/props.conf:

[bcoat_proxysg] TZ=UTC

1. Fixed incorrect log file parsing for source IP address. The original FIELDS statement in <splulnkroot>/etc/apps/SplunkforBlueCoat/default/transforms.conf:

FIELDS="date","time","time_taken","dvc_ip","user","user_group","x_exception_id","filter_result","category","http_referrer","holder","http_response","action","http_method","http_content_type","uri_scheme","dest_host","dest_ip","dest_port","uri_path","uri_query","uri_extension","http_user_agent","src_ip","sc_bytes","cs_bytes","x_virus_id"

"dvc_ip and "src_ip" fields need to switch places.

I copied transforms.conf file into ../local directory and modified FIELDS clause as following:

FIELDS="date","time","time_taken","src_ip","user","user_group","x_exception_id","filter_result","category","http_referrer","holder","http_response","action","http_method","http_content_type","uri_scheme","dest_host","dest_ip","dest_port","uri_path","uri_query","uri_extension","http_user_agent","dvc_ip","sc_bytes","cs_bytes","x_virus_id"

This fixes "Top Clients by IP" report

1. Defined names for each proxy. In order for "Requests per Proxy" graph to look right, I configured each proxy as a separate input with a different source name. In <splulnkroot>/etc/apps/SplunkforBlueCoat/local/inputs.conf:

[tcp://1.1.1.1:34000] sourcetype = bcoat_proxysg connection_host = none source = proxy1.mycompany.com

[tcp://1.1.2.1:34001] sourcetype = bcoat_proxysg connection_host = none source = proxy2.mycompany.com

[tcp://1.1.3.1:34002] sourcetype = bcoat_proxysg connection_host = none source = proxy3.mycompany.com

1. It is imperative for me to be able to correlate between firewall and BlueCoat proxy logs. Documentation for the plug-in requires that BlueCoat is configured to log in bcreportermain_v1 format. The problem with bcreportermain_v1 default format is that it does not log destination IP addresses of web traffic. It places the IP address of BlueCoat proxies, which is not really helpful.

The default bcreportermain_v1 format is:

date time time-taken c-ip cs-username cs-auth-group x-exception-id scfilter- result cs-categories cs(Referer) sc-status s-action cs-method rs(Content-Type) cs-uri-scheme cs-host cs-uri-port cs-uri-path cs-uriquery cs-uri-extension cs(User-Agent) s-ip sc-bytes cs-bytes x-virusid

Please, refer to BlueCoat documentation for the meaning of each field. According to the documentation:

s-ip -- IP address of the appliance on which the client established its connection

What I want to replace it with:

cs-ip -- IP address of the destination of the client's connection

BlueCoat proxy, however, does not allow changing bcreportermain_v1 format, since it is predefined. I have created a different format "SplunkTCP" with the following configuration:

date time time-taken c-ip cs-username cs-auth-group x-exception-id sc-filter-result cs-categories cs(Referer) sc-status s-action cs-method rs(Content-Type) cs-uri-scheme cs-host cs-uri-port cs-uri-path cs-uri-query cs-uri-extension cs(User-Agent) cs-ip sc-bytes cs-bytes x-virus-id

and configured BlueCoat proxies to send log events to Splunk in this format.

However, it broke some of the field parsing. Troubleshooting the issue, I found "holder" field in FIELDS statement in transforms.conf. That field was no longer necessary. When I removed it, everything started working fine and I got an opportunity to get statistical information based on destination IP addresses.

1. Added "Base Report - Destination IP".

Created savedsearches.conf file in <splulnkroot>/etc/apps/SplunkforBlueCoat/local/ with the following content:

[Base Reports - Destination IP] action.email.reportServerEnabled = 0 cron_schedule = 0 * * dispatch.earliest_time = -1h dispatch.latest_time = +0s displayview = flashtimeline enableSched = 1 request.ui_dispatch_view = flashtimeline search = sourcetype="bcoat_proxysg" | makemv delim=";" allowempty=false setsv=true http_content_type | makemv delim=";" allowempty=false setsv=true category | stats count by dest_ip | rename dest_ip as "Destination IP" | sort -count