Description

Splunk for Isilon integrates the EMC Scale-out NAS Platform "Isilon" with Splunk

Splunk for Isilon

Overview

• Splunk for Isilon integrates the EMC Scale-out NAS Platform "Isilon" with Splunk
• Features:
  • Cluster Performance Dashboard with Graphs of CPU Usage, Network Usage, Disk Usage, and Protocol Usage
  • Nagios Integrated Dashboard with Graphs of Storage Usage, Quota Usage, and Connections by Protocol (Requires Splunk for Nagios)

• This is version 1.0 of Splunk for Isilon - any feedback, including requests for enhancement are most welcome. Email: luke@verypowerful.info
• This app has been created for the specifics of our Isilon environment, so it may or may not suit your specific purposes
• Copyright (c) 2011 Luke Harris. All Rights Reserved.

Screenshots

• http://verypowerful.info/home/splunkforisilonscreenshots

Setup Splunk for Isilon

Add an Index to Splunk:

• Create an index called storage then restart Splunk
  • Note: the dashboards use searches based on index = storage

Add new Data Inputs:

Here are two methods to ingest the log file from your Isilon Cluster to your Splunk indexer (chose only one method):

1. Configure a 'Universal Forwarder' on the Solutions Enabler server

• http://www.splunk.com/base/Documentation/latest/Deploy/Deployanixdfmanually
  • cd $SPLUNK_HOME/bin (eg. cd /opt/splunkforwarder/bin)
    • ./splunk start
    • ./splunk add forward-server splunk.abc.com.au:9997
    • Note: replace $LOG_HOME with the relevant directory (eg. /log/isilon)
      • ./splunk add monitor $LOG_HOME/isi-statistics.log -sourcetype isi_statistics -hostname hostname.abc.com.au
    • edit $SPLUNK_HOME/etc/apps/search/local/inputs.conf on the Solutions Enabler server and add the following key/value pair:
      • index = storage
    • restart the Splunk UF agent:
    • ./splunk restart

OR

2. Configure Isilon log file ingestion using 'rsync' on the Splunk indexer

isi-statistics.log :-

• Click Manager > Data inputs > Files & Directories > New
• Specify the source: Continuously index data from a file or directory this Splunk instance can access
• Full path to your data: eg. /log/isilon/isi-statistics.log
• Tick More settings
• Set host: constant value
• Host field value: eg. hostname.abc.com.au
• Set the source type: Manual
• Source type: isi_statistics
• Index: storage
• Click Save

Setup rsync cron job on the Splunk server:
Note: replace /log/isilon with the relevant path to your log files and replace isiloncluster with the hostname of your Isilon Cluster

*/5 * * * * rsync -q -az isiloncluster:/var/log/isi-statistics.log /log/isilon/isi-statistics.log

Isilon Cluster Configuration (REQUIRED)

Add the following cron job to /etc/crontab on *one* of your Isilon nodes:

0 0 * * * root /usr/bin/isi statistics system --nodes --running=300 -i 300 -r 288 --timestamp > /var/log/isi-statistics.log

Dashboards

Each of the following dashboards use one base search to feed all downstream panels to save search resources:

• Cluster Performance Graphs
  • Featuring graphs of CPU Usage, Network Usage, Disk Usage, and Protocol Usage for a given Cluster
    • Enter the hostname of your Isilon Cluster
• Nagios Isilon Performance Graphs (Requires Splunk for Nagios)
  • Featuring graphs of Storage Usage, Quota Usage, and Connections by Protocol for a given Cluster
    • Enter the hostname of your Isilon Cluster

Disclaimer

• This app has been created for the specifics of our Isilon environment (EMC Isilon with OneFS V6.0.2.47) and it may or may not suit your specific purposes.

License

• GNU GENERAL PUBLIC LICENSE Version 3

v1.0

• initial release