Description

Splunk for Symmetrix integrates the EMC Enterprise Storage Array "Symmetrix VMAX" with Splunk

Splunk for Symmetrix

Overview

• Splunk for Symmetrix integrates the EMC Enterprise Storage Array "Symmetrix VMAX" with Splunk
  • Features:
    • VMAX Storage Dashboard with Graphs of FAST VP Pool Usage, Savvol Usage, and TDEV Capacity
    • VMAX IOPS Dashboard with Graphs of IOPS by Storage Group
    • Nagios Integrated Dashboard with Graphs of IOPS, MB per second, and Write Pending (Requires Splunk for Nagios)

• This is version 1.0 of Splunk for Symmetrix - any feedback, including requests for enhancement are most welcome. Email: luke@verypowerful.info
• This app has been created for the specifics of our Symmetrix environment, so it may or may not suit your specific purposes
• Copyright (c) 2011 Luke Harris. All Rights Reserved.

Screenshots

• http://verypowerful.info/home/splunkforsymmetrixscreenshots

Setup Splunk for Symmetrix

Add an Index to Splunk:

• Create an index called storage then restart Splunk
  • Note: the dashboards use searches based on index = storage

Add new Data Inputs:

Here are two methods to ingest the symmetrix log files from your Solutions Enabler server (linux & unix supported) to your Splunk indexer (chose only one method):

1. Configure a 'Universal Forwarder' on the Solutions Enabler server

• http://www.splunk.com/base/Documentation/latest/Deploy/Deployanixdfmanually
  • cd $SPLUNK_HOME/bin (eg. cd /opt/splunkforwarder/bin)
    • ./splunk start
    • ./splunk add forward-server splunk.abc.com.au:9997
      • Note: replace $LOG_HOME with the relevant directory (eg. /log/symmetrix)
        • ./splunk add monitor $LOG_HOME/EMCcapacity.log -sourcetype symmetrix -hostname hostname.abc.com.au
        • ./splunk add monitor $LOG_HOME/symstat_0042.log -sourcetype symstat_0042 -hostname hostname.abc.com.au
          • Required: Replace 0042 with the SID of your EMC Symmetrix VMAX Array
    • edit $SPLUNK_HOME/etc/apps/search/local/inputs.conf on the Solutions Enabler server and add the following key/value pair:
      • index = storage
    • restart the Splunk UF agent:
    • ./splunk restart

OR

2. Configure symmetrix log file ingestion using 'rsync' on the Splunk indexer

a/ EMCcapacity.log :-

• Click Manager > Data inputs > Files & Directories > New
• Specify the source: Continuously index data from a file or directory this Splunk instance can access
• Full path to your data: eg. /log/symmetrix/EMCcapacity.log
• Tick More settings
• Set host: constant value
• Host field value: eg. hostname.abc.com.au
• Set the source type: Manual
• Source type: symmetrix
• Index: storage
• Click Save

b/ symstat_0042.log :-

• Click Manager > Data inputs > Files & Directories > New
• Specify the source: Continuously index data from a file or directory this Splunk instance can access
• Full path to your data: eg. /log/symmetrix/symstat_0042.log
• Tick More settings
• Set host: constant value
• Host field value: eg. hostname.abc.com.au
• Set the source type: Manual
• Source type: symstat_0042
• Index: storage
• Click Save

Setup rsync cron jobs on the Splunk server:
Note: replace /log/symmetrix with the relevant path to your log files and replace solutionsenabler with the hostname of your Solutions Enabler server

30 * * * * rsync -q -az solutionsenabler:/var/log/EMCcapacity.`date +\%F`.log /log/symmetrix/EMCcapacity.log
*/5 * * * * rsync -q -az solutionsenabler:/var/log/symstat/symstat_0042.log /log/symmetrix/symstat_0042.log
30 05 * * * rsync -q -az solutionsenabler:/var/log/EMCsg_0042.`date +\%Y-\%m-\%d`.log /opt/splunk/etc/apps/symmetrix/lookups/emcsg_0042_lookup.csv

Required: Replace 0042 with the SID of your EMC Symmetrix VMAX Array

Solutions Enabler Configuration (REQUIRED)

1. Download EMCcapacity.ksh and EMCsg.ksh from the following google code project:
   • http://code.google.com/p/check-emc-symmetrix/
2. Update EMCcapacity.ksh and EMCsg.ksh with the SID of your EMC Symmetrix VMAX arrays
3. Setup the following cron jobs on your Solutions Enabler server:

00 * * * * /usr/local/bin/EMCcapacity.ksh 1>/dev/console 2>&1
00 05 * * * /usr/local/bin/EMCsg.ksh 1>/dev/console 2>&1
0 0 * * * /usr/symcli/bin/symstat -sid 42 -type REQUEST -i 300 -c 288 > /var/log/symstat/symstat_0042.log

Note: Replace the -sid number above with the SID of your EMC Symmetrix VMAX array and also replace 0042 with your SID

Nagios Plugins supported by Splunk for Symmetrix

• check-emc-symmetrix: http://code.google.com/p/check-emc-symmetrix/
  • check_3raid5_efd.sh - Check Enterprise Flash Disk Thin Pool Usage
  • check_7raid5_fc.sh - Check Fibre Channel Thin Pool Usage
  • check_raid6_sata.sh - Check SATA Thin Pool Usage
  • check_fast_vp_state.pl - Check FAST VP State
  • check_vmax_iops.sh - Check IOPS (Read & Write)
  • check_vmax_wr_pending.sh - Check Write Pending

Dashboards

Each of the following dashboards use one base search to feed all downstream panels to save search resources:

• VMAX IOPS Graphs
  • Featuring an auto-populating drop-down list of Storage Arrays to easily display the IOPS of your Storage Groups
    • Uses lookup tables which are configured in props.conf & transforms.conf (ie. one lookup table per SID)
      • lookup tables are generated on the Solutions Enabler server by EMCsg.ksh
• VMAX Storage Graphs
  • Featuring graphs of FAST VP Pool Usage, Savvol Usage, and TDEV Capacity for a given Storage Array
    • Enter the SID of your Storage Array
• Nagios VMAX Performance Graphs (Requires Splunk for Nagios)
  • Featuring graphs of IOPS, MB per second, and Write Pending for a given Storage Array
    • Enter the hostname of your Solutions Enabler server

Disclaimer

• This app has been created for the specifics of our Symmetrix environment (EMC VMAX running at least Enginuity 5875 with Solutions Enabler V7.3.0.1) and it may or may not suit your specific purposes.

License

• GNU GENERAL PUBLIC LICENSE Version 3

v1.0

• initial release