Splunk for Unix and Linux

currently running debian 6.0.6 uname -a Linux dev 2.6.32-5-xen-amd64 #1 SMP Sun Sep 23 13:49:30 UTC 2012 x86_64 GNU/Linux

I should note that I have atsar installed and not just sar $ aptitude show atsar Package: atsar
New: yes State: installed Automatically installed: no Version: 1.7-2 Priority: optional Section: admin Maintainer: Michael Stone mstone@debian.org Uncompressed Size: 184 k Depends: libc6 (>= 2.3.5-1) Recommends: cron Description: system activity reporter Monitor system resources such as CPU, network, memory & disk I/O, and record data for later analysis

noticed the cpu.sh output wasn't working

here are the changes I made to get things working

$ diff -uNr  cpu.org.sh cpu.new.sh 
--- cpu.org.sh  2012-10-10 18:39:05.000000000 -0700
+++ cpu.new.sh  2012-10-10 18:38:18.000000000 -0700
@@ -1,4 +1,5 @@
 #!/bin/sh                                                                                                
+#set -x
 # Copyright 2009-2011 Splunk, Inc.                                                                       
 #                                                                                                        
 #   Licensed under the Apache License, Version 2.0 (the "License");                                      
@@ -20,11 +21,17 @@
 PRINTF='{printf "%-3s  %9s  %9s  %9s  %9s  %9s\n", cpu, pctUser, pctNice, pctSystem, pctIowait, pctIdle}'

if [ "x$KERNEL" = "xLinux" ] ; then
+   queryHaveCommand atsar
+   FOUND_ATSAR=$?
    queryHaveCommand sar
    FOUND_SAR=$?
    queryHaveCommand mpstat
    FOUND_MPSTAT=$?
-    if [ $FOUND_SAR -eq 0 ] ; then
+    if [ $FOUND_ATSAR -eq 0 ] ; then
+       CMD='sar -S -u 1 1'
+       #18:08:13  cpu %usr %nice   %sys %irq %softirq    %wait %idle 
+       FORMAT='{cpu=$2; pctUser=$3; pctNice=$4; pctSystem=$5; pctIowait=$8; pctIdle=$NF}'
+    elif [ $FOUND_SAR -eq 0 ] ; then
        CMD='sar -P ALL 1 1'
        FORMAT='{cpu=$3; pctUser=$4; pctNice=$5; pctSystem=$6; pctIowait=$7; pctIdle=$NF}'
    elif [ $FOUND_MPSTAT -eq 0 ] ; then

*NIX 4.6 on Linux: Not sure if this counts as a bug but nonetheless a small error.

The "Percent Load by Host" box in the "CPU by Host" view shows the result of this search:

index=os sourcetype=cpu host=* | multikv fields pctIdle | eval Percent_CPU_Load = 100 - pctIdle | timechart avg(Percent_CPU_Load) by host

However, the sourcetype cpu data (from cpu.sh) contains an "all" line, such as below:

CPU pctUser pctNice pctSystem pctIowait pctIdle all 9.27 0.00 15.54 0.75 74.44 0 0.00 0.00 1.01 3.03 95.96 1 38.00 0.00 58.00 0.00 4.00 2 0.00 0.00 1.98 0.00 98.02 3 0.00 0.00 0.00 0.00 100.00

The Linux manual for "mpstat" says:

CPU Processor number. The keyword all indicates that statistics are calculated as averages among all processors.

Which means that Splunk is including an calculated average in doing "timechart avg(Percent_CPU_Load) by host".

[EDIT]: This seem to not happen on a particular SuSE box. I noticed the small difference in the output from "sar -P ALL 1 1" on two different Linux boxes:

SuSE SLES 11.1: 10:11:40 CPU %user %nice %system %iowait %steal %idle 10:11:41 all 2.72 0.00 0.24 0.00 0.00 97.04 10:11:41 0 5.94 0.00 1.98 0.00 0.00 92.08 10:11:41 1 0.00 0.00 0.00 0.00 0.00 100.00 10:11:41 2 0.00 0.00 0.00 0.00 0.00 100.00 10:11:41 3 0.00 0.00 0.00 0.00 0.00 100.00

Ubuntu 12.04: 10:22:11 AM CPU %user %nice %system %iowait %steal %idle 10:22:12 AM all 0.25 0.00 0.00 0.00 0.00 99.75 10:22:12 AM 0 0.00 0.00 0.00 0.00 0.00 100.00 10:22:12 AM 1 0.00 0.00 0.00 0.00 0.00 100.00 10:22:12 AM 2 0.00 0.00 0.00 0.00 0.00 100.00 10:22:12 AM 3 0.00 0.00 0.00 0.00 0.00 100.00

the "AM" part seems to be what messes up the cpu.sh script.

Running version 4.5 on Splunk 4.3, many of the charts in this app are still Flash-based. It would be nice to see these converted to HTML5-based charts.

Scripts run slow, and reports are super-slow, cpu-bounded instead of disk-bounded.

For example, CPU usage per process report, last 7 days, one machine, takes minutes to complete in a 4-core CPU machine, 4 GB RAM, linux.

In rought numbers, extract, parse, and sum numbers of ~10k events produced by the ps scripted input took about 7 minutes (given ps.sh is executed every minute 60x24x7=10080).

Using vmstat,iostats and top shows splunkd is eating two cores, but there is very few disk activity.

any hint to diagnose why is this slow?

/opt/splunk/etc/apps/unix/bin/interfaces.sh contains a typo in line number 27:

CMD='ifconfig'.

causes "command not found errors", see also message from tikums, 09 Nov 11. Removing the trailiing dot will fix the problem.

CMD='ifconfig'

Running Ubuntu 10.04.03 LTS on Citrix Xenserver and Hyper-V. Neither of these provides speed or duplex settings within dmesg. It's not a matter of changing the pattern, the information, just isn't there.

Ethtool only displays a link status, so perhaps this information is not available for the virtual network drivers?

The fixes above for awk fixed the syntax errors, so that is working nicely now. Thanks @wfroning

Just like others have mentioned, their are a couple bugs in the interfaces.sh script. Besides the trailing period on CMD the awk lines for getting the speed and duplex of the interface needed some fix up on my ubuntu server.

--- interfaces.sh_orig  2011-12-06 12:00:59.000000000 +0400
+++ interfaces.sh   2011-12-06 12:01:48.000000000 +0400
@@ -24,7 +24,7 @@
    assertHaveCommand dmesg

    CMD_LIST_INTERFACES="eval ifconfig | tee $TEE_DEST | grep 'Link encap:Ethernet' | tee -a $TEE_DEST | cut -d' ' -f1 | tee -a $TEE_DEST"
-   CMD='ifconfig'.
+   CMD='ifconfig'
    GET_MAC='{NR == 1 && mac = $5}'
    GET_IPv4='{if ($0 ~ /inet addr:/) {split($2, a, ":"); IPv4 = a[2]}}'
    GET_IPv6='{$0 ~ /inet6 addr:/ && IPv6 = $3}'
@@ -38,8 +38,8 @@
    for iface in `$CMD_LIST_INTERFACES`
    do
        # ethtool(8) would be preferred, but requires root privs; so we use dmesg(8), whose [a] source can be cleared, and [b] output format varies (so we have less confidence in parsing)
-       SPEED=`dmesg | awk '/[Ll]ink( is | )[Uu]p/ && /'$iface'/ {for (i=1; i<=NF; ++i) {if (match($i, /([0-9]+)([Mm]bps)/, array)) {print array[1] "Mb/s"} else { if (match($i,/[Mm]bps/)) {print $(i-1) "Mb/s"} } } }' | sed '$!d'`
-       DUPLEX=`dmesg | awk '/[Ll]ink( is | )[Uu]p/ && /'$iface'/ {for (i=1; i<=NF; ++i) {if (match($i, /([\-\_a-zA-Z0-9]+)([Dd]uplex)/, array)) {print array[1]} else { if (match($i, /[Dd]uplex/)) {print $(i-1)} } } }' | sed 's/[-_]//g; $!d'`
+       SPEED=`dmesg | awk '/[Ll]ink( is | )[Uu]p/ && /'$iface'/ {for (i=1; i<=NF; ++i) {if (match($i, /([0-9]+)([Mm]bps)/)) {pattern = substr($i,RSTART,RLENGTH); print pattern "Mb/s"} else { if (match($i,/[Mm]bps/)) {print $(i-1) "Mb/s"} } } }' | sed '$!d'`
+       DUPLEX=`dmesg | awk '/[Ll]ink( is | )[Uu]p/ && /'$iface'/ {for (i=1; i<=NF; ++i) {if (match($i, /([\-\_a-zA-Z0-9]+)([Dd]uplex)/)) {pattern = substr($i,RSTART,RLENGTH); print pattern} else { if (match($i, /[Dd]uplex/)) {print $(i-1)} } } }' | sed 's/[-_]//g; $!d'`
        $CMD $iface | tee -a $TEE_DEST | awk "$GET_ALL $FILL_BLANKS $PRINTF" name=$iface speed=$SPEED duplex=$DUPLEX
        echo "Cmd = [$CMD $iface];     | awk '$GET_ALL $FILL_BLANKS $PRINTF' name=$iface speed=$SPEED duplex=$DUPLEX" >> $TEE_DEST
    done

FYI, the new versions of the app are NOT compatible with old versions of splunk (like 4.1.*) and will cause splunkweb to fail. Please use the old versions of the app, or upgrade your splunk.

On Centos 6:

11-09-2011 09:15:31.878 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/unix/bin/interfaces.sh" /opt/splunk/etc/apps/unix/bin/interfaces.sh: line 43: ifconfig.: command not found

Feature Request: Please expanded rlog.sh to cover Solaris 10 auditd created logs. They are binary and can be viewed with praudit -l (1 line per record) or praudit -r (raw). Otherwise, tool looks very cool.

Glad to see support for Splunk 4.x, and also that top.sh has the equivalent Solaris 10 command, prstat.

Brian Jester