Splunk for Unix and Linux

Running version 4.5 on Splunk 4.3, many of the charts in this app are still Flash-based. It would be nice to see these converted to HTML5-based charts.

Scripts run slow, and reports are super-slow, cpu-bounded instead of disk-bounded.

For example, CPU usage per process report, last 7 days, one machine, takes minutes to complete in a 4-core CPU machine, 4 GB RAM, linux.

In rought numbers, extract, parse, and sum numbers of ~10k events produced by the ps scripted input took about 7 minutes (given ps.sh is executed every minute 60x24x7=10080).

Using vmstat,iostats and top shows splunkd is eating two cores, but there is very few disk activity.

any hint to diagnose why is this slow?

On the ps.sh script, is there a reason the start_time was not added to the output? It would be most helpful to know the start time of a process.

/opt/splunk/etc/apps/unix/bin/interfaces.sh contains a typo in line number 27:

CMD='ifconfig'.

causes "command not found errors", see also message from tikums, 09 Nov 11. Removing the trailiing dot will fix the problem.

CMD='ifconfig'

Running Ubuntu 10.04.03 LTS on Citrix Xenserver and Hyper-V. Neither of these provides speed or duplex settings within dmesg. It's not a matter of changing the pattern, the information, just isn't there.

Ethtool only displays a link status, so perhaps this information is not available for the virtual network drivers?

The fixes above for awk fixed the syntax errors, so that is working nicely now. Thanks @wfroning

Can you guys or this app monitor user activities with in Linux environments and the commands that they used ??

Just like others have mentioned, their are a couple bugs in the interfaces.sh script. Besides the trailing period on CMD the awk lines for getting the speed and duplex of the interface needed some fix up on my ubuntu server.

--- interfaces.sh_orig  2011-12-06 12:00:59.000000000 +0400
+++ interfaces.sh   2011-12-06 12:01:48.000000000 +0400
@@ -24,7 +24,7 @@
    assertHaveCommand dmesg

    CMD_LIST_INTERFACES="eval ifconfig | tee $TEE_DEST | grep 'Link encap:Ethernet' | tee -a $TEE_DEST | cut -d' ' -f1 | tee -a $TEE_DEST"
-   CMD='ifconfig'.
+   CMD='ifconfig'
    GET_MAC='{NR == 1 && mac = $5}'
    GET_IPv4='{if ($0 ~ /inet addr:/) {split($2, a, ":"); IPv4 = a[2]}}'
    GET_IPv6='{$0 ~ /inet6 addr:/ && IPv6 = $3}'
@@ -38,8 +38,8 @@
    for iface in `$CMD_LIST_INTERFACES`
    do
        # ethtool(8) would be preferred, but requires root privs; so we use dmesg(8), whose [a] source can be cleared, and [b] output format varies (so we have less confidence in parsing)
-       SPEED=`dmesg | awk '/[Ll]ink( is | )[Uu]p/ && /'$iface'/ {for (i=1; i<=NF; ++i) {if (match($i, /([0-9]+)([Mm]bps)/, array)) {print array[1] "Mb/s"} else { if (match($i,/[Mm]bps/)) {print $(i-1) "Mb/s"} } } }' | sed '$!d'`
-       DUPLEX=`dmesg | awk '/[Ll]ink( is | )[Uu]p/ && /'$iface'/ {for (i=1; i<=NF; ++i) {if (match($i, /([\-\_a-zA-Z0-9]+)([Dd]uplex)/, array)) {print array[1]} else { if (match($i, /[Dd]uplex/)) {print $(i-1)} } } }' | sed 's/[-_]//g; $!d'`
+       SPEED=`dmesg | awk '/[Ll]ink( is | )[Uu]p/ && /'$iface'/ {for (i=1; i<=NF; ++i) {if (match($i, /([0-9]+)([Mm]bps)/)) {pattern = substr($i,RSTART,RLENGTH); print pattern "Mb/s"} else { if (match($i,/[Mm]bps/)) {print $(i-1) "Mb/s"} } } }' | sed '$!d'`
+       DUPLEX=`dmesg | awk '/[Ll]ink( is | )[Uu]p/ && /'$iface'/ {for (i=1; i<=NF; ++i) {if (match($i, /([\-\_a-zA-Z0-9]+)([Dd]uplex)/)) {pattern = substr($i,RSTART,RLENGTH); print pattern} else { if (match($i, /[Dd]uplex/)) {print $(i-1)} } } }' | sed 's/[-_]//g; $!d'`
        $CMD $iface | tee -a $TEE_DEST | awk "$GET_ALL $FILL_BLANKS $PRINTF" name=$iface speed=$SPEED duplex=$DUPLEX
        echo "Cmd = [$CMD $iface];     | awk '$GET_ALL $FILL_BLANKS $PRINTF' name=$iface speed=$SPEED duplex=$DUPLEX" >> $TEE_DEST
    done

FYI, the new versions of the app are NOT compatible with old versions of splunk (like 4.1.*) and will cause splunkweb to fail. Please use the old versions of the app, or upgrade your splunk.

On Centos 6:

11-09-2011 09:15:31.878 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/unix/bin/interfaces.sh" /opt/splunk/etc/apps/unix/bin/interfaces.sh: line 43: ifconfig.: command not found

Feature Request: Please expanded rlog.sh to cover Solaris 10 auditd created logs. They are binary and can be viewed with praudit -l (1 line per record) or praudit -r (raw). Otherwise, tool looks very cool.

Glad to see support for Splunk 4.x, and also that top.sh has the equivalent Solaris 10 command, prstat.

Brian Jester