Splunk for JMX

Description

SPLUNK for JMX can be used to poll local or remote JMX Management Servers running in Java Virtual Machines across your entire infrastructure and index MBean attributes and outputs from MBean operations.

Welcome to SPLUNK 4 JMX v1.5.3
by Damien T. Dallimore
April 2012
damien@dtdsoftware.com

OVERVIEW

This SPLUNK app provides the means to :

• connect to any local or remote JVM's JMX server, either via the remote JMX interface,attaching to a local JVM process or using an MX4J HTTP based connector.
• query any MBean running on that server
• extract any MBean attributes (simple, composite or tabular)
• invoke MBean operations
• write attributes and operation results out in a default key/value format, or plugin your own custom format, for SPLUNK indexing and searching
• transport events over STD out(default), TCP, Syslog, Splunk REST endpoint or direct to file.
• declare clusters of JVM's for large scale JVM deployments
• runs on *Nix and Windows

Tested to date against Hotspot ,JRockit and IBM J9 JVM's.

Currently supports the following JMX Connectors :

• rmi (JSR160 Standard Implementation and MX4J's JSR160 Implementation)
• iiop (JSR160 Standard Implementation and MX4J's JSR160 Implementation)
• local (MX4J only)
• soap (MX4J only)
• hessian (MX4J only)
• burlap (MX4J only)

This app is basically a custom scripted input comprising of a bash/bat script and a Java program. I have bundled it up as an app for convenience
and included some demo dashboards that graph up a selection of MBean attributes from the java.lang domain

Also refer to the application's "docs" directory for :

• PDF User Guide
• Javadoc API for creating a custom Formatter or Transport implementation
• Configuration schema documentation

INSTALLATION

• set the JAVA_HOME environment variable to the location of your JRE install location(version 1.6 +)
• set the SPLUNK_HOME environment variable to the install root of SPLUNK
• uncompress SPLUNK4JMX-1.5.3.tar.gz to SPLUNK_HOME/etc/apps
• restart SPLUNK

The SPLUNK4JMX app sets up a custom index(index=jms) and a custom script input (sourcetype=jmx) that parses and executes an XML config file
By default, the input script is disabled and the schedule time is 60 seconds
After you configure your config XML file, you should then enable the input script for the platform you are running on.
There is a "sh" script for Nix and a "bat" script for Windows.

INSTALLED COMPONENTS

The only components you really need to know about are in the SPLUNK_HOME/etc/apps/SPLUNK4JMX/bin directory

• bin/poll_jmx - this is the script that SPLUNK fires at scheduled intervals, this script invokes a Java program that has the logic for querying the JMX Server(s).
• bin/poll_mx4j_jmx - same as the above script, however the MX4J JMX implementation will be used.You must use this script should you want to use the MX4J specific JMX connectors.
• bin/config/config.xml - this is the config file where you specify JMX server(s), MBeans and MBean attributes/operations.You can create as many config files,with any name, as you want and set them up as the argument to the poll_jmx script on the SPLUNK Manager Data Inputs page.

CONFIG.XML

Note : also refer to the configuration schema documentation in the docs directory

Included in the SPLUNK4JMX app distribution is a comprehensive sample config.xml
This sample file queries various java.lang MBeans and obtains attributes
You can use this as a basis for creating your own config files for whatever MBeans and attributes/operations you need to poll across your environment
To try out this sample config.xml file , just set the values for your environment in the jmxserver element , and you should be good to go.

"jmxpoller" is the root xml element

A custom formatter can be specified.Only 1 may be specified for the entire configuration.
This is a Java implementation of the "com.dtdsoftware.splunk.formatter.Formatter" interface.
The formatter class must be placed on the java classpath.
If the optional formatter declaration is omitted, then the default formatter will be used.

A custom transport can be specified.Only 1 may be specified for the entire configuration.
This is a Java implementation of the "com.dtdsoftware.splunk.transport.Transport" interface.
The transport class must be placed on the java classpath.
If the optional transport declaration is omitted, then the default transport(STD out) will be used.

Refer to the javadoc API in the docs directory for viewing bundled Formatter and Transport implementations or the signature
of the interfaces for creating your own custom implementations.

1 or more "jmxserver" elements can be included in this XML defintion ie: if you need to accesss multiple different jmx sources within the same scheduled execution
Each "jmxserver" element you declare will be run in parallel.

"jmxuser" and "jmxpass" are optional , if not specified, they will be ignored

"host" can be a hostname, dns alias or ip address.

"jmxport" is the JMX server port to connect to.

"protocol"(default is rmi) is the jmx service protocol to use : rmi, iiop and mx4j specific protocols soap, hessian etc..

For more advanced finer grained control over the format of the jmx service URL you can also specify the "stubSource"(default is jndi), "encodedStub" and url "lookupPath"(default is jmxrmi).
Refer to the schema docs for more details.

Or, to facilitate complete user flexibility, you can enter the full jmx service url as a raw string using the "jmxServiceURL" attribute.

This is raw jmx service url in standard format "service:jmx:protocol:sap"

If this attribute is set, it will override any other set connection related attributes("host", "jmxport", "protocol", "stubSource", "encodedStub" and "lookupPath")

You can also attach directly to a locally running JVM process by specifying the Process ID in the "pid" attribute.
In this case, the "host", "jmxport", "jmxuser","jmxpass" attributes will not be used.
The host value will default to the localhost's name.

For more dynamic flexibility,rather than specifying a static Process ID value in the config xml , you can also specfy the name of a PID file that contains the JVM's Process ID.
You set this in the "pidFile" attribute.
Many long running Java processes output the PID value to a file, particularly if using a JVM Service Wrapper such as Tanuki.
Each time the Splunk JMX Poller runs, it will dynamically inspect the value of the PID from this file.
This way, you dont have to alter the config.xml each time you do a JVM restart.

You can also specify the path to a command to execute to output the JVM's PID.
You set this in the "pidCommand" attribute.
This command might be a custom script that looks for the JVM process and extracts the PID to STD OUT.
On Linux you could do this with a simple script that uses ps, grep and awk.

Example : ps -eafH | grep "java" | grep -v "grep" | awk '{print $2}'

Look at the example xml config file in the bin/config directory for usage examples.

Groups of jmxserver's that share that same mbeans can be grouped together in a cluster element,
so that you only have to declare the MBeans/MBean attributes/operations once ie: a cluster of JEE appservers, a cluster
of Hadoop or Cassandra nodes etc...This will be useful for enterprises with very large scale JVM deployments.
MBean inheritance is supported, so you can have mbeans defined that are common to the cluster, and mbeans that are
specific to a cluster member.

At index time the "host" field is extracted and transformed into the SPLUNK internal host value.

"jvmDescription" is just meta data, useful for searching on in SPLUNK where you might have multiple JVM's on the same host

By default, no timestamp is added , instead relying on the SPLUNK index time as the event time.
However you may customise the 3 bundled formatters and configure them to prepend a timestamp if you wish.
See the examples in the default config.xml file

For MBean definitions , standard JMX object name wildcard patterns * and ? are supported for the "domain" and "properties" attributes
http://download.oracle.com/javase/1,5.0/docs/api/javax/management/ObjectName.html

If no values are specified for the "domain" and "properties" attributes , the value will default to the * wildcard

The MBean's canonical objectname will be written out to SPLUNK.

If you would like the components of the MBean canonical objectname broken out into individual fields, then there is a custom formatter available
which you can declare in your config XML file to achieve this :

<formatter className="com.dtdsoftware.splunk.formatter.TokenizedMBeanNameFormatter" />

Single, composite and tabular MBean attributes are supported.

You can set the "dumpAllAttributes" value to true and all of an MBean's attributes will be extracted.

ie: dump all the attributes in the java.lang domain
<mbean domain="java.lang" properties="*" dumpAllAttributes="true"/>

ie: dump all the attributes in all of the cassandra domains
<mbean domain="org.apache.cassandra.*" properties="*" dumpAllAttributes="true" />

Or you can specify each individual attribute you want to extract using the "attribute" element.

For attributes that are multi level ie: composite and tabular attributes , then you can use a ":" delimited notation for specifying the attribute name.
See examples of this in config.xml

MBean operation invocation is supported.
You can declare operations that return a result or not , take arguments or not.
Operation overloading is supported for methods of the same name with different signatures.

The following parameter types are supported :

• int
• float
• double
• long
• short
• byte
• boolean
• char
• string

Internally these get autoboxed into their respective Object counterparts.

See config_operations_example.xml for usage scenarios.

Example output line (default formatter)

host=damien-laptop,jvmDescription="My test JVM",mbean="java.lang:type=Threading",count="47"

SUPPORTED RETURN DATA TYPES FOR OPERATIONS AND ATTRIBUTES

Simple, Composite and Tabular MBean data structures are supported.

The value obtained from the Attribute or as the return type of an Operation can be :

1. Any primitive type(Object wrapped)
2. String
3. Array
4. List
5. Map
6. Set

Collection types will be recursively deeply inspected, so you can have Collections of Collections etc..

JVM "ATTACH" NATIVE LIBRARYS

When connecting directly to local JVM using a process ID, native librarys are used. You shouldn't need to do anything, they are loaded automatically.

Windows : JRE_HOME/bin/attach.dll
Linux : JDK_HOME/jre/lib/i386/libattach.so

NOTE : For some reason the Linux "attach" library is only packaged in the JRE that is part of a JDK install. Weird.

CREATING ADDITIONAL CONFIGURATION FILES

If required, you can easily create more config files to run at different schedules by "cloning"
the script input in Splunk Manager and specifying the name of the new config file as an argument to the poll_jmx script.
The config file should reside in SPLUNK_HOME/etc/apps/SPLUNK4JMX/bin/config

JMX SERVERS

To set up a JMX server for remote access via jvm system properties at startup, see this :
http://download.oracle.com/javase/1.5.0/docs/guide/management/agent.html#remote

MX4J SUPPORT

If you wish to use MX4J as your JMX implementation for remote connectors(rmi and iiop) or use any of the MX4J specific JMX connectors(soap, burlap, hessian, local), then it is simply a matter of using the poll_mx4j_jmx script rather than the standard poll_jmx script.

For more details about MX4J browse here : http://mx4j.sourceforge.net

Note : If using any of the HTTPS connectors(soap+ssl, hessian+ssl, burlap+ssl), the root certification authority must be present in the trusted certificates, normally stored in the "$JAVA_HOME/jre/lib/security/cacerts" file.

ADDING JAVA CLASSES TO THE APPLICATION CLASSPATH

You can dump a jar file in the "SPLUNK_HOME/etc/apps/SPLUNK4JMX/bin/lib/ext" directory and it will be automatically loaded.
Why would you want to do this ? Well perhaps you have created a custom formatter/transport implementation as described above.

ADDING JAVA CLASSES TO THE BOOT CLASSPATH

Any classes that you need prepended to the java bootclasspath should be put in "SPLUNK_HOME/etc/apps/SPLUNK4JMX/bin/lib/boot".
They won't be automatically loaded, you will need to specify each jar in the poll_jmx/poll_mx4j_jmx script.
Why would you want to do this ? Well perhaps you are targeting a JVM with some proprietary JMX logic that requires additional libraries on the JMX client side ie: using the IIOP connector with IBM Websphere products is one such scenario I've encountered.

LOGGING

By default , "error" level log events will be written to SPLUNK_HOME/etc/apps/SPLUNK4JMX/logs
The log file has a max size of 1MB and 1 backup.
If you need to tweak log settings, then the log4j.xml config file resides in jmxpoller.jar
The SPLUNK4JMX app will by default monitor it's own log files and present a "JMX Connectivity" view on the navigation
menu bar showing any errors.

TRANSPORTS

Bundled with the release are transport implementations for :

• STD out : this is the default
• TCP : stream events over a TCP socket.You will also need to setup a TCP input in SPLUNK

These 3 transports also use appenders/loggers in log4j.xml to facilitate their functionality
So if using one of these transports, you'll need to uncomment and configure the relevant appender/logger
definition in log4j.xml that resides in jmxpoller.jar

• Syslog : send events over SYSLOG UDP.You will also need to setup a UDP input in SPLUNK
• Splunk REST endpoint : stream events over HTTP/REST to Splunk.
• File : write events to file.You will also need to setup a monitor input in Splunk.

TROUBLESHOOTING

• check your firewall setup
• check the correct spelling/case of your MBeans and MBean attribute/operation definitions
• check your hostname and port
• check your user and pass
• check that your JVM remote JMX access is correctly setup , can you connect using JConsole ?
• check that the SPLUNK_HOME and JAVA_HOME env variables have been correctly setup and exported
• ensure you have enabled the poll_jmx.sh / poll_jmx.bat script input
• check your process ID if trying to attach to a local JVM
• ensure your XML config adheres to the XSD
• look for errors in the log files or via Splunk search
• check for JMX connection timeouts
• check that the MBean server is not adding/removing domains and mbeans whilst you are concurrently polling them