Splunk Java Agent v1.4.1

Overview

This JVM(Java Virtual Machine)agent can be used to obtain metrics for Java APM(Application Performance Monitoring).

The types of metrics extracted by the agent and streamed to Splunk are :

  1. class loading
  2. method execution
  3. method timings (cumulative, min, avg, max, std deviation)
  4. method call tracing(count of calls, group by app/app node(for clustered systems)/thread/class/package)
  5. application/thread stalls
  6. errors/exceptions/throwables
  7. JVM heap analysis, object/array allocation count/size,class dumps, leak detection, stack traces, frames
  8. JMX attributes/operations/notifications from the JVM or Application layer MBean Domains

The agent is able to obtain these metrics dynamically at runtime by "weaving" the necessary bytecode into loaded classes using the Java instrumentation API and the ASM framework. There is no source code changes required by the end user and no class files on disk are altered.

JMX metrics are obtained via polling MBeans attributes, invoking operations & listening for notifications in the locally running Platform MBeanServer.

JVM heap profiling metrics are obtained via decoding a dynamically generated HPROF dump.

By default , the metrics will be streamed directly into Splunk over TCP, however the transport mechanism is configurable and extensible.

In the Splunk UI you can then create Splunk searches over the agent data and visualizations, reports,alerts etc.. over the results of the searches. The events are already being fed into Splunk in best practice semantic format, key=value pairs , no additional field extractions are required. As Splunk is being using to index all the data and perform searches(real time if you wish) massive amounts of tracing data from as many JVMs as you need to monitor can be indexed and correlated and you can leverage all of the scalability and HA features of the Splunk platform to deliver an end to end Java APM solution.

You can refer to some search examples in this presentation online : http://www.slideshare.net/damiendallimore/splunk-conf-2014-splunking-the-java-virtual-machine

Supported Java Runtime

  1. JRE 5,6,7,8
  2. JVMs : Hotspot , JRockit, OpenJDK, IBM J9

Install

  1. Uncompress splunkagent.tar.gz
  2. The agent is just a single jar file, splunkagent.jar, you should see this in the uncompressed directory.

Activation Key

You require an activation key to use this agent. Visit http://www.baboonbones.com/#activation to obtain a non-expiring key

Transports

Data is transported to Splunk via TCP or HEC (HTTP Event Collector)

Create a TCP input in Splunk

Login to Splunk and browse to Data Inputs to setup a new TCP input. This is how the agent will by default stream the metrics to Splunk. Whatever port you designate must be configured in the agent properties also (splunk.transport.tcp.port)

Create a HEC input in Splunk

Login to Splunk and browse to Data Inputs to setup a new HEC input. You can then obtain your HEC token and any other HEC settings to be used in the configuration options detailed below.

Setup

Pass the follow argument to your JVM at startup:

-javaagent:splunkagent.jar

or

-javaagent:lib/splunkagent.jar

etc..

The location of the jar file should be relative to the directory where you are executing "java" from.

All dependencies and resources are bundled into the jar file.

Configuration files

You can configure the agent with the properties file "splunkagent.properties" that resides inside the jar file. The various options are detailed below. Open the jar , edit the file, close the jar.

Alternatively you can pass the properties file in as an argument , rather than having to bundle it into the agent jar.

Example : -javaagent:splunkagent.jar=/Users/foo/splunkagent.properties

You can configure the JMX polling with 1 or more config files that reside either inside the jar file at an external location outside of the jar. The JMX logic is just an embedded version of the "Splunk for JMX" app(http://splunk-base.splunk.com/apps/25505/splunk-for-jmx) , so refer to that app's documentation for config file options. The names of the JMX config files and the frequency at which they are fired is configured in "splunkagent.properties"

Dynamic reload

For configuration files that reside outside of the agent jar file , if you make any changes to this file during the JVM runtime, these changes will be automatically detected , reloaded and the agent re initialised without having to perform a JVM restart. Pretty cool huh !

Tracing verbosity

Unless you want incredibly verbose tracing , you will want to specify just the packages/classes/methods you are interested in profiling in the "agent.whitelist" property

Logging

Logging levels can be configured in splunkagent.properties.By default logging will get written to a file in the runtime directory named splunkagent.yyyy-MM-dd.log. This gets rolled daily.

Remote control via JMX

The agent can be completely controlled and introspected dynamically at runtime via JMX, either locally or remotely. I recommend using JConsole that ships with Java to connect to the JVM's JMX MBeanServer and browse the MBeans in the "splunkjavaagent" domain. For remote JMX connectivity to your target JVM , you will also need to enable remote JMX. More info here : http://docs.oracle.com/javase/7/docs/technotes/guides/management/agent.html

Properties Options

Core Settings

Common Transport Options

TCP Transport

HEC Transport

Tracing Options

HPROF Options

JMX Options

Whitelist/Blacklist Patterns

Contact

www.baboonbones.com