This JVM(Java Virtual Machine)agent can be used to obtain metrics for Java APM(Application Performance Monitoring).
The types of metrics extracted by the agent and streamed to Splunk are :
The agent is able to obtain these metrics dynamically at runtime by "weaving" the necessary bytecode into loaded classes using the Java instrumentation API and the ASM framework. There is no source code changes required by the end user and no class files on disk are altered.
JMX metrics are obtained via polling MBeans attributes, invoking operations & listening for notifications in the locally running Platform MBeanServer.
JVM heap profiling metrics are obtained via decoding a dynamically generated HPROF dump.
By default , the metrics will be streamed directly into Splunk over TCP, however the transport mechanism is configurable and extensible.
In the Splunk UI you can then create Splunk searches over the agent data and visualizations, reports,alerts etc.. over the results of the searches. The events are already being fed into Splunk in best practice semantic format, key=value pairs , no additional field extractions are required. As Splunk is being using to index all the data and perform searches(real time if you wish) massive amounts of tracing data from as many JVMs as you need to monitor can be indexed and correlated and you can leverage all of the scalability and HA features of the Splunk platform to deliver an end to end Java APM solution.
You can refer to some search examples in this presentation online : http://www.slideshare.net/damiendallimore/splunk-conf-2014-splunking-the-java-virtual-machine
You require an activation key to use this agent. Visit http://www.baboonbones.com/#activation to obtain a non-expiring key
Data is transported to Splunk via TCP or HEC (HTTP Event Collector)
Login to Splunk and browse to Data Inputs to setup a new TCP input. This is how the agent will by default stream the metrics to Splunk. Whatever port you designate must be configured in the agent properties also (splunk.transport.tcp.port)
Login to Splunk and browse to Data Inputs to setup a new HEC input. You can then obtain your HEC token and any other HEC settings to be used in the configuration options detailed below.
Pass the follow argument to your JVM at startup:
The location of the jar file should be relative to the directory where you are executing "java" from.
All dependencies and resources are bundled into the jar file.
You can configure the agent with the properties file "splunkagent.properties" that resides inside the jar file. The various options are detailed below. Open the jar , edit the file, close the jar.
Alternatively you can pass the properties file in as an argument , rather than having to bundle it into the agent jar.
Example : -javaagent:splunkagent.jar=/Users/foo/splunkagent.properties
You can configure the JMX polling with 1 or more config files that reside either inside the jar file at an external location outside of the jar. The JMX logic is just an embedded version of the "Splunk for JMX" app(http://splunk-base.splunk.com/apps/25505/splunk-for-jmx) , so refer to that app's documentation for config file options. The names of the JMX config files and the frequency at which they are fired is configured in "splunkagent.properties"
For configuration files that reside outside of the agent jar file , if you make any changes to this file during the JVM runtime, these changes will be automatically detected , reloaded and the agent re initialised without having to perform a JVM restart. Pretty cool huh !
Unless you want incredibly verbose tracing , you will want to specify just the packages/classes/methods you are interested in profiling in the "agent.whitelist" property
Logging levels can be configured in splunkagent.properties.By default logging will get written to a file in the runtime directory named splunkagent.yyyy-MM-dd.log. This gets rolled daily.
The agent can be completely controlled and introspected dynamically at runtime via JMX, either locally or remotely. I recommend using JConsole that ships with Java to connect to the JVM's JMX MBeanServer and browse the MBeans in the "splunkjavaagent" domain. For remote JMX connectivity to your target JVM , you will also need to enable remote JMX. More info here : http://docs.oracle.com/javase/7/docs/technotes/guides/management/agent.html