Monitoring your Tomcat application server, or any other Java process, using JMX and OpenNMS

This post will show you how to monitor a Tomcat application server using JMX MBeans and an SNMP management application. The first part will guide you through securely enabling JMX management on the Java process. Once you have done this, you can use tools like Visual VM to monitor your process and its MBeans. If you want to take a step further and have this data gathered all the time, you might want to continue with the second part and integrate it with OpenNMS.

Requirements

It is assumed that you have Tomcat on a server you want to manage, and you have also got Sun Java 7 SDK (not JRE) to run on your machine (they both can be the same machine, but we keep them separated here). You can apply this to any other Java process with little modification (to the MBean names maybe).

Enable JMX on Server

Add a few Java arguments to your CATALINA_OPTS to enable remote connection to JMX, and then restart the application server:

export CATALINA_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=1100 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.password.file=/jre/lib/management/jmxremote.password"

Replacing with the appropriate path for your Java, copy /jre/lib/management/jmxremote.password.template into /jre/lib/management/jmxremote.password and set the passwords for monitorRole and controlRole at the end of the file as you wish. We will assume you have set them to MONITOR_PASS and CONTROL_PASS here .

Make sure that your firewall doesn’t block port 1100. You may change the port number as you wish, but remember to change it everywhere mentioned in this guide.

Now run a JMX utility to connect to your JMX server, like JConsole or Visual VM. If using JConsole, connect to a remote host providing hostname, port (1100), username (controlRole), and password (CONTROL_PASS). If using Visual VM with a VisualVM-MBeans plugin installed, you might be able to browse among the MBeans and call some operations (click to enlarge):

This might be enough if you want to see what’s going on in the JVM at the moment. You can even double click the numbers (only the ones highlighted) to see a graph over time.

But if you want to keep historical records of different things over time, keep reading. We will be demonstrating how to use a network management software to connect to JMX and gather information. This will be based on an open source application called OpenNMS, but you can probably use your own software so long as it supports JSR160.

Install OpenNMS

This document provides instructions for Fedora 16. Please follow the OpenNMS installation guide for other operating systems.

Add OpenNMS repository.

rpm -Uvh http://yum.opennms.org/repofiles/opennms-repo-stable-fc16.noarch.rpm

Install, init, and start PostgreSQL server, if you don’t have one.

yum -y install postgresql postgresql-server
/sbin/service postgresql initdb
/sbin/service postgresql start
/sbin/chkconfig postgresql on

To allow OpenNMS which is run as root to connect to PostgreSQL as opennms, you need to relax some access requirements. Edit /var/lib/pgsql/data/pg_hba.conf and make sure you have entries like this (change the last column):

local   all         all                               trust
host    all         all         127.0.0.1/32          trust
host    all         all         ::1/128               trust

Now restart PostgreSQL for these changes to take effect:

/sbin/service postgresql restart

Now you are ready to install OpenNMS:

yum -y install opennms
/opt/opennms/bin/runjava -S /usr/java/latest/bin/java
/opt/opennms/bin/install -dis
/sbin/service opennms start
Discover and Configure

If you have followed the instructions in the previous section, you are now able to access the web interface of OpenNMS via http://:8980/opennms/. Enter admin for both username and password when prompted. Just remember to open port 8980 in your firewall for hosts you will be accessing OpenNMS web interface from.
The easiest way to set up OpenNMS to monitor a JMX service is to hijack the configuration to set up its own JMX interface. The only things we need to change is port number, username, password, and a few names that are different in Tomcat. Please see this wiki page for an explanation.
Please modify your configuration files stated below to make sure that it matches what provided here.

/opt/opennms/etc/capsd-configuration.xml

<protocol-plugin protocol="OpenNMS-JVM" class-name="org.opennms.netmgt.capsd.plugins.Jsr160Plugin" scan="on" user-defined="false">
    <property key="port" value="1100" />
    <property key="factory" value="PASSWORD-CLEAR"/>
    <property key="username" value="controlRole" />
    <property key="password" value="CONTROL_PASS" />
    <property key="protocol" value="rmi"/>
    <property key="urlPath" value="/jmxrmi"/>
    <property key="timeout" value="3000" />
    <property key="retry" value="2" />
    <property key="type" value="default" />

/opt/opennms/etc/collectd-configuration.xml

<service name="OpenNMS-JVM" interval="300000" user-defined="false" status="on">
        <parameter key="port" value="1100"/>
        <parameter key="factory" value="PASSWORD-CLEAR"/>
        <parameter key="username" value="controlRole" />
        <parameter key="password" value="CONTROL_PASS" />
        <parameter key="retry" value="2"/>
        <parameter key="timeout" value="3000"/>
        <parameter key="protocol" value="rmi"/>
        <parameter key="urlPath" value="/jmxrmi"/>
        <parameter key="rrd-base-name" value="java" />
        <parameter key="ds-name" value="opennms-jvm"/>
        <parameter key="friendly-name" value="opennms-jvm"/>
        <parameter key="collection" value="jsr160"/>
        <parameter key="thresholding-enabled" value="true"/>

/opt/opennms/etc/poller-configuration.xml

<service name="OpenNMS-JVM" interval="300000" user-defined="false" status="on">
  <parameter key="port" value="1100"/>
  <parameter key="factory" value="PASSWORD-CLEAR"/>
  <parameter key="username" value="controlRole"/>
  <parameter key="password" value="CONTROL_PASS"/>
  <parameter key="retry" value="2"/>
  <parameter key="timeout" value="3000"/>
  <parameter key="rrd-repository" value="/opt/opennms/share/rrd/response" />
  <parameter key="ds-name" value="opennms-jvm"/>
  <parameter key="friendly-name" value="opennms-jvm"/>

/opt/opennms/etc/jmx-datacollection-config.xml

...
<mbean name="JVM MemoryPool:Eden Space" objectname="java.lang:type=MemoryPool,name=PS Eden Space">
...
<mbean name="JVM MemoryPool:Survivor Space" objectname="java.lang:type=MemoryPool,name=PS Survivor Space">
...
<mbean name="JVM MemoryPool:Perm Gen" objectname="java.lang:type=MemoryPool,name=PS Perm Gen">
...
<mbean name="JVM MemoryPool:Old Gen" objectname="java.lang:type=MemoryPool,name=PS Old Gen">
...

Now restart the server after making these changes.

/sbin/service opennms restart

Once restarted successfully, go to the web interface and perform the following steps:

  • In the Admin tab, click “Add Interface for Scanning”, then enter and add.
  • In the Events tab, click “All Events” and look for services being discovered.
  • In the Reports tab, click “Resource Graphs”, select Tomcat server in the standard reports, select the opennms-jvm, then click “Graph Selection”.

Here are your graphs.

You need to follow the same pattern to monitor other MBeans. For example to graph total compilation time we need to follow these steps.
First, add an entry in jmx-datacollection-config.xml file to query the MBean (use JConsole or JVisualVM to find the name of the MBean you are interested in):

<mbean name="JVM Compilation" objectname="java.lang:type=Compilation">
  <attrib name="TotalCompilationTime" alias="TotCompilationTime" type="gauge"/>

Then, add a report template in the snmp-graph.properties section:

report.jvm.compilation.name=JVM Compilation
report.jvm.compilation.columns=TotCompilationTime
report.jvm.compilation.type=interfaceSnmp
report.jvm.compilation.command=--title="JVM Compilation" \
 DEF:compilationTime={rrd1}:TotCompilationTime:AVERAGE \
 LINE2:compilationTime#0000ff:"Compilation Time" \
 GPRINT:compilationTime:AVERAGE:" Avg \\: %8.2lf %s\\n"

Finally, don’t forget to add this new graph to the list of graphs (mind the semicolon and backslash at the end of the line):

reports=mib2.HCbits, mib2.bits, mib2.percentdiscards, mib2.percenterrors, \
...
jvm.gc.copy, jvm.gc.msc, jvm.gc.parnew, jvm.gc.cms, jvm.gc.psms, jvm.gc.pss, jvm.compilation, \
...
Advertisements

Android app to "Google" images

Some time ago I wrote a simple Android app to search for images using Google Search API, I thought it is worth sharing in case someone else needs to do the same. I had to create a custom search engine (https://www.google.com/cse/) and create an API project (https://code.google.com/apis/console) to get set up. The rest is ordinary stuff you all know.

There is one thing to mention though: Using Java search API you have to specify the web sites you want to search in. You can’t search the whole Internet using this API. You could do this using image search API, but since it is deprecated it is not worth investing in.

The complete working code along with a pre-built .apk is also provided. I have implemented other interesting features too like search suggestions and infinite loading, which will become handy down for you the road.

Simply clone https://github.com/normanatashbar/imagesearch.git or download by clicking here. Please remember to change the search engine ID and API key with your own if you are using this code as a base.