Monitoring the JVM with SNMP

By , 21 January 2007

Monitoring the JVM with SNMP

Since Java 1.5, Sun's JVM has included an SNMP agent which is quite handy for keeping an eye on your Java apps using your existing monitoring toolset. Here's how to set up OpenNMS to monitor an app server and produce pretty graphs such as the one below, alongside your other SNMP collected metrics like CPU load and memory usage.

Monitoring the JVM with SNMP

1. Install OpenNMS

First you need to install OpenNMS [1] onto the machine which is going to do the monitoring. OpenNMS is not difficult to install if you've had any experience with maven and Tomcat. Their website covers installation fairly accurately, so I'll skip this step and go straight into how to setup the JVM.

2. Enable Java SNMP

Enabling the SNMP agent on the JVM is pretty simple [2], and can be done adding either -Dcom.sun.management.snmp.port=161 or -Dcom.sun.management.config.file=snmp.properties to the java command line. Using the first method gives you an SNMP agent with all the defaults, which includes only listening on the loopback interface. Using the second method allows you to specify a few more properties in the snmp.properties file (or any other file of your choice).

For this task, I wanted SNMP on port 1161 since the app server runs as an underprivileged user and I already had another agent on port 161. I also needed to listen on all interfaces because the machine is being monitored remotely. Finally, since my firewall keeps out unwanted guests, I've also disabled the access-control list. The complete snmp.properties file looks like this:

com.sun.management.snmp.interface=0.0.0.0
com.sun.management.snmp.port=1161
com.sun.management.snmp.acl=false

Test your configuration by trying to query the Java SNMP agent first from the localhost, then from the monitoring machine:

$ snmpwalk -c public -v2c java.example.com:1161 .1.3.6.1.4.1.42.2.145.3.163.1.1.4.1
SNMPv2-SMI::enterprises.42.2.145.3.163.1.1.4.1.0 = STRING: "pid@host"
 

3. Configure OpenNMS

Next, we need to configure OpenNMS, which is a little more involved. Conceptually, there are four steps:

  1. Configure the capabilities daemon to look for the snmp-jvm service on your chosen port (capsd-configuration.xml).
  2. Configure the collect daemon to fetch data for this service (collectd-configuration.xml).
  3. Configure the OIDs for the data you want to collect (datacollection-config.xml).
  4. Configure the snmp graphs to display the collected values (snmp-graph.properties).

capsd configuration

This tells OpenNMS to look for the SNMP agent you have set up. Add the following configuration to capsd-configuration.xml, restart OpenNMS and rescan the host's services using the web interface. You should see a new service 'SNMP-JVM' on the host.

<protocol-plugin protocol="SNMP-JVM" class-name="org.opennms.netmgt.capsd.plugins.SnmpPlugin" 
                 scan="on" user-defined="false">
    <property key="timeout" value="5000" />
    <property key="retry" value="3" />
    <property key="port" value="1161" />
    <property key="vbname" value=".1.3.6.1.4.1.42.2.145.3.163.1.1.4.1" />
</protocol-plugin>

collectd configuration

To tell OpenNMS to actually collect the data from this service, you have to add the following to the collectd-configuration.xml file.

<service name="SNMP-JVM" interval="300000" user-defined="false" status="on">
    <parameter key="collection" value="jvm"/>
    <parameter key="port" value="1161"/>
    <parameter key="retry" value="3"/>
    <parameter key="timeout" value="3000"/>
    <parameter key="oid" value=".1.3.6.1.4.1.42.2.145.3.163.1.1.4.1"/>
</service>

OID configuration

You also need to decide what data you want to monitor by reading the Java MIB [3]. Here are the values I chose to monitor:

Variable OID
jvmMemoryHeapUsed 1.3.6.1.4.1.42.2.145.3.163.1.1.2.11
jvmMemoryHeapCommitted 1.3.6.1.4.1.42.2.145.3.163.1.1.2.12
jvmMemoryHeapMaxSize 1.3.6.1.4.1.42.2.145.3.163.1.1.2.13
jvmMemoryNonHeapUsed 1.3.6.1.4.1.42.2.145.3.163.1.1.2.21
jvmMemoryNonHeapCommited 1.3.6.1.4.1.42.2.145.3.163.1.1.2.22
jvmMemoryNonHeapMaxSize 1.3.6.1.4.1.42.2.145.3.163.1.1.2.23
jvmThreadCount 1.3.6.1.4.1.42.2.145.3.163.1.1.3.1

This information is added to datacollection-config.xml with the snippet below. An entire <snmp-collection/> needs to be created, because the jvm SNMP agent doesn't include a system table that might be used to distinguish it as a unique system in the default data collection.

<snmp-collection name="jvm" maxVarsPerPdu="10" snmpStorageFlag="primary">
  <rrd step="300">
    <rra>RRA:AVERAGE:0.5:1:8928</rra>
    <rra>RRA:AVERAGE:0.5:12:8784</rra>
    <rra>RRA:MIN:0.5:12:8784</rra>
    <rra>RRA:MAX:0.5:12:8784</rra>
  </rrd>

  <groups>
    <group name="jvm" ifType="all">
      <mibObj oid=".1.3.6.1.4.1.42.2.145.3.163.1.1.2.11" instance="0" alias="jvmHeapUsed" type="Gauge64" />
      <mibObj oid=".1.3.6.1.4.1.42.2.145.3.163.1.1.2.12" instance="0" alias="jvmHeapCommitted" type="Gauge64" />
      <mibObj oid=".1.3.6.1.4.1.42.2.145.3.163.1.1.2.13" instance="0" alias="jvmHeapMax" type="Gauge64" />
      <mibObj oid=".1.3.6.1.4.1.42.2.145.3.163.1.1.2.21" instance="0" alias="jvmNonHeapUsed" type="Gauge64" />
      <mibObj oid=".1.3.6.1.4.1.42.2.145.3.163.1.1.2.22" instance="0" alias="jvmNonHeapCommitted" type="Gauge64" />
      <mibObj oid=".1.3.6.1.4.1.42.2.145.3.163.1.1.2.23" instance="0" alias="jvmNonHeapMax" type="Gauge64" />
      <mibObj oid=".1.3.6.1.4.1.42.2.145.3.163.1.1.3.1" instance="0" alias="jvmThreadCount" type="Gauge64" />
    </group>
  </groups>

  <systems>
    <systemDef name="JVM">
      <sysoidMask></sysoidMask>
      <collect>
        <includeGroup>jvm</includeGroup>
      </collect>
    </systemDef>
  </systems>
</snmp-collection>

Now restart OpenNMS and look for the new RRD data files in share/rrd/**/jvm* to make sure the collection is working. Also check the collectd.log file for error messages.

Creating graphs

Phew! Almost there. If your RRD files are being created all you have to do is edit the snmp-graph.properties config file and reload the graphs page. Here's the configuration I used to create the graphs in this blog:

report.jvm.heap.name=JVM Heap Memory
report.jvm.heap.columns=jvmHeapUsed, jvmHeapCommitted, jvmHeapMax
report.jvm.heap.type=nodeSnmp
report.jvm.heap.command=--title="JVM Heap Memory" \
 DEF:used={rrd1}:jvmHeapUsed:AVERAGE \
 DEF:comm={rrd2}:jvmHeapCommitted:AVERAGE \
 DEF:max={rrd3}:jvmHeapMax:AVERAGE \
 AREA:used#0000ff:"Used     " \
 GPRINT:used:AVERAGE:" Avg  \\: %5.2lf %s" \
 GPRINT:used:MIN:"Min  \\: %5.2lf %s" \
 GPRINT:used:MAX:"Max  \\: %5.2lf %s\\n" \
 LINE2:comm#00ff00:"Committed" \
 GPRINT:comm:AVERAGE:" Avg  \\: %5.2lf %s" \
 GPRINT:comm:MIN:"Min  \\: %5.2lf %s" \
 GPRINT:comm:MAX:"Max  \\: %5.2lf %s\\n" \
 LINE2:max#ff0000:"Max           " \
 GPRINT:max:AVERAGE:" Avg  \\: %5.2lf %s" \
 GPRINT:max:MIN:"Min  \\: %5.2lf %s" \
 GPRINT:max:MAX:"Max  \\: %5.2lf %s\\n"

report.jvm.nonheap.name=JVM Non-Heap Memory
report.jvm.nonheap.columns=jvmNonHeapUsed, jvmNonHeapCommitted, jvmNonHeapMax
report.jvm.nonheap.type=nodeSnmp
report.jvm.nonheap.command=--title="JVM Non-Heap Memory" \
 DEF:used={rrd1}:jvmNonHeapUsed:AVERAGE \
 DEF:comm={rrd2}:jvmNonHeapCommitted:AVERAGE \
 DEF:max={rrd3}:jvmNonHeapMax:AVERAGE \
 AREA:used#0000ff:"Used     " \
 GPRINT:used:AVERAGE:" Avg  \\: %5.2lf %s" \
 GPRINT:used:MIN:"Min  \\: %5.2lf %s" \
 GPRINT:used:MAX:"Max  \\: %5.2lf %s\\n" \
 LINE2:comm#00ff00:"Committed" \
 GPRINT:comm:AVERAGE:" Avg  \\: %5.2lf %s" \
 GPRINT:comm:MIN:"Min  \\: %5.2lf %s" \
 GPRINT:comm:MAX:"Max  \\: %5.2lf %s\\n" \
 LINE2:max#ff0000:"Max          " \
 GPRINT:max:AVERAGE:" Avg  \\: %5.2lf %s" \
 GPRINT:max:MIN:"Min  \\: %5.2lf %s" \
 GPRINT:max:MAX:"Max  \\: %5.2lf %s\\n"

report.jvm.threads.name=JVM Threads
report.jvm.threads.columns=jvmThreadCount
report.jvm.threads.type=nodeSnmp
report.jvm.threads.command=--title="JVM Thread Count" \
 DEF:threads={rrd1}:jvmThreadCount:AVERAGE \
 LINE2:threads#0000ff:"Threads" \
 GPRINT:threads:AVERAGE:" Avg \\: %8.2lf %s" \
 GPRINT:threads:MIN:"Min  \\: %8.2lf %s" \
 GPRINT:threads:MAX:"Max  \\: %8.2lf %s\\n"

Add these report definitions towards the end of the file, and their names (jvm.heap, jvm.nonheap and jvm.threads) to the list at the top of the file to have them display on the graphs page.

JMX Monitoring

OpenNMS also does JMX monitoring, but this is likely to be the subject of another blog post. Unfortunately, AFAICT, it does not yet have support for CompositeTypes which means we can't (yet) collect heap and non-heap usage via the Memory managed bean.

References

[1] http://www.opennms.org
[2] http://java.sun.com/j2se/1.5.0/docs/guide/management/SNMP.html
[3] http://java.sun.com/j2se/1.5.0/docs/guide/management/JVM-MANAGEMENT-MIB.mib

About Roger Keays

Monitoring the JVM with SNMP

Roger Keays is an artist, an engineer, and a student of life. He has no fixed address and has left footprints on 40-something different countries around the world. Roger is addicted to surfing. His other interests are music, psychology, languages, the proper use of semicolons, and finding good food.

Leave a Comment

Please visit https://rogerkeays.com/blog/monitoring-the-jvm-with-snmp to add your comments.

Comment posted by: Adnan, 13 years ago

Dear Fellows,

Please do help us in our study. It is very important for us to have feedback from you (who is using OpenNMS). I bet I would not take you more than 5 minutes, all questions are multiple choice and on a single page. We will be highly thankful to all the participants.

Please click on following link to answer this survey.

http://www.kwiksurveys.com/online-survey.php?surveyID=KBINNF_9b336176

Regards

Comment posted by: , 14 years ago

I've posted an equivalent configuration using MRTG instead of OpenNMS at

http://www.ilikespam.com/blog/monitoring-the-jvm-with-snmp-mrtg

Comment posted by: syed, 15 years ago

I am not aware of this SNMP trap, agent and opneNMS.

Please give me knowledge about this SNMP trap to monitor host's event.

Env:

Windows 2x

OpenNMS and postgresql

Comment posted by: Oka, 15 years ago

While this information is interesting, I have found its not easy to implement nor does it work for everyone, if you want to just graph your JVM I suggest the following opennms url  www.opennms.org/index.php/JVM_Monitoring_using_SNMP it was the only way I could get it to reliably graph the jvm.... however the GC protion of the code does need some work.

Comment posted by: , 15 years ago

Well, FWIW I can't seem to get anything to work in OpenNMS these days. I switched to MRTG for SNMP visualisations and it took about 5 minutes to set up. Sooo much simpler!

Comment posted by: AG, 15 years ago

Has anyone gotten this to work?  If so, could you please post the details?  I am not getting any data collected with the exact settings above. only indicator in the capsd.log is "protocol SNMP-JVM supported? false"

Comment posted by: Seb, 16 years ago

When I run opennms.
I have this in capsd.log :
    2008-01-31 16:25:27,503 ERROR [main] Capsd: Failed to load poller
configuration
    ValidationException: rrd is a required field.;
     - location of error: XPATH: poller-configuration/package
       at org.exolab.castor.xml.FieldVal

idator.validate(Unknown Source)
       at org.exolab.castor.xml.util.XMLClassDescriptorImpl.validate(Unknown
Source)
       at org.exolab.castor.xml.Validator.validate(Unknown Source)
       at org.exolab.castor.xml.FieldValidator.validate(Unknown Source)
       at org.exolab.castor.xml.util.XMLClassDescriptorImpl.validate(Unknown
Source)
       at org.exolab.castor.xml.Validator.validate(Unknown Source)
       at org.exolab.castor.xml.UnmarshalHandler.endElement(Unknown Source)
       at org.apache.xerces.parsers.AbstractSAXParser.endElement(
AbstractSAXParser.java:552)
       ......
 
What can i do ?
Comment posted by: Anuj, 16 years ago
Hello All: I finally figured out the problem with the SNMP trap data collection (mentioned above). Looks like the OpenNMS doesnot recognize a custom community string. hence use public in your agent as the community string and use the same (public) in your snmp-copnfig.xml and everything will be fine (assuming the snmp walk is successful). Thanks
Comment posted by: Anuj, 16 years ago
After following the instructions (which were extremely helpful, thanks), I am seeing this in collectd log: 007-10-17 13:12:40,559 DEBUG [CollectdScheduler-50 Pool-fiber0] Collectd: scheduleExistingInterfaces: dbConn = com.mchange.v2.c3p0.impl.NewProxyConnection@1c1ac46, svcName = SNMP-JVM 2007-10-17 13:12:40,560 DEBUG [CollectdScheduler-50 Pool-fiber0] CollectdConfigFactory: interfaceInPackage: Interface 10.24.53.191 passed filter and specific/range for package example1?: true 2007-10-17 13:12:40,570 DEBUG [CollectdScheduler-50 Pool-fiber0] SnmpCollector: initialize: SNMP storage flag: 'all' 2007-10-17 13:12:40,570 DEBUG [CollectdScheduler-50 Pool-fiber0] SnmpCollector: initialize: maxVarsPerPdu=50 2007-10-17 13:12:40,572 DEBUG [CollectdScheduler-50 Pool-fiber0] SnmpCollector: initialize: db retrieval info: nodeid = 1, address = 10.24.53.191, primaryIfIndex = 3, isSnmpPrimary = N 2007-10-17 13:12:40,573 WARN [CollectdScheduler-50 Pool-fiber0] Collectd: scheduleInterface: Unable to schedule 10.24.53.191 for service SNMP-JVM, reason: Interface 10.24.53.191 is not the primary SNMP interface for nodeid 1 2007-10-17 13:13:34,627 DEBUG [CollectdScheduler-50 Pool] Scheduler: run: scheduler exiting, state = STOPPED What does "Interface 10.24.53.191 is not the primary SNMP interface for nodeid 1" mean and how can it be resolved.
Comment posted by: Francesco, 16 years ago

I am edited caps.log and obtain follows listen:

testing plugin SNMP-JVM

protocol SNMP-JVM supported? false

plugin SNMP-JVM completed!

The problem is who the services is not discovered!!!!!!!

Regards

Francesco

Comment posted by: maframan, 16 years ago

2007-08-24 17:50:01,397 DEBUG [Capsd Rescan Pool-fiber0] IfCollector: 192.168.47.12 testing plugin SNMP-JVM
2007-08-24 17:50:01,431 DEBUG [Capsd Rescan Pool-fiber0] IfCollector: 192.168.47.12 protocol SNMP-JVM supported? false
2007-08-24 17:50:01,431 DEBUG [Capsd Rescan Pool-fiber0] IfCollector: 192.168.47.12 plugin SNMP-JVM completed!
2007-08-24 17:53:40,487 DEBUG [Main] JdbcCapsdDbSyncer: syncServicesTable: checking protocol 'SNMP-JVM'.

 

What is the problem?

Help me!!!!!

Comment posted by: , 16 years ago

Hey Vince. Thanks for letting me know about the problem. Somehow all the escape characters got dropped when I first posted the blog. I've put them back in now.

Comment posted by: vince, 16 years ago

It works after I added "\\n" to the snmp-graph configuration:

GPRINT:max:MAX:"Max : %5.2lf %s \\n"

...
GPRINT:threads:MAX:"Max  : %8.2lf %s \\n"
Comment posted by: , 16 years ago

Hi homerlex. Do you have the same collectd-configuration used in the blog above? You may be missing the oid parameter. Also check that this oid is available when you use snmpwalk. Make sure all your oids in all config files begin with a dot. Finally, have you checked logs/collectd.log for errors?

Comment posted by: homerlex, 16 years ago

I'm having some trouble with this as well.  A couple things I had to add to get it closer to working:

In snmp-config I had to add something a definition like:

definition version="v2c" port="1161" - and I added a "specific" tag to my server_ip

And in collectd-configuration I had to add a collector service def:

collector service="SNMP-JVM"       class-name="org.opennms.netmgt.collectd.SnmpCollector"/

 At this point I see the SNMP-JVM service for my node in the web interface but no data is being collected.  snmpwalk runs fine, I'm not seeing any errors - I just don't know why data is not being collected.

Comment posted by: , 16 years ago

Hey Vince. This should definitely work (I rely on this data heavily!). If you post your problem to the opennms-discuss list I'll be able to reply by email even if I can't post to the list.

Comment posted by: Vince, 17 years ago
Hi, I tried this but i got no luck. Does this really work?
Comment posted by: , 17 years ago

There were a few questions on opennms-discuss about this blog, which I did try to answer, but it seems that the gmane gateway for this list is broken:

<discuss@lists.opennms.org>: mail for lists.opennms.org loops back to myself

Sorry!