Analyzing Memory Leaks in JRockit Behind a Firewall

Posted by on June 2, 2010
Filed Under Performance Engineering | Comments Off

We were recently trying to identify the source of a memory leak in a Weblogic Portal 10.3 application that was using JRockit build R27.6 on a RedHat 5 Linux 64 bit.  We were performance testing the application in a production-like environment.  The Weblogic servers were behind a firewall, and this created problems using tools to analyze the leak.  We did not have access to the source code or have a local system that we could use tools on for the analysis. 

Oracle has a large number of JDK versions and tools with varying options for each release.  It is very confusing trying to figure out what options you have for the version you are running and how to use them.  Searching the Internet brings up documention about using the JRockit Memory Leak Detector to find leaks.  One set of Oracle documentation said you could run the Memory Leak Detector by typing memleak at the command prompt.  Searching the directories didn’t turn up that command for me, and the Oracle on-line documentation didn’t seem to give any indication about what version that was for, or where to find the file. 

JRockit Mission Control

We had previously downloaded JRockit Mission Control 3.1, which has the Memory Leak Detector as part of the application.  Reading the documentaion, this looked like a very nice tool and would tell us where the leak is.  It turns out that the tool is more suited for a development environment than a production environment that has servers behind firewalls.  To use Mission Control, you need to start your server JVM with the -Xmanagement command line option.  You can also specify a port for it to run on.  So far so good.  It sounds like you just need a firewall hole for the port you specified on the command line.  We tried running Mission Control and giving it the port that we specified, but we couldn’t get Mission Control to connect.  We could see the socket connection on the server open up on that port, but Mission Control would still say it couldn’t connect.  Checking the FAQ in the Mission Control Tool (we didn’t see any mention of this on the Internet), Mission Control needs to open up a second random port to complete the connection.  This rendered the tool useless for us.  There is a configuration option in the Preferences pane for the Memory Leak detector that allows you to specify a port, and the documentation indicates that this is the setting to use if you are behind a firewall.  However, whenever we ran the Memory Leak detector, it seemed to ignore this setting and use the port number in the main connection configuration.  It apparently still needed the second, random port and did not connect.

Another possible option for running the Mission Control would be to run it on the server that we were trying to analyze.  Even though the software was installed with the default JRockit installation, we were unable to run it.  The server didn’t have the correct graphics libraries installed to handle the GUI display since it is a production app server.  Even if the correct libraries were installed, we would not have been able to run X Windows to display the GUI on our desktop since the X11 protocol is blocked by the firewall for security reasons. 

We also found documentation about running JRockit Management Console in “headless” mode.  It looks like there was a version in the past that had the ability to run without the GUI and ouput its data to a file.  It descibed the startup process as executing the ManagementConsole.jar.  However, we couldn’t find a ManagementConsole.jar on any of the installations that we had, so this feature doesn’t seem to still be available.

HProf Dump

We have used .hprof dumps on several occassions in the past to solve memory leaks using Sun JVMs.  Since the GUI tools provided with JRockit weren’t an option for us, we next tried to get an old fashioned heap dump and do off-line analysis on it.  We searched for documentation on JRockit heap dumps with similar commands that we had used for Sun JVM’s in the past.  This turned up documentation from Oracle with several promising -XX options such as -XX:+HeapDumpOnCtrlBreak, -XX:HeapDumpPath, and -XX:+HeapDumpOnOutOfMemory.  It turned out those options were only available on R28 and not on the version we were running, R27.  If you are running R28, here is the documentation for the command line options  http://download.oracle.com/docs/cd/E15289_01/doc.40/e15062/optionxx.htm#BABIDFGC

 jrcmd

We eventually identified a tool that would get us the information we needed from the R27 version of JRockit.  There is a command line tool that is part of the JRockit install that allows you to send diagnostics commands to a running JRockit JVM process.  It is lacking in that it doesn’t have the ability to generate an .hprof dump, but it has several very cool features that we haven’t seen with other tools.  For instance, it has the ability to report on the memory usage of the C-Heap in the JVM process and give a thread dump of the C-level  threads.  This would have come in handy for diagnosing memory leaks in JNI code that we have encountered in the past.  For this application, we were able to get the information that we needed by executing the print_object_summary and heap_diagnostics command at several intervals during a load test.  Below are some useful references.



Comments

Comments are closed.