Using SiteScope 9.5 to Monitor Weblogic 10.3

Posted by on March 22, 2010
Filed Under Performance Engineering | 4 Comments

I recently tried to set up SiteScope 9.5 to monitor Weblogic Portal.  It was not a pleasant experience.  There are posts in various blogs about setting up JMX based monitoring of Weblogic.  Some of the information is older, inaccurate, or didn’t apply to my configuration.  So I’m consolidating the relevant information to my situation together here.  

Note: HP does not support monitoring Weblogic 10.3 with SiteScope 9.5.  They recommend upgrading to SiteScope 10.10.  I was able to monitor with the steps below, but it may not work in all cases.  I had no problem running these JMX monitors for Weblogic 10.3 and Linux memory and CPU monitors together.  But it is possible that the wlfullclient.jar will cause a problem with other monitors. 

Here is my configuration.

These are the steps I needed.  Links to the posts that helped me out are at the bottom.

Create a User for Monitoring

This step isn’t required.  You can use the sytem user if you like, but  I prefer to set up a separate user for monitoring so the system user’s password can be modified without impacting monitoring.  It is a better security practice, and can be helpful in troubleshooting issues as well.  To set up the user, go to the Security Realms/myrealm/Users and Groups/Users page and create a new user.  You need to also put the user in the “Monitors” Weblogic group so the user has access to read the MBeans.

Enable IIOP and set the Default IIOP Username / Password

The key point here is that if you don’t set the Default IIOP Username and Password you will get the error message “org.omg.CORBA.NO_PERMISSION” when you try to connect.  You do not have to actually connect as this user when your monitoring session authenticates.  Your monitoring user can be different, but this entry needs to be there, and it must be a valid username/password.  I used the monitoring user that I set up in the first step.  This is done by going to the Weblogic server configuration/Protocols/IIOP.  Select to Enable IIOP, click Advanced, and set the “Default IIOP Username” and “Default IIOP Password”.

Upgrade the Java version to 1.6

In order to monitor Weblogic 10.3 that is running in a Java VM version 1.6, you must upgrade the JVM that is running Sitescope to 1.6.  Sitescope 9.53 seems to run fine on a Sun 1.6 version of the JVM.  After you download and install a 1.6 JVM, of course you have to update your JAVA_HOME and registry settings for Sitescope to get it to use the new JVM. 

Put the wlfullclient.jar on the SiteScope’s Classpath

This was the biggest issue I was having.  Weblogic 10.x uses additional JMX capabilities that are not provided by weblogic.jar.  You MUST use wlfullclient.jar to monitor Weblogic 10.x as discussed here http://download-llnw.oracle.com/docs/cd/E11035_01/wls100/client/jarbuilder.html#wp1077742.  If you don’t have the wlfullclient.jar,  SiteScope will still connect to Weblogic, and it appears to be working.  But, you won’t get the list of monitors, and the log will show the error: “java.rmi.MarshalException: CORBA MARSHAL 1398079745 Maybe; nested exception is:  org.omg.CORBA.MARSHAL:   vmcid: SUN  minor code: 257 completed: Maybe”.  You can modify the SiteScope classpath and put the jar file on it, or you can just put the wlfullclient.jar in the C:SiteScopejavalibext directory.  Either approach should work. 

Modify SiteScope Startup Parameters

If you are having CORBA timeout errors in the log, you can edit the registry entry “HKEY_LOCAL_MACHINE/SYSTEM/CurrentControlSet/Services/SiteScope/ServiceParam”.  That is the SiteScope startup command line.  You can increase the CORBA timeout settings by putting “-Dcom.sun.CORBA.transport.ORBTCPReadTimeouts=10:60000:500:10″ in the command line.  Connection timeout errors can be misleading however.  I was seeing them in the log even though the real issue was that I didn’t have the wlfullclient.jar file in the classpath.  Weblogic eventually closed the connection because it didn’t understand the request, and SiteScope logged it as a connection timeout.  If you see timeout errors, make the change above.  If you are still seeing them, you probably have some other issue. 

You can also increase the JVM heap size in the registry entry if you are seeing out of memory errors when pulling back the list of counters.

Create a JMX Monitor

Weblogic 9 and up use JMX for monitoring.  You cannot use the Weblogic SiteScope monitor.  You need to set up a JMX SiteScope monitor. 

The <managed server> can be a DNS name or the ip address.  One suggestion on the web is that you have to use the Machine Name that is configured in weblogic.  I didn’t find that to be the case.  Just using the ip address or the host name worked fine.  The port can be just the regular port number that the instance is listening on.  Some suggestions of using command line options to have weblogic listen for JMX on a separate port are not needed.  You can just use the normal listen port.  

The Domain Runtime MBean is only available on the Admin instance.  Oracle recommends monitoring the managed instances by connecting only to the Admin instance and using the Domain Runtime MBean.  It gives you a single place to go to configure all of your monitors.  I had trouble with the CORBA connection timing out when it was trying to pull back all of the MBean data for the Domain MBean, but was eventually able to get over that error by increasing the second value in the CORBA timeout string above from 60000 to 180000 and reducing the last value from 10 to 1.   After solving that issue,  I then got “ERROR – Failed to get class property details for property: availableCountersHierarchical in monitor type: JMXMonitor” in the SiteScope error log.  Doing a little research on-line, it sounds like some of the data coming back from the Domain MBean isn’t being understood by SiteScope, so it just quits.  Rather than continue trying to get monitoring from the Domain MBean working, I just set up a monitor for each managed instance separately.  I was able to get JConsole to query the data from the Domain MBean on the Admin instance, so connectivity and permissions were fine.  SiteScope just wasn’t able to work with the data coming back.

Select Your Counters

When you click on the counter button, the list of counters should return fairly quickly.  I had some out of memory conditions at times using the previous max memory setting on SiteScope when clicking on “Get Counters”.  The instance seemed to recover and display them after clicking again, but I increased the JVM memory to alleviate that issue.   Some of the counters we are using are listed below.  Of course <managed server name> would be the name of your managed server.

You’re Done

The monitors should work at this point

Other Monitoring Tools

These same basic steps can be used to monitor Weblogic 10.x with any JMX monitoring tool.  JConsole is much better than sitescope or the Weblogic Admin console for real time monitoring during performance tests.  SiteScope is good to monitor a few key health statistics on an on-going basis, but it isn’t granular enough or as useful for detailed monitoring during tuning or troubleshooting efforts. 

These are the posts that helped me get things working.

 

 Troubleshooting

We recently had problems when using Sun jdk 1.6.0 communicating with jrockit 1.5.0 running Weblogic 10.2.  We were getting the CORBA error below.  This appears to be a bug in IIOP with Sun JDKs as discussed in this bug report http://forums.sun.com/thread.jspa?threadID=5259072.  Using Sun jdk 1.5 to communicate with the jrockit jdk 1.5 worked fine.

Jun 22, 2010 2:13:03 PM com.sun.corba.se.impl.io.InputStreamHook throwOptionalDataIncompatibleException
WARNING: "IOP00800008: (MARSHAL) Not enough space left in current chunk"
org.omg.CORBA.MARSHAL: vmcid: OMG minor code: 8 completed: No
at com.sun.corba.se.impl.logging.OMGSystemException.rmiiiopOptionalDataIncompatible2(OMGSystemException.java:2709)
at com.sun.corba.se.impl.logging.OMGSystemException.rmiiiopOptionalDataIncompatible2(OMGSystemException.java:2731)
at com.sun.corba.se.impl.io.InputStreamHook.throwOptionalDataIncompatibleException(InputStreamHook.java:348)
at com.sun.corba.se.impl.io.InputStreamHook$InReadObjectPastDefaultsRemoteDidNotUseWOState.readData(InputStreamHook.java:342)
at com.sun.corba.se.impl.io.IIOPInputStream.read(IIOPInputStream.java:655)
at javax.management.MBeanInfo.readObject(MBeanInfo.java:671)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

 Some Tips

Some of the things I ran into are listed below.

 

 



Comments

4 Responses to “Using SiteScope 9.5 to Monitor Weblogic 10.3”

  1. xboxer21 on July 22nd, 2010 2:41 pm

    What registry keys do i need to modify for sitescope use java 1.6?

  2. Doug Baber on July 23rd, 2010 10:34 am

    We’ve upgraded to SiteScope 10.11 at this point, so I don’t have a 9.5 install to reference, but I believe it was in the same one mentioned in the section “Modify SiteScope Startup Parameters”. The registry key HKEY_LOCAL_MACHINE/SYSTEM/CurrentControlSet/Services/SiteScope/ServiceParam has the startup command line for SiteScope. The path to the java executable is in that key. You would need to change that path to the location of the 1.6 version that you installed. If there is a ServicePath key though, it would be that entry. That is the one that 10.x uses.

  3. ethanb on July 29th, 2010 12:12 pm

    Hi, I’m trying to set up a JMX monitor for an instance of Tomcat. I can get data from the JMX service with VisualVM, but not with SiteScope 8.9. It’s not a networking or permissions issue, because when I try to have SiteScope authenticate with the JMX service using an incorrect password, I get the response “Connection failed due to security error”. I checked sitescope’s error log, and I’m getting the same error you had mentioned above: “ERROR – Failed to get class property details for property: availableCountersHierarchical in monitor type: JMXMonitor”. I saw that because you were using WebLogic, you used the Managed Instance JMX URL instead of the Admin Instance URL and that fixed your problem. I am not familiar with WebLogic at all, so I was wondering if you knew of an equivalent Tomcat solution for this problem. The service URL format I am using in VisualVM is service:jmx:rmi:///jndi/rmi://(host):(post)/jmxrmi

  4. Doug Baber on July 29th, 2010 4:42 pm

    I don’t have a SiteScope 8.9 installation to try to replicate your issue. I was able to use JConsole that comes with Sun JDK 1.5 to monitor a Tomcat 5.5 instance that was running on Sun JDK 1.5 using the url that you gave in your post. All I had to do was turn on remote monitoring on the Tomcat instance using the command line options

    set CATALINA_OPTS=-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=7890 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false

    Those options turn off authentication and SSL for remote monitoring. Once you got it working, you could turn them back on.

    I’m guessing that your issue is related to the version of the JVM that SiteScope is running on vs. the Tomcat version. I haven’t fully researched it, but you probably need a minimum JVM version of 1.5 to use JMX, and your Tomcat version probably needs to be 5.5. There may be differences in the JMX classes if you are using 1.6 on one side and 1.5 on the other, and 1.4 probably won’t work at all. These are some of the URLs that I found useful for Tomcat JMX monitoring. Tomcat Doc and Sun Doc. I hope something here is helpful.