Web application Unloading classes of WL Stubs every several hours

Question

Hello,
We have a WEB applications, which connects to Server Application using EJB technology - both deployed on the same
domain on Weblogic 10.
Recently, we started to experience a strange phenomenon - WEB application's threads get stalled and get a STUCK
state.
Sometimes, it is accompanied with the java.lang.OutOfMemoryError exception.
Only reboot helps to recover from it.
Trying to identify a problem we added some parameters to get some more information in the application logs.
What we found is that once in a while (depend on a system load), we see the following messages in console logs:
[Unloading class <class name>]
These are the examples about how often the events happen and it’s durations:
2011-01-30 10:14:11 - 2011-01-30 10:14:28
2011-01-30 12:27:51 - 2011-01-30 12:28:08
2011-01-30 13:59:46 - 2011-01-30 14:00:02
2011-01-30 14:56:59 - 2011-01-30 14:57:13
2011-01-30 14:57:46 - 2011-01-30 14:58:00
2011-01-30 17:11:21 - 2011-01-30 17:11:39
2011-01-30 18:29:24 - 2011-01-30 18:29:44
2011-01-30 20:05:34 - 2011-01-30 20:05:53
2011-01-30 21:48:39 - 2011-01-30 21:48:55
. - Each time are about 20000 - 30000 classes are unloaded.
. - About 150000 classes are unloaded daily,
........- 90% out of it are the classes, which ends with "*..Impl_1001_WLStub*"
............- 30% out of Stubs are Home Stubs, which ends with "*..HomeImpl_1001_WLStub*"
*(!) The most interesting that out of all this huge amount of classes, which are unloaded daily,*
only +23+ types of different Stubs are unloaded, whereas out of it +10+ different Home Stubs:
$> grep "Unloading class.*Impl_1001_WLStub" Web1.out | sort -u | wc -l
23
$> grep "Unloading class.*HomeImpl_1001_WLStub" Web1.out | sort -u | wc -l
10Sometimes (as I said above) it ends with the "+java.lang.OutOfMemoryError+" exception.
So, the questions are:
1. What could be a reason for such behavior of a system?
2. What message "Unloading class" says?
3. Is it normal that (as you see from above), the same type of Stubs are loaded?
It looks like a new Stub is created for every EJB invocation, isn't it?
Is it normal? If not, what could be a reason for this?
--------------- S Y S T E M ---------------
jvm_args: -Xms2048m -Xmx2048m -XX:MaxPermSize=512m
OS: Solaris 10 5/08 s10s_u5wos_10 SPARC
uname:SunOS 5.10 Generic_142900-02 sun4v (T2 libthread)
rlimit: STACK 8192k, CORE 1024000k, NOFILE 65536, AS infinity
CPU:total 64 has_v8, has_v9, has_vis1, has_vis2, is_ultra3, is_sun4v, is_niagara1
Memory: 8k page, physical 66977792k
vm_info: Java HotSpot(TM) Server VM (1.5.0_11-b03) for solaris-sparc, built on Dec 15 2006 01:18:11

Answer 1

You could see how your memory is managed by the JVM, by using for example a tool such as jconsole.
Unloading class means that loaded classes are garbage collected. To see when the classes are loaded
you can use the JVM option -verbose:class. I think somehow you are constantly loading the classes in the application,
maybe by creating the stub over and over again when you need it. You can create the stub once by using
a singleton pattern.
A good overview of the available tools for the Sun JVM is given here: http://www.oracle.com/technetwork/java/javase/index-137495.html