How to diagnose increase in permanent generation?

Question

Hello,
My team is observing the following problem which is important to us and that we
did not yet manage to identify:
We observe from the gc traces (see extracts below) that the permanent memory
size keeps increasing slowly (to the point were an java.lang.OutOfMemoryError is
thrown) but the -verbose:class traces do not show anymore class loading after
the system "warmup" phase. This was observed using Sun's JDK version 1.4.1.01.04 on
HPUX, with the following JVM options java -verbose:gc
-Xverbosegc:file=/tmp/gc.log -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
-XX:+PrintHeapAtGC -Xms256M -Xmx256M -XX:NewSize=16M
-XX:MaxNewSize=16M -XX:SurvivorRatio=2 -XX:+UseConcMarkSweepGC
I would need help to answer the following questions:
1- what other actions than class loading could possibly explain an increase of
used space in the permanent generation? The GC FAQ says "The permanent
generation is used to hold reflective of the VM itself such as class objects and
method objects". Could use of JNI also affect this space?
2- how can I configure or instrument the JVM 1.4.1 to track cause of increase of
used space in the permanent generation? The program we are testing uses 500 MB
RAM and we often lack physical space on the server to make use of standard
profiler tools such as JProbe or -hprof standard JVM option: HPUX JVM complains
about insufficient MAXDSIZE kernel parameters that is already set to 512 MB, and
plenty swap space is available)
3- In which version are the "-XX:+TraceClassloading and
-XX:+TraceClassUnloading" options supported? I read from
http://java.sun.com/docs/hotspot/gc1.4.2/faq.html that the options
"-XX:+TraceClassloading and -XX:+TraceClassUnloading" should trace class
loading, however they don't seem available under JDK 1.4.1 on Windows, and they
are not either documented in the toolsdocs for JDK 1.4.2
4- What does -verbose:class option really shows? The toolsdoc says
"-verbose:class Display information about each class loaded.". Would this also
display classes dynamically generated such as dynamic proxies (or classes
loading from custom classloaders)? I suspect this only displays class loaded
from the system classloader...
Many thanks for your help,
Regards,
Guillaume Berche.
19735.5: [DefNew: 10511K->2379K(12288K), 0.1241661 secs] 160415K->152624K(258048K) Heap after GC invocations=11590:
Heap
def new generation total 12288K, used 2379K [77c00000, 78c00000, 78c00000)
eden space 8192K, 0% used [77c00000, 77c00000, 78400000)
from space 4096K, 58% used [78400000, 78652e48, 78800000)
to space 4096K, 0% used [78800000, 78800000, 78c00000)
concurrent mark-sweep generation total 245760K, used 150245K [68c00000, 77c00000, 77c00000)
CompactibleFreeListSpace space 245760K, 61% used [68c00000, 77c00000)
concurrent-mark-sweep perm gen total 43772K, used 33105K [64c00000, 676bf000, 68c00000)
CompactibleFreeListSpace space 43772K, 75% used [64c00000, 676bf000)
} , 0.1262338 secs]
19737: [GC  {Heap before GC invocations=11590:
Heap
def new generation   total 12288K, used 10571K [77c00000, 78c00000, 78c00000)
  eden space   8192K, 100% used [77c00000, 78400000, 78400000)
  from space   4096K,  58% used [78400000, 78652e48, 78800000)
  to   space   4096K,   0% used [78800000, 78800000, 78c00000)
concurrent mark-sweep generation total 245760K, used 150245K [68c00000, 77c00000, 77c00000)
CompactibleFreeListSpace space 245760K,  61% used [68c00000, 77c00000)
concurrent-mark-sweep perm gen total 43772K, used 33106K [64c00000, 676bf000, 68c00000)
CompactibleFreeListSpace space  43772K,  75% used [64c00000, 676bf000)
} , 0.1262338 secs]

Answer 1

I did check the JDK bug database, and you are right, String.intern() does indeed seem to increase size of permanent generation (argnn!)
Have you experienced cases in which "internalised" String are prevented by the JVM 1.4.1 from being GCed?
I saw an old bug on JDK 1.1 similar to that:
http://developer.java.sun.com/developer/bugParade/bugs/4497186.html
Thanks for your help,
Guillaume.