PermGen errors on Storage Nodes

Question

The error in itself is....
OutOfMemoryError: PermGen space
This occurs simultaneously on all storage nodes after the cluster has been up a number of hours. Cluster is configured as partitioned and actual data size is only being reported as about 12Mb through RTView.
We're running 3 storage nodes at 4096mb each and 3 proxies of 2048m. We are NOT specifically setting the PermSize (i.e. not setting -XX:PermSize=).
Coherence Version = 3.5.3
JVM is Hotspot 64bit on windows server 2008........
java version "1.6.0_21"
Java(TM) SE Runtime Environment (build 1.6.0_21-b07)
Java HotSpot(TM) 64-Bit Server VM (build 17.0-b17, mixed mode)
Machine is an HP blade (8 core [16 core with Hyperthreads] and 24Gb memoy) so we are way off the limits in terms of object data storage.
Possible fixes - we are going to set =XX:PermSize=512m and probably move to JRockit however I'd like to know if anyone has experienced this and how they fixed it.
Edited by: Kunal on Jan 23, 2012 8:55 PM

Answer 1

Kunal wrote:
The error in itself is....
OutOfMemoryError: PermGen space
This occurs simultaneously on all storage nodes after the cluster has been up a number of hours. Cluster is configured as partitioned and actual data size is only being reported as about 12Mb through RTView.
We're running 3 storage nodes at 4096mb each and 3 proxies of 2048m. We are NOT specifically setting the PermSize (i.e. not setting -XX:PermSize=).
Coherence Version = 3.5.3
JVM is Hotspot 64bit on windows server 2008........
java version "1.6.0_21"
Java(TM) SE Runtime Environment (build 1.6.0_21-b07)
Java HotSpot(TM) 64-Bit Server VM (build 17.0-b17, mixed mode)
Machine is an HP blade (8 core [16 core with Hyperthreads] and 24Gb memoy) so we are way off the limits in terms of object data storage.
Possible fixes - we are going to set =XX:PermSize=512m and probably move to JRockit however I'd like to know if anyone has experienced this and how they fixed it.
Edited by: Kunal on Jan 23, 2012 8:55 PMThis can be caused by a few things, but JVM implementation most likely will not make a difference (except JRockit may give you something to diagnose this with), although someone mentioned a bug which was JVM implementation specific and was causing PermGen OOM.
You should get some memory snapshots at the time of OutOfMemory (look up the documentation for the exact JVM option).
Afterwards look at how many instances what classes sit in the Perm generation (I believe, Visual VM should be able to help you with this), that may give you more information on what is polluting the perm gen.
It may simply be too many classes loaded (increase this to a sane value, I saw JVMs used in production which needed around 80-100Mb PermGen). Half a gig seems to be too much.
Other things which use PermGen is intern()-ed Strings. There are a few libraries which used a horrible and what is worse, unbounded amount of intern()-ed strings, there was at least one XML processing library suffering from this.
Best regards,
Robert