Heap Memory Issue in weblogic 9.2 for a JSF 1.1 web application

Question

Hi,
We are running a JSF application (Myfaces, facelets, tomahawk, richfaces & iBATIS) in weblogic 9.2 server on Solaris 10. This application is deployed in production and works fine under normal circumstances. But when there is a heavy user load we are facing a memory issue. The memory usage is gradually increasing and when it reaches to max, Full GC kicks in again & again which choks up all requests. We don't save anything in session scope. All our backing beans are saved in request scope hence they should be garbage collected after each request done, but this is not happening.
We took the heap dump from production after this issue and analyzed it. After my analysis, I found all objects which are set in request object not being garbage collected and the root referers of all these objects is weblogic.servlet.internal.MuxableSocketHTTP.
I reproduced the similar behaviour in one of our development environment using JMeter. I ran 100 concurrent users in JMeter for almost 1 hour and saw the similar behaviour. Below is the result of all weblogic objects which are still hanging in heap after test was over (I also ran manual Garbage Collector from admin server).
1) weblogic.servlet.internal.MuxableSocketHTTP - 1774 objects - retained heap (1 GB)
2) weblogic.servlet.internal.ServletRequestImpl - 1774 objects - retained heap (1 GB)
My understanding is that every request made to weblogic server goes through the MuxableSocketHTTP object which creates the ServletRequestImpl to serve it. Once the request is served these objects are suppose to be removed. As a result of that whatever is saved in your request will still be hanging.
I am not able to understand why these objects are hanging after request is done. Could anybody answer to my question. I appreciate your help in advance.
The GC setting for weblogic server while startup is:
-XX:MaxTenuringThreshold=15 -XX:+PrintTenuringDistribution -XX:+AggressiveHeap -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:PermSize=128m -XX:MaxPermSize=128m -Xms3g -Xmx3g -XX:NewSize=512m -XX:MaxNewSize=1024m
Thanks MaKK

Answer 1

What happened with this issue? We are seeing something similar on WebLogic 9.2 MP1 in Solaris (Jdk 1.5. patch 10. 32 bit). Out of Memory's with thousands if instances of weblogic.socket.MuxableSocket hanging around.
Our thinking was initally the Java heap, then we thought that maybe the sockets weren't being closed properly, possible in WebLogic or in LiveCycle.
Any info would be greatly appreciated.
Snippet of our stack trace:
<16-Feb-2010 04:30:13 o'clock GMT> <Error> <Kernel> <BEA-000802> <ExecuteRequest failed
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
javax.ejb.EJBException: EJB encountered System Exception: : java.lang.OutOfMemoryError: Java heap space
     at weblogic.ejb.container.internal.EJBRuntimeUtils.throwEJBException(EJBRuntimeUtils.java:145)
     at weblogic.ejb.container.internal.BaseLocalObject.postInvokeCleanup(BaseLocalObject.java:550)
     at weblogic.ejb.container.internal.BaseLocalObject.postInvokeCleanup(BaseLocalObject.java:496)
     at com.adobe.idp.um.businesslogic.directoryservices.DirectorySynchronizationManagerBean_f5g74_ELOImpl.synchronizeProviders(DirectorySynchronizationManagerBean_f5g74_ELOImpl.java:267)
Joel