App Server Freeze - All threads waiting for replication data

Question

          We are running Weblogic 6.1 SP2, on solaris 8 with jdk1.3 When running load at our
          application - full GC is taking between 15 -20 seconds. At a certain point after
          a full GC on one of the servers in the cluster we see session replication errors
          in the logs. The application eventually freezes - the execute queue is huge on each
          of the JVMS and when we do a thread dump we see that all threads are waiting for
          replication data. All threads have the same stack trace as follows:
          "ExecuteThread: '1' for queue: 'default'" daemon prio=5 tid=0x2c58b0 nid=0xe waiting
          on monitor [0xbf980000..0xbf9819d8]
               at java.lang.Object.wait(Native Method)
               at weblogic.rjvm.ResponseImpl.waitForData(ResponseImpl.java:72)
               at weblogic.rjvm.ResponseImpl.getTxContext(ResponseImpl.java:97)
               at weblogic.rmi.internal.BasicOutboundRequest.sendReceive(BasicOutboundRequest.java:80)
               at weblogic.rmi.internal.BasicRemoteRef.invoke(BasicRemoteRef.java:135)
               at weblogic.rmi.internal.ProxyStub.invoke(ProxyStub.java:35)
               at $Proxy74.lookupROIDS(Unknown Source)
               at weblogic.servlet.internal.HttpServer.lookupROIDS(HttpServer.java:761)
               at weblogic.servlet.internal.session.ReplicatedSessionContext.getSessionInternal(ReplicatedSessionContext.java:225)
               at weblogic.servlet.internal.ServletRequestImpl.getValidSession(ServletRequestImpl.java:2130)
               at weblogic.servlet.internal.ServletRequestImpl.getSession(ServletRequestImpl.java:1921)
               at weblogic.servlet.security.internal.ServletSecurityManager.checkAccess(ServletSecurityManager.java:107)
               at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:2467)
               at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2204)
               at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:139)
               at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:120)
          

Answer 1

          Yes this is reproducable. We are now working with BEA Support under Case : 321199.
          Thanks
          Anna
          Kumar Allamraju <[email protected]> wrote:
          >You should work with [email protected]
          >Can this problem be reproduced with an isolated test case?
          >If yes, it will be really helpful for them to work with Engg.
          >
          >--
          >Kumar
          >
          >anna bancroft wrote:
          >
          >> We are running Weblogic 6.1 SP2, on solaris 8 with jdk1.3 When running
          >load at our
          >> application - full GC is taking between 15 -20 seconds. At a certain
          >point after
          >> a full GC on one of the servers in the cluster we see session replication
          >errors
          >> in the logs. The application eventually freezes - the execute queue is
          >huge on each
          >> of the JVMS and when we do a thread dump we see that all threads are waiting
          >for
          >> replication data. All threads have the same stack trace as follows:
          >> "ExecuteThread: '1' for queue: 'default'" daemon prio=5 tid=0x2c58b0 nid=0xe
          >waiting
          >> on monitor [0xbf980000..0xbf9819d8]
          >>      at java.lang.Object.wait(Native Method)
          >>      at weblogic.rjvm.ResponseImpl.waitForData(ResponseImpl.java:72)
          >>      at weblogic.rjvm.ResponseImpl.getTxContext(ResponseImpl.java:97)
          >>      at weblogic.rmi.internal.BasicOutboundRequest.sendReceive(BasicOutboundRequest.java:80)
          >>      at weblogic.rmi.internal.BasicRemoteRef.invoke(BasicRemoteRef.java:135)
          >>      at weblogic.rmi.internal.ProxyStub.invoke(ProxyStub.java:35)
          >>      at $Proxy74.lookupROIDS(Unknown Source)
          >>      at weblogic.servlet.internal.HttpServer.lookupROIDS(HttpServer.java:761)
          >>      at weblogic.servlet.internal.session.ReplicatedSessionContext.getSessionInternal(ReplicatedSessionContext.java:225)
          >>      at weblogic.servlet.internal.ServletRequestImpl.getValidSession(ServletRequestImpl.java:2130)
          >>      at weblogic.servlet.internal.ServletRequestImpl.getSession(ServletRequestImpl.java:1921)
          >>      at weblogic.servlet.security.internal.ServletSecurityManager.checkAccess(ServletSecurityManager.java:107)
          >>      at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:2467)
          >>      at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2204)
          >>      at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:139)
          >>      at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:120)
          >>
          >>
          >