Help on stuck thread
Hi, From my server log I noticed that there are some stuck threads. After looked at the thread dump thread 18 is blocking. MainPortalServlet.java is extends PortalServlet and line 96 is super.service(request,response); Please help me to analyse why it is blocking. Thanks.
"ExecuteThread: '20' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x0090e6b0 nid=0x3b waiting for monitor entry [0x600fd000..0x600ffc28]
at com.bea.wlw.netui.script.el.util.ParseUtils.parse(ParseUtils.java:100)
- waiting to lock <0x8d3a3f18> (a com.bea.wlw.netui.script.el.util.ParseUtils$Cache)
at com.bea.wlw.netui.script.el.ExpressionEvaluatorImpl.containsExpression(ExpressionEvaluatorImpl.java:239)
at com.bea.wlw.netui.pageflow.ProcessPopulate.populate(ProcessPopulate.java:295)
at com.bea.wlw.netui.pageflow.PageFlowRequestProcessor.processPopulate(PageFlowRequestProcessor.java:402)
at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:254)
at com.bea.wlw.netui.pageflow.PageFlowRequestProcessor.process(PageFlowRequestProcessor.java:691)
at com.bea.wlw.netui.pageflow.AutoRegisterActionServlet.process(AutoRegisterActionServlet.java:527)
at com.bea.wlw.netui.pageflow.PageFlowActionServlet.process(PageFlowActionServlet.java:152)
at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:507)
at com.bea.wlw.netui.pageflow.PageFlowUtils.strutsLookup(PageFlowUtils.java:1786)
at com.bea.portlet.adapter.scopedcontent.ScopedContentCommonSupport.executeAction(ScopedContentCommonSupport.java:561)
at com.bea.portlet.adapter.scopedcontent.ScopedContentCommonSupport.processActionInternal(ScopedContentCommonSupport.java:121)
at com.bea.portlet.adapter.scopedcontent.PageFlowStubImpl.processAction(PageFlowStubImpl.java:98)
at com.bea.netuix.servlets.controls.content.NetuiContent.raiseScopedAction(NetuiContent.java:154)
at com.bea.netuix.servlets.controls.content.NetuiContent.raiseScopedAction(NetuiContent.java:113)
at com.bea.netuix.servlets.controls.content.NetuiContent.handlePostbackData(NetuiContent.java:228)
at com.bea.netuix.nf.ControlLifecycle$3.visit(ControlLifecycle.java:171)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:356)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:126)
at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:105)
at com.bea.netuix.nf.Lifecycle.runInbound(Lifecycle.java:173)
at com.bea.netuix.nf.Lifecycle.run(Lifecycle.java:137)
at com.bea.netuix.servlets.manager.UIServlet.runLifecycle(UIServlet.java:324)
at com.bea.netuix.servlets.manager.UIServlet.processControlTree(UIServlet.java:220)
at com.bea.netuix.servlets.manager.PortalServlet.doPost(PortalServlet.java:820)
at com.bea.netuix.servlets.manager.UIServlet.service(UIServlet.java:150)
at com.cd.servlets.portal.MainPortalServlet.service(MainPortalServlet.java:96)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:28)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
at com.bea.p13n.servlets.PortalServletFilter.doFilter(PortalServletFilter.java:293)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:7053)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3902)
at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2773)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)
"ExecuteThread: '19' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x0090dab8 nid=0x3a runnable [0x601fb000..0x601ffc28]
at java.util.HashMap.getEntry(HashMap.java:361)
at java.util.LinkedHashMap.get(LinkedHashMap.java:266)
at com.bea.wlw.netui.script.el.util.ParseUtils.parse(ParseUtils.java:84)
at com.bea.wlw.netui.script.el.util.ParseUtils.evaluate(ParseUtils.java:123)
at com.bea.wlw.netui.script.el.ExpressionEvaluatorImpl.evaluateStrict(ExpressionEvaluatorImpl.java:87)
at com.bea.wlw.netui.tags.AbstractBaseTag.evaluateExpressionInternal(AbstractBaseTag.java:573)
at com.bea.wlw.netui.tags.AbstractBaseTag.evaluateExpression(AbstractBaseTag.java:362)
at com.bea.wlw.netui.tags.databinding.script.GetData.doEndTag(GetData.java:249)
at jsp_servlet._portlets.___portlet1._jspService(_portlet1.jsp:27)
at weblogic.servlet.jsp.JspBase.service(JspBase.java:33)
at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:28)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
at com.bea.p13n.servlets.PortalServletFilter.doFilter(PortalServletFilter.java:293)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
at com.bea.wlw.netui.pageflow.PageFlowJspFilter.doFilter(PageFlowJspFilter.java:250)
- locked <0x78ccf6c8> (a portlets.Portlet1Controller)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:652)
at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:431)
at com.bea.wlw.netui.pageflow.scoping.internal.ScopedRequestDispatcher.include(ScopedRequestDispatcher.java:120)
at com.bea.netuix.servlets.controls.content.JspContent.beginRender(JspContent.java:534)
at com.bea.netuix.servlets.controls.content.NetuiContent.beginRender(NetuiContent.java:425)
at com.bea.netuix.nf.ControlLifecycle$1.visit(ControlLifecycle.java:495)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:543)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:247)
at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:105)
at com.bea.netuix.nf.Lifecycle.run(Lifecycle.java:356)
at com.bea.netuix.nf.UIControl.render(UIControl.java:536)
at com.bea.netuix.servlets.controls.PresentationContext.render(PresentationContext.java:405)
at com.bea.netuix.servlets.util.RenderToolkit.renderChild(RenderToolkit.java:123)
at com.bea.netuix.servlets.jsp.taglib.RenderChild.doStartTag(RenderChild.java:58)
at jsp_servlet._framework._skeletons._cd.__flowlayout._jspService(flowlayout.jsp:30)
at weblogic.servlet.jsp.JspBase.service(JspBase.java:33)
at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:348)
at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:646)
at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:431)
at com.bea.netuix.servlets.controls.JspRenderer.renderAlt(JspRenderer.java:194)
at com.bea.netuix.servlets.controls.JspRenderer.beginRender(JspRenderer.java:96)
at com.bea.netuix.nf.ControlLifecycle$1.visit(ControlLifecycle.java:491)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:543)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:247)
at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:105)
at com.bea.netuix.nf.Lifecycle.run(Lifecycle.java:356)
at com.bea.netuix.nf.UIControl.render(UIControl.java:536)
at com.bea.netuix.servlets.controls.PresentationContext.render(PresentationContext.java:405)
at com.bea.netuix.servlets.util.RenderToolkit.renderChild(RenderToolkit.java:123)
at com.bea.netuix.servlets.jsp.taglib.RenderChild.doStartTag(RenderChild.java:58)
at jsp_servlet._framework._skeletons._cd.__flowlayout._jspService(flowlayout.jsp:30)
at weblogic.servlet.jsp.JspBase.service(JspBase.java:33)
at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:348)
at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:646)
at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:431)
at com.bea.netuix.servlets.controls.JspRenderer.renderAlt(JspRenderer.java:194)
at com.bea.netuix.servlets.controls.JspRenderer.beginRender(JspRenderer.java:96)
at com.bea.netuix.nf.ControlLifecycle$1.visit(ControlLifecycle.java:491)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:543)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:247)
at com.bea.netuix.nf.Lifecycle.runOutbound(Lifecycle.java:204)
at com.bea.netuix.nf.Lifecycle.run(Lifecycle.java:153)
at com.bea.netuix.servlets.manager.UIServlet.runLifecycle(UIServlet.java:324)
at com.bea.netuix.servlets.manager.UIServlet.processControlTree(UIServlet.java:220)
at com.bea.netuix.servlets.manager.PortalServlet.doPost(PortalServlet.java:820)
at com.bea.netuix.servlets.manager.PortalServlet.doGet(PortalServlet.java:671)
at com.bea.netuix.servlets.manager.UIServlet.service(UIServlet.java:147)
at com.cd.servlets.portal.MainPortalServlet.service(MainPortalServlet.java:116)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:28)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
at com.bea.p13n.servlets.PortalServletFilter.doFilter(PortalServletFilter.java:293)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:7053)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3902)
at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2773)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)
"ExecuteThread: '18' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x0090cec0 nid=0x39 runnable [0x602fd000..0x602ffc28]
at java.util.LinkedHashMap.transfer(LinkedHashMap.java:224)
at java.util.HashMap.resize(HashMap.java:452)
at java.util.LinkedHashMap.addEntry(LinkedHashMap.java:399)
at java.util.HashMap.put(HashMap.java:392)
at com.bea.wlw.netui.script.el.util.ParseUtils.parse(ParseUtils.java:100)
- locked <0x8d3a3f18> (a com.bea.wlw.netui.script.el.util.ParseUtils$Cache)
at com.bea.wlw.netui.script.el.ExpressionEvaluatorImpl.containsExpression(ExpressionEvaluatorImpl.java:239)
at com.bea.wlw.netui.pageflow.ProcessPopulate.populate(ProcessPopulate.java:295)
at com.bea.wlw.netui.pageflow.PageFlowRequestProcessor.processPopulate(PageFlowRequestProcessor.java:402)
at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:254)
at com.bea.wlw.netui.pageflow.PageFlowRequestProcessor.process(PageFlowRequestProcessor.java:691)
at com.bea.wlw.netui.pageflow.AutoRegisterActionServlet.process(AutoRegisterActionServlet.java:527)
at com.bea.wlw.netui.pageflow.PageFlowActionServlet.process(PageFlowActionServlet.java:152)
at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:507)
at com.bea.wlw.netui.pageflow.PageFlowUtils.strutsLookup(PageFlowUtils.java:1786)
at com.bea.portlet.adapter.scopedcontent.ScopedContentCommonSupport.executeAction(ScopedContentCommonSupport.java:561)
at com.bea.portlet.adapter.scopedcontent.ScopedContentCommonSupport.processActionInternal(ScopedContentCommonSupport.java:121)
at com.bea.portlet.adapter.scopedcontent.PageFlowStubImpl.processAction(PageFlowStubImpl.java:98)
at com.bea.netuix.servlets.controls.content.NetuiContent.raiseScopedAction(NetuiContent.java:154)
at com.bea.netuix.servlets.controls.content.NetuiContent.raiseScopedAction(NetuiContent.java:113)
at com.bea.netuix.servlets.controls.content.NetuiContent.handlePostbackData(NetuiContent.java:228)
at com.bea.netuix.nf.ControlLifecycle$3.visit(ControlLifecycle.java:171)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:356)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:126)
at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:105)
at com.bea.netuix.nf.Lifecycle.runInbound(Lifecycle.java:173)
at com.bea.netuix.nf.Lifecycle.run(Lifecycle.java:137)
at com.bea.netuix.servlets.manager.UIServlet.runLifecycle(UIServlet.java:324)
at com.bea.netuix.servlets.manager.UIServlet.processControlTree(UIServlet.java:220)
at com.bea.netuix.servlets.manager.PortalServlet.doPost(PortalServlet.java:820)
at com.bea.netuix.servlets.manager.UIServlet.service(UIServlet.java:150)
at com.cd.servlets.portal.MainPortalServlet.service(MainPortalServlet.java:96)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:28)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
at com.bea.p13n.servlets.PortalServletFilter.doFilter(PortalServletFilter.java:293)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:7053)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3902)
at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2773)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)
Hi user9503787,
Can you please confirm if the Weblogic instance / java process you took the Thread Dump from was showing high CPU utilization?
Your Thread Dump data is quite interesting and a perfect example of a non Thread safe HashMap combined with a read/write race condition:
Thread #23 (hanging for 600 seconds+) is showing a potential infinite looping condition in at java.util.HashMap.getEntry(HashMap.java:361)
Thread #22 (hanging for 600 seconds+) is showing a potential infinite looping condition while performing a resize() following a put() operation
Weblogic Portal has some known issues which regards to usage of non Thread safe HashHap data structures; which under load can cause infinite looping and sharp surge of CPU (due to corruption of the internal HashMap index preventing exiting the internal loop); ultimately hanging the whole JVM.
We had the same issue for one of our Weblogic Portal 10.0 production environment and Oracle provided a patch (via Collections.synchronizedMap() ).
Please open a SR and Oracle, share these 2 Threads and discuss with the engineer about a patch / know issues regarding non Thread safe HashMap and infinite looping.
Regards,
P-H
http://javaeesupportpatterns.blogspot.com/
Similar Messages
-
Need help with STUCK Threads. What's causing them?
In our production environment, we have a soa suite cluster with 3 nodes. All 3 of the servers keep turning up stuck and hogging threads, and eventually they all grind to a halt where I can't even view composite instances through em. There is a bpel process that is using a dbadapter to poll for records every 5 seconds and then performs some service call outs and other dbadapters. This process is generating thousands of instances a day and I think it is the culprit. If I view the logs, I see one of the dbadapters that it calls out to fails occasionally with a PK violation. I looked at the developer's code, and they have no error handling in the process at all. In the logs, there will the a hit for the PK violation, then about 10 minutes later the a STUCK thread gets logged(the configured timeout is 600 seconds, or 10 minutes). Would not handling an error in a BPEL process hog/STUCK a thread? I need to be able to show what code is causing the issue so I can have a resolution path.
I also see a note in the trace about something holding a lock:
Holding lock: oracle.jdbc.driver.T4CConnection@1fbc3486[thin lock]....would that be an un-rolled back transaction?
Here is the dump of the STUCK thread. Any help is greatly appreciated. Thanks!
<Aug 5, 2011 10:30:02 AM EDT> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '64' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "634" seconds working on the request "oracle.soa.management.internal.ejb.impl.FacadeFinderBean_4vacyo_FacadeFinderBeanImpl", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-517 "[STUCK] ExecuteThread: '64' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, priority=1, DAEMON> {
jrockit.net.SocketNativeIO.readBytesPinned(SocketNativeIO.java:???)
jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:24)
java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)
java.net.SocketInputStream.read(SocketInputStream.java:107)
oracle.net.nt.MetricsEnabledInputStream.read(TcpNTAdapter.java:553)
oracle.net.ns.Packet.receive(Packet.java:249)
oracle.net.ns.DataPacket.receive(DataPacket.java:94)
oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:168)
oracle.net.ns.NetInputStream.read(NetInputStream.java:113)
oracle.net.ns.NetInputStream.read(NetInputStream.java:96)
oracle.net.ns.NetInputStream.read(NetInputStream.java:81)
oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:120)
oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:75)
oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1081)
oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1062)
oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:588)
oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:70)
oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:854)
oracle.jdbc.driver.T4CPreparedStatement.executeMaybeDescribe(T4CPreparedStatement.java:918)
oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1252)
oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3541)
oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3605)
^-- Holding lock: oracle.jdbc.driver.T4CConnection@1fbc3486[thin lock]
oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1667)
weblogic.jdbc.wrapper.PreparedStatement.executeQuery(PreparedStatement.java:123)
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeSelect(DatabaseAccessor.java:887)
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:541)
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:530)
org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:521)
org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:202)
org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:192)
org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelectCall(DatasourceCallQueryMechanism.java:254)
org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelect(DatasourceCallQueryMechanism.java:245)
org.eclipse.persistence.queries.DataReadQuery.executeNonCursor(DataReadQuery.java:188)
org.eclipse.persistence.queries.DataReadQuery.executeDatabaseQuery(DataReadQuery.java:141)
org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:603)
org.eclipse.persistence.queries.DataReadQuery.execute(DataReadQuery.java:115)
org.eclipse.persistence.queries.DatabaseQuery.executeInUnitOfWork(DatabaseQuery.java:589)
org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2898)
org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1218)
org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1190)
org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1176)
org.eclipse.persistence.internal.jpa.EJBQueryImpl.executeReadQuery(EJBQueryImpl.java:378)
org.eclipse.persistence.internal.jpa.EJBQueryImpl.getSingleResult(EJBQueryImpl.java:701)
weblogic.deployment.TransactionalEntityManagerProxyImpl$QueryImpl.getSingleResult(TransactionalEntityManagerProxyImpl.java:232)
com.collaxa.cube.persistence.dao.impl.jpa.JpaCubeInstanceDaoImpl.getNumberOfInstances(JpaCubeInstanceDaoImpl.java:439)
oracle.fabric.CubeServiceEngine.getNumberOfComponentInstances(CubeServiceEngine.java:1962)
oracle.soa.management.internal.ejb.impl.FacadeFinderBeanImpl.getNumberOfComponentInstances(FacadeFinderBeanImpl.java:1618)
oracle.soa.management.internal.ejb.impl.FacadeFinderBeanImpl.getComponentSnapShot(FacadeFinderBeanImpl.java:1653)
sun.reflect.GeneratedMethodAccessor5852.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.jee.intercept.MethodInvocationInvocationContext.proceed(MethodInvocationInvocationContext.java:104)
oracle.security.jps.ee.ejb.JpsAbsInterceptor$1.run(JpsAbsInterceptor.java:88)
oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:299)
oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:406)
oracle.security.jps.wls.JpsWeblogicEjbInterceptor.runJaasMode(JpsWeblogicEjbInterceptor.java:61)
oracle.security.jps.ee.ejb.JpsAbsInterceptor.intercept(JpsAbsInterceptor.java:81)
oracle.security.jps.ee.ejb.JpsInterceptor.intercept(JpsInterceptor.java:101)
sun.reflect.GeneratedMethodAccessor863.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
com.bea.core.repackaged.springframework.jee.intercept.JeeInterceptorInterceptor.invoke(JeeInterceptorInterceptor.java:66)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:102)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.jee.spi.MethodInvocationVisitorImpl.visit(MethodInvocationVisitorImpl.java:37)
weblogic.ejb.container.injection.EnvironmentInterceptorCallbackImpl.callback(EnvironmentInterceptorCallbackImpl.java:22)
com.bea.core.repackaged.springframework.jee.spi.EnvironmentInterceptor.invoke(EnvironmentInterceptor.java:49)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:86)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:102)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:151)
$Proxy309.getComponentSnapShot(Unknown Source)
oracle.soa.management.internal.ejb.impl.FacadeFinderBean_4vacyo_FacadeFinderBeanImpl.getComponentSnapShot(FacadeFinderBean_4vacyo_FacadeFinderBeanImpl.java:1604)
oracle.soa.management.internal.ejb.impl.FacadeFinderBean_4vacyo_FacadeFinderBeanImpl_WLSkel.invoke(Unknown Source)
weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:567)
weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:477)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:350)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:143)
weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:425)
weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:117)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:198)
weblogic.work.ExecuteThread.run(ExecuteThread.java:165)
>
<Aug 5, 2011 10:30:02 AM EDT> <Notice> <Diagnostics> <BEA-320068> <Watch 'StuckThread' with severity 'Notice' on server 'soa_server1' has triggered at Aug 5, 2011 10:30:02 AM EDT. Notification details:
WatchRuleType: Log
WatchRule: (SEVERITY = 'Error') AND (MSGID = 'BEA-000337')
WatchData: DATE = Aug 5, 2011 10:30:02 AM EDT SERVER = soa_server1 MESSAGE = [STUCK] ExecuteThread: '64' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "634" seconds working on the request "oracle.soa.management.internal.ejb.impl.FacadeFinderBean_4vacyo_FacadeFinderBeanImpl", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-517 "[STUCK] ExecuteThread: '64' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, priority=1, DAEMON> {
jrockit.net.SocketNativeIO.readBytesPinned(SocketNativeIO.java:???)
jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:24)
java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)
java.net.SocketInputStream.read(SocketInputStream.java:107)
oracle.net.nt.MetricsEnabledInputStream.read(TcpNTAdapter.java:553)
oracle.net.ns.Packet.receive(Packet.java:249)
oracle.net.ns.DataPacket.receive(DataPacket.java:94)
oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:168)
oracle.net.ns.NetInputStream.read(NetInputStream.java:113)
oracle.net.ns.NetInputStream.read(NetInputStream.java:96)
oracle.net.ns.NetInputStream.read(NetInputStream.java:81)
oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:120)
oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:75)
oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1081)
oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1062)
oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:588)
oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:70)
oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:854)
oracle.jdbc.driver.T4CPreparedStatement.executeMaybeDescribe(T4CPreparedStatement.java:918)
oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1252)
oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3541)
oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3605)
^-- Holding lock: oracle.jdbc.driver.T4CConnection@1fbc3486[thin lock]
oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1667)
weblogic.jdbc.wrapper.PreparedStatement.executeQuery(PreparedStatement.java:123)
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeSelect(DatabaseAccessor.java:887)
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:541)
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:530)
org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:521)
org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:202)
org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:192)
org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelectCall(DatasourceCallQueryMechanism.java:254)
org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelect(DatasourceCallQueryMechanism.java:245)
org.eclipse.persistence.queries.DataReadQuery.executeNonCursor(DataReadQuery.java:188)
org.eclipse.persistence.queries.DataReadQuery.executeDatabaseQuery(DataReadQuery.java:141)
org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:603)
org.eclipse.persistence.queries.DataReadQuery.execute(DataReadQuery.java:115)
org.eclipse.persistence.queries.DatabaseQuery.executeInUnitOfWork(DatabaseQuery.java:589)
org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2898)
org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1218)
org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1190)
org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1176)
org.eclipse.persistence.internal.jpa.EJBQueryImpl.executeReadQuery(EJBQueryImpl.java:378)
org.eclipse.persistence.internal.jpa.EJBQueryImpl.getSingleResult(EJBQueryImpl.java:701)
weblogic.deployment.TransactionalEntityManagerProxyImpl$QueryImpl.getSingleResult(TransactionalEntityManagerProxyImpl.java:232)
com.collaxa.cube.persistence.dao.impl.jpa.JpaCubeInstanceDaoImpl.getNumberOfInstances(JpaCubeInstanceDaoImpl.java:439)
oracle.fabric.CubeServiceEngine.getNumberOfComponentInstances(CubeServiceEngine.java:1962)
oracle.soa.management.internal.ejb.impl.FacadeFinderBeanImpl.getNumberOfComponentInstances(FacadeFinderBeanImpl.java:1618)
oracle.soa.management.internal.ejb.impl.FacadeFinderBeanImpl.getComponentSnapShot(FacadeFinderBeanImpl.java:1653)
sun.reflect.GeneratedMethodAccessor5852.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.jee.intercept.MethodInvocationInvocationContext.proceed(MethodInvocationInvocationContext.java:104)
oracle.security.jps.ee.ejb.JpsAbsInterceptor$1.run(JpsAbsInterceptor.java:88)
oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:299)
oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:406)
oracle.security.jps.wls.JpsWeblogicEjbInterceptor.runJaasMode(JpsWeblogicEjbInterceptor.java:61)
oracle.security.jps.ee.ejb.JpsAbsInterceptor.intercept(JpsAbsInterceptor.java:81)
oracle.security.jps.ee.ejb.JpsInterceptor.intercept(JpsInterceptor.java:101)
sun.reflect.GeneratedMethodAccessor863.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
com.bea.core.repackaged.springframework.jee.intercept.JeeInterceptorInterceptor.invoke(JeeInterceptorInterceptor.java:66)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:102)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.jee.spi.MethodInvocationVisitorImpl.visit(MethodInvocationVisitorImpl.java:37)
weblogic.ejb.container.injection.EnvironmentInterceptorCallbackImpl.callback(EnvironmentInterceptorCallbackImpl.java:22)
com.bea.core.repackaged.springframework.jee.spi.EnvironmentInterceptor.invoke(EnvironmentInterceptor.java:49)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:86)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:102)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:151)
$Proxy309.getComponentSnapShot(Unknown Source)
oracle.soa.management.internal.ejb.impl.FacadeFinderBean_4vacyo_FacadeFinderBeanImpl.getComponentSnapShot(FacadeFinderBean_4vacyo_FacadeFinderBeanImpl.java:1604)
oracle.soa.management.internal.ejb.impl.FacadeFinderBean_4vacyo_FacadeFinderBeanImpl_WLSkel.invoke(Unknown Source)
weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:567)
weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:477)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:350)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:143)
weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:425)
weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:117)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:198)
weblogic.work.ExecuteThread.run(ExecuteThread.java:165)
SUBSYSTEM = WebLogicServer USERID = <WLS Kernel> SEVERITY = Error THREAD = [ACTIVE] ExecuteThread: '96' for queue: 'weblogic.kernel.Default (self-tuning)' MSGID = BEA-000337 MACHINE = soaprod TXID = CONTEXTID = f5d0b85a929630a2:6eada0f7:1304e0a2bdc:-7ffb-00000000017e1605 TIMESTAMP = 1312554602698
WatchAlarmType: AutomaticReset
WatchAlarmResetPeriod: 600000
>
Edited by: Brutus35 on Aug 5, 2011 8:04 AMmy impression is that you have trouble on the DB,
monitor the sessions to see if there is some table locked, and increase the number of processes on the DB
you can use SQLDeveloper for that
anyway your TX should rollback after some time, check that you have a timeout in place, it's better to rollback a TX than having the entire server crash -
Need some help with stucked threads
Hi, I am running an ADF application on Weblogic 10.3.4. I get a lot of stucked threads and can't figure out the reason.
Here is my stack dump:
<31.mar.2011 kl 08.36 CEST> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '21' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "625" seconds working on the request "weblogic.servlet.internal.ServletRequestImpl@237aaa4c[
POST /faces/hovedside?_adf.ctrl-state=13zq3miluz_4 HTTP/1.1
Connection: keep-alive
Referer: https://please.help.me/?_afrLoop=8410499084123&_afrWindowMode=0&_afrWindowId=null
Content-Length: 356
Adf-Ads-Page-Id: 3
Origin: https://please.help.me
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.204 Safari/534.16
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Adf-Rich-Message: true
Accept: */*
Accept-Encoding: gzip,deflate,sdch
Accept-Language: nb-NO,nb;q=0.8,no;q=0.6,nn;q=0.4,en-US;q=0.2,en;q=0.2
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Cookie: JSESSIONID=1dQ7NJpXZhYGM11Sf2ppDz4LZwQM5FpL6yxDrj0cSlMTsTdWvPPN!-222523808
]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-133 "[STUCK] ExecuteThread: '21' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, suspended, priority=1, DAEMON> {
java.util.ArrayList.clone(ArrayList.java:250)
oracle.jbo.uicli.binding.JUCtrlHierNodeBinding.getChildren(JUCtrlHierNodeBinding.java:719)
oracle.jbo.uicli.binding.JUCtrlHierNodeBinding.findChildNode(JUCtrlHierNodeBinding.java:844)
oracle.jbo.uicli.binding.JUCtrlHierBinding.bringNodeToRangeKeyPath(JUCtrlHierBinding.java:772)
oracle.adfinternal.view.faces.model.binding.FacesCtrlHierBinding.bringNodeToRangeKeyPath(FacesCtrlHierBinding.java:101)
oracle.adfinternal.view.faces.model.binding.RowDataManager.setRowKey(RowDataManager.java:119)
oracle.adfinternal.view.faces.model.binding.FacesCtrlHierBinding$FacesModel.setRowKey(FacesCtrlHierBinding.java:633)
org.apache.myfaces.trinidad.model.RowKeySetTreeImpl$PathIterator.nextModelKey(RowKeySetTreeImpl.java:745)
org.apache.myfaces.trinidad.model.RowKeySetTreeImpl$PathIterator.nextItem(RowKeySetTreeImpl.java:740)
org.apache.myfaces.trinidad.model.RowKeySetTreeImpl$PathIterator.next(RowKeySetTreeImpl.java:721)
oracle.adfinternal.view.faces.renderkit.rich.TreeRendererUtils.initModel(TreeRendererUtils.java:288)
oracle.adfinternal.view.faces.renderkit.rich.TreeRendererUtils.wrapModel(TreeRendererUtils.java:527)
oracle.adfinternal.view.faces.renderkit.rich.TreeRendererUtils.getCollectionObject(TreeRendererUtils.java:340)
oracle.adfinternal.view.faces.renderkit.rich.TreeTableRenderer.getCollectionObject(TreeTableRenderer.java:762)
oracle.adfinternal.view.faces.renderkit.rich.TableRenderer.storeHeightAutosized(TableRenderer.java:2365)
oracle.adfinternal.view.faces.renderkit.rich.TableRenderer.encodeAll(TableRenderer.java:482)
oracle.adfinternal.view.faces.renderkit.rich.TreeTableRenderer.encodeAll(TreeTableRenderer.java:585)
oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1252)
org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:326)
org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:758)
org.apache.myfaces.trinidad.component.UIXCollection.encodeEnd(UIXCollection.java:527)
javax.faces.component.UIComponent.encodeAll(UIComponent.java:920)
org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:398)
oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2608)
oracle.adfinternal.view.faces.renderkit.rich.PanelGroupLayoutRenderer._encodeChild(PanelGroupLayoutRenderer.java:398)
oracle.adfinternal.view.faces.renderkit.rich.PanelGroupLayoutRenderer.access$300(PanelGroupLayoutRenderer.java:30)
oracle.adfinternal.view.faces.renderkit.rich.PanelGroupLayoutRenderer$EncoderCallback.processComponent(PanelGroupLayoutRenderer.java:582)
oracle.adfinternal.view.faces.renderkit.rich.PanelGroupLayoutRenderer$EncoderCallback.processComponent(PanelGroupLayoutRenderer.java:573)
org.apache.myfaces.trinidad.component.UIXComponent.processFlattenedChildren(UIXComponent.java:154)
org.apache.myfaces.trinidad.component.UIXComponent.processFlattenedChildren(UIXComponent.java:285)
org.apache.myfaces.trinidad.component.UIXComponent.encodeFlattenedChildren(UIXComponent.java:252)
oracle.adfinternal.view.faces.renderkit.rich.PanelGroupLayoutRenderer.encodeAll(PanelGroupLayoutRenderer.java:236)
oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1252)
org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:326)
org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:758)
javax.faces.component.UIComponent.encodeAll(UIComponent.java:920)
org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:398)
oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2608)
org.apache.myfaces.trinidad.render.CoreRenderer.encodeAllChildren(CoreRenderer.java:407)
oracle.adfinternal.view.faces.renderkit.rich.PanelHeaderRenderer.renderChildrenAfterHelpAndInfo(PanelHeaderRenderer.java:471)
oracle.adfinternal.view.faces.renderkit.rich.PanelHeaderRenderer._renderContentCell(PanelHeaderRenderer.java:939)
oracle.adfinternal.view.faces.renderkit.rich.PanelHeaderRenderer.renderContentRow(PanelHeaderRenderer.java:425)
oracle.adfinternal.view.faces.renderkit.rich.PanelHeaderRenderer.encodeAll(PanelHeaderRenderer.java:160)
oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1252)
org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:326)
org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:758)
javax.faces.component.UIComponent.encodeAll(UIComponent.java:920)
org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:398)
javax.faces.component.UIComponent.encodeAll(UIComponent.java:920)
org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:398)
oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2608)
oracle.adfinternal.view.faces.renderkit.rich.PanelGroupLayoutRenderer._encodeChild(PanelGroupLayoutRenderer.java:398)
oracle.adfinternal.view.faces.renderkit.rich.PanelGroupLayoutRenderer.access$300(PanelGroupLayoutRenderer.java:30)
oracle.adfinternal.view.faces.renderkit.rich.PanelGroupLayoutRenderer$EncoderCallback.processComponent(PanelGroupLayoutRenderer.java:582)
oracle.adfinternal.view.faces.renderkit.rich.PanelGroupLayoutRenderer$EncoderCallback.processComponent(PanelGroupLayoutRenderer.java:573)
org.apache.myfaces.trinidad.component.UIXComponent.processFlattenedChildren(UIXComponent.java:154)
org.apache.myfaces.trinidad.component.UIXComponent.processFlattenedChildren(UIXComponent.java:285)
org.apache.myfaces.trinidad.component.UIXComponent.encodeFlattenedChildren(UIXComponent.java:252)
oracle.adfinternal.view.faces.renderkit.rich.PanelGroupLayoutRenderer.encodeAll(PanelGroupLayoutRenderer.java:236)
oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1252)
org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:326)
org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:758)
javax.faces.component.UIComponent.encodeAll(UIComponent.java:920)
org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:398)
oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2608)
oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer.encodeCenterFacet(PanelStretchLayoutRenderer.java:769)
oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer._encodeCenterPane(PanelStretchLayoutRenderer.java:1055)
oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer._encodeMiddlePanes(PanelStretchLayoutRenderer.java:345)
oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer.encodeAll(PanelStretchLayoutRenderer.java:122)
oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1252)
org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:326)
org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:758)
javax.faces.component.UIComponent.encodeAll(UIComponent.java:920)
org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:398)
oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2608)
oracle.adf.view.rich.render.RichRenderer.encodeStretchedChild(RichRenderer.java:1998)
oracle.adfinternal.view.faces.renderkit.rich.RegionRenderer.access$400(RegionRenderer.java:46)
oracle.adfinternal.view.faces.renderkit.rich.RegionRenderer$ChildEncoderCallback.processComponent(RegionRenderer.java:562)
oracle.adfinternal.view.faces.renderkit.rich.RegionRenderer$ChildEncoderCallback.processComponent(RegionRenderer.java:553)
org.apache.myfaces.trinidad.component.UIXComponent.processFlattenedChildren(UIXComponent.java:154)
org.apache.myfaces.trinidad.component.UIXComponent.processFlattenedChildren(UIXComponent.java:285)
org.apache.myfaces.trinidad.component.UIXComponent.encodeFlattenedChildren(UIXComponent.java:252)
oracle.adfinternal.view.faces.renderkit.rich.RegionRenderer._encodeChildren(RegionRenderer.java:229)
oracle.adfinternal.view.faces.renderkit.rich.RegionRenderer.encodeAll(RegionRenderer.java:144)
oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1252)
org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:326)
org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:758)
oracle.adf.view.rich.component.fragment.UIXRegion.encodeEnd(UIXRegion.java:292)
javax.faces.component.UIComponent.encodeAll(UIComponent.java:920)
oracle.adfinternal.view.faces.taglib.region.IncludeTag$FacetWrapper.encodeAll(IncludeTag.java:511)
org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:398)
oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2608)
oracle.adfinternal.view.faces.renderkit.rich.DecorativeBoxRenderer.encodeFacet(DecorativeBoxRenderer.java:361)
oracle.adfinternal.view.faces.renderkit.rich.DecorativeBoxRenderer._encodeCenterPane(DecorativeBoxRenderer.java:529)
oracle.adfinternal.view.faces.renderkit.rich.DecorativeBoxRenderer.encodeAll(DecorativeBoxRenderer.java:167)
oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1252)
org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:326)
org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:758)
javax.faces.component.UIComponent.encodeAll(UIComponent.java:920)
org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:398)
oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2608)
oracle.adf.view.rich.render.RichRenderer.encodeStretchedChild(RichRenderer.java:1998)
oracle.adfinternal.view.faces.renderkit.rich.PanelSplitterRenderer._renderPane(PanelSplitterRenderer.java:1267)
oracle.adfinternal.view.faces.renderkit.rich.PanelSplitterRenderer.encodeAll(PanelSplitterRenderer.java:88)
oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1252)
org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:326)
org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:758)
javax.faces.component.UIComponent.encodeAll(UIComponent.java:920)
org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:398)
oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2608)
oracle.adf.view.rich.render.RichRenderer.encodeStretchedChild(RichRenderer.java:1998)
oracle.adfinternal.view.faces.renderkit.rich.PanelSplitterRenderer._renderPane(PanelSplitterRenderer.java:1267)
oracle.adfinternal.view.faces.renderkit.rich.PanelSplitterRenderer.encodeAll(PanelSplitterRenderer.java:88)
oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1252)
oracle.adfinternal.view.faces.renderkit.rich.PanelSplitterRenderer.encodeAll(PanelSplitterRenderer.java:88)
oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1252)
org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:326)
org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:758)
javax.faces.component.UIComponent.encodeAll(UIComponent.java:920)
org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:398)
oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2608)
oracle.adfinternal.view.faces.renderkit.rich.DecorativeBoxRenderer.encodeFacet(DecorativeBoxRenderer.java:361)
oracle.adfinternal.view.faces.renderkit.rich.DecorativeBoxRenderer._encodeCenterPane(DecorativeBoxRenderer.java:529)
oracle.adfinternal.view.faces.renderkit.rich.DecorativeBoxRenderer.encodeAll(DecorativeBoxRenderer.java:167)
oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1252)
org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:326)
org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:758)
javax.faces.component.UIComponent.encodeAll(UIComponent.java:920)
org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:398)
oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2608)
oracle.adf.view.rich.render.RichRenderer.encodeStretchedChild(RichRenderer.java:1998)
oracle.adfinternal.view.faces.renderkit.rich.PanelSplitterRenderer._renderPane(PanelSplitterRenderer.java:1267)
oracle.adfinternal.view.faces.renderkit.rich.PanelSplitterRenderer.encodeAll(PanelSplitterRenderer.java:88)
oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1252)
org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:326)
org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:758)
javax.faces.component.UIComponent.encodeAll(UIComponent.java:920)
org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:398)
oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2608)
oracle.adf.view.rich.render.RichRenderer.encodeStretchedChild(RichRenderer.java:1998)
oracle.adfinternal.view.faces.renderkit.rich.PanelSplitterRenderer._renderPane(PanelSplitterRenderer.java:1267)
oracle.adfinternal.view.faces.renderkit.rich.PanelSplitterRenderer.encodeAll(PanelSplitterRenderer.java:88)
oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1252)
org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:326)
org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:758)
javax.faces.component.UIComponent.encodeAll(UIComponent.java:920)
org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:398)
oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2608)
org.apache.myfaces.trinidad.render.CoreRenderer.encodeAllChildren(CoreRenderer.java:407)
oracle.adfinternal.view.faces.renderkit.rich.PageTemplateRenderer.encodeAll(PageTemplateRenderer.java:69)
oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1252)
org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:326)
org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:758)
oracle.adf.view.rich.component.fragment.ContextSwitchingComponent.encodeEnd(ContextSwitchingComponent.java:155)
javax.faces.component.UIComponent.encodeAll(UIComponent.java:920)
org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:398)
oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2608)
org.apache.myfaces.trinidad.render.CoreRenderer.encodeAllChildren(CoreRenderer.java:407)
oracle.adfinternal.view.faces.renderkit.rich.FormRenderer.encodeAll(FormRenderer.java:158)
oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1252)
org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:326)
org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:758)
javax.faces.component.UIComponent.encodeAll(UIComponent.java:920)
org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:398)
oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2608)
org.apache.myfaces.trinidad.render.CoreRenderer.encodeAllChildren(CoreRenderer.java:407)
oracle.adfinternal.view.faces.renderkit.rich.DocumentRenderer.encodeAll(DocumentRenderer.java:725)
oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1252)
org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:326)
org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:758)
javax.faces.component.UIComponent.encodeAll(UIComponent.java:920)
javax.faces.component.UIComponent.encodeAll(UIComponent.java:920)
com.sun.faces.application.ViewHandlerImpl.doRenderView(ViewHandlerImpl.java:256)
com.sun.faces.application.ViewHandlerImpl.renderView(ViewHandlerImpl.java:131)
javax.faces.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:189)
org.apache.myfaces.trinidadinternal.application.ViewHandlerImpl.renderView(ViewHandlerImpl.java:155)
oracle.adfinternal.view.faces.lifecycle.LifecycleImpl._renderResponse(LifecycleImpl.java:773)
oracle.adfinternal.view.faces.lifecycle.LifecycleImpl._executePhase(LifecycleImpl.java:252)
oracle.adfinternal.view.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:207)
javax.faces.webapp.FacesServlet.service(FacesServlet.java:246)
oracle.adfinternal.view.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:207)
javax.faces.webapp.FacesServlet.service(FacesServlet.java:246)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:206)
weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.adf.model.servlet.ADFBindingFilter.doFilter(ADFBindingFilter.java:147)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.adfinternal.view.faces.webapp.rich.RegistrationFilter.doFilter(RegistrationFilter.java:69)
org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:437)
oracle.adfinternal.view.faces.activedata.AdsFilter.doFilter(AdsFilter.java:44)
org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:437)
org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl._doFilterImpl(TrinidadFilterImpl.java:231)
org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl.doFilter(TrinidadFilterImpl.java:133)
org.apache.myfaces.trinidad.webapp.TrinidadFilter.doFilter(TrinidadFilter.java:91)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.adf.library.webapp.LibraryFilter.doFilter(LibraryFilter.java:163)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:111)
oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:299)
oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:405)
oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:94)
oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:98)
oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:70)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:86)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:25)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3687)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3681)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:308)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:116)
weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2213)
weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2135)
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1420)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
weblogic.work.ExecuteThread.run(ExecuteThread.java:168)
Please help.
ThomasHi Thomas,
Your Thread is showing a processor computing operation ArrayList.clone() triggered from the Oracle JBO framework / API.
Few questions for you:
- Are you using Solaris OS? If yes can you please capture the PRSTAT at the same time as your JVM Thread Dump using command:
prstat -L 0 1 | grep <java pid>;kill -3 <java pid>
- How many threads are showing this condition?
- Was CPU utilization of the affected Java process very high on the server at that time?
- What is the utilization of your Java Heap at the time of the Thread Dump capture?
This Thread could be stuck in an infinite loop since it is stuck for at least 600 seconds (10 minutes). It would probably help to generate a few more snapshots earlier and understand if Thread is hanging in the same code portion (possibility Oracle JBO) for every snapshot or not.
This could also be data related (too many data rows) returned and causing high processing time for the clone operation from the Myfaces / JSF table that you are trying to render.
Regards,
P-H
http://javaeesupportpatterns.blogspot.com/ -
WL Server Stuck Thread on Network Read
I am experiencing an issue with the Weblogic deployment at my organisation.
Firstly an introduction; My Application works by receiving a Web Service (SOAP) Request from a client, the Web Service request is then processed and a internal message packet is generated and sent to a C++ Application ( which is listening on a particular port). The C++ application then processes the information, and returns the response to the web service Layer, which then process the Web Service request back to the calling client.
Currently we are having a situation where some packets are sent to the C++ application as expected. But then we are having situations where some packets NEVER reach the C++ side and the calling client time outs the entire request ( configured timeout is set for 180 seconds) and on the WebLogic Layer we have noticed a significant amount of Stuck ( and Hogging ) Threads. It should be noted that even after a stuck thread is recorded some requests are still being processed normally.
A deeper investigation into the Weblogic Thread dump shows the Weblogic thread is stuck at:
_at sun/nio/ch/IOUtil.read(Ljava/io/FileDescriptor;Ljava/nio/ByteBuffer;JLsun/nio/ch/NativeDispatcher;Ljava/lang/Object;)I(IOUtil.java)[inlined]_+
_at sun/nio/ch/SocketChannelImpl.read(SocketChannelImpl.java:236)[optimized]_+
+_^-- Holding lock: java/lang/Object@0x1a27db60[thin lock]_+
_at com/reso/number/service/comm/SoaSocket.receive(SoaSocket.java:128)[optimized]_+
_at com/reso/number/service/event/OTEvent.submitEvent(TEvent.java:219)[optimized]_+
_at com/reso/number/service/event/SIMQueryEvent.process(QQEvent.java:368)[optimized]_+
It seems that a write is sent to the C++ Application and the thread is 'WAITING' for the response.
Now, the problem is that due to the Infrastructure in place and architectural setup ( We are using Solaris Zone for Node deployment and we have the WebLogic AS on the same node/zone as the C++ application) we are unable to do a network trace to determine whether WebLogic actually sends the request.
Now my questions: i) Are there any known ways to trace the networks packets sent by Java ( maybe via the JVM) ii) Could Garbage collection cause the symptoms being observed? iii)Any recommended ways to resolve this kind of problem?
Thanks for all the help.Increasing the Stuck Thread Max Time will not help. " Stuck Thread Max Time" tells us, after how much amount of time the thread should mark as stuck. Because by default it is 600 sec, any thread which takes more than 600sec for execution will mark as stuck. If you mention 700 sec, after 700sec the thread is marked as stuck. Even though the thread is marked stuck it still process.
Here we need to find why the thread is stuck at
]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-15 "[STUCK] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, priority=1, DAEMON> {
sun.nio.ch.SocketDispatcher.read1(SocketDispatcher.java:???)
sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:46)
sun.nio.ch.IOUtil.read(IOUtil.java:228)
sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:159)
^-- Holding lock: java.lang.Object@16a47a8b[thin lock]
weblogic.socket.NIOSocketMuxer$NIOInputStream.read(NIOSocketMuxer.java:505)
weblogic.servlet.internal.PostInputStream.read(PostInputStream.java:156)
weblogic.servlet.internal.ServletInputStreamImpl.read(ServletInputStreamImpl.java:228)
weblogic.servlet.internal.ServletRequestImpl$RequestParameters.mergePostParams(ServletRequestImpl.java:2062)
You may have to increase the buffer size at sun.nio. -
Stuck Threads on weblogic 10.3.2 Holding lock HttpURLConnection
We are having issue with STUCK THREADS on weblogic 10.3.2. Basically my application makes a lot of web service calls over HTTPS and once in a while I see few STUCK THREADS and most of the time they clear but sometimes I have to bounce my server to clean.
I have a thread dump, according to this its holding lock at weblogic.net.http.HttpURLConnection.getInputStream
Any help to resolve this problem is highly appreciated.
ThreadDump:
<May 24, 2011 9:59:08 AM EDT> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "604" seconds working on the request "weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl@b607d75", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-75 "[STUCK] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, priority=1, DAEMON> {
jrockit.net.SocketNativeIO.readBytesPinned(SocketNativeIO.java:???)
jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:31)
java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)
java.net.SocketInputStream.read(SocketInputStream.java:107)
weblogic.utils.io.ChunkedInputStream.read(ChunkedInputStream.java:149)
java.io.InputStream.read(InputStream.java:85)
com.certicom.tls.record.ReadHandler.readFragment(Unknown Source)
com.certicom.tls.record.ReadHandler.readRecord(Unknown Source)
com.certicom.tls.record.ReadHandler.read(Unknown Source)
^-- Holding lock: com.certicom.tls.record.ReadHandler@bec563c[thin lock]
com.certicom.io.InputSSLIOStreamWrapper.read(Unknown Source)
java.io.BufferedInputStream.fill(BufferedInputStream.java:189)
java.io.BufferedInputStream.read(BufferedInputStream.java:236)
^-- Holding lock: java.io.BufferedInputStream@bec5d9f[thin lock]
weblogic.net.http.MessageHeader.isHTTP(MessageHeader.java:214)
weblogic.net.http.MessageHeader.parseHeader(MessageHeader.java:141)
weblogic.net.http.HttpClient.parseHTTP(HttpClient.java:452)
^-- Holding lock: weblogic.net.http.HttpsClient@bec5202[thin lock]
weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:328)
weblogic.net.http.SOAPHttpsURLConnection.getInputStream(SOAPHttpsURLConnection.java:37)
^-- Holding lock: weblogic.net.http.SOAPHttpsURLConnection@bd3b209[thin lock]
weblogic.net.http.HttpURLConnection.getResponseCode(HttpURLConnection.java:939)
com.sun.xml.ws.transport.http.client.HttpClientTransport.readResponseCodeAndMessage(HttpClientTransport.java:213)
com.sun.xml.ws.transport.http.client.HttpTransportPipe.process(HttpTransportPipe.java:116)
com.sun.xml.ws.transport.http.client.HttpTransportPipe.processRequest(HttpTransportPipe.java:101)
com.sun.xml.ws.api.pipe.Fiber.__doRun(Fiber.java:580)
com.sun.xml.ws.api.pipe.Fiber._doRun(Fiber.java:553)
com.sun.xml.ws.api.pipe.Fiber.doRun(Fiber.java:535)
com.sun.xml.ws.api.pipe.Fiber.runSync(Fiber.java:433)
^-- Holding lock: com.sun.xml.ws.api.pipe.Fiber@bd3afa8[thin lock]
com.sun.xml.ws.client.Stub.process(Stub.java:238)
com.sun.xml.ws.client.sei.SEIStub.doProcess(SEIStub.java:135)
com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:102)
com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:89)
com.sun.xml.ws.client.sei.SEIStub.invoke(SEIStub.java:116)
$Proxy107.jobSubmit(Unknown Source)
com.somejob.invokeService(JobOperation.java:53)
com.somejobws.operations.ticket.execute(ticket.java:30)
com.somejobutil.Somehelper.process(Somehelper.java:174)
com.somejobutil.Somehelper.processJob(Somehelper.java:147)
com.somejobutil.Somehelper.processMDB(Somehelper.java:91)
com.somejobejb.ProcessMDB.onMessage(ProcessMDB.java:28)
sun.reflect.GeneratedMethodAccessor109.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:102)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:86)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:102)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:151)
$Proxy67.onMessage(Unknown Source)
weblogic.ejb.container.internal.MDListener.execute(MDListener.java:403)
weblogic.ejb.container.internal.MDListener.transactionalOnMessage(MDListener.java:352)
weblogic.ejb.container.internal.MDListener.onMessage(MDListener.java:264)
weblogic.jms.client.JMSSession.onMessage(JMSSession.java:4564)
weblogic.jms.client.JMSSession.execute(JMSSession.java:4060)
weblogic.jms.client.JMSSession.executeMessage(JMSSession.java:3746)
weblogic.jms.client.JMSSession.access$000(JMSSession.java:114)
weblogic.jms.client.JMSSession$UseForRunnable.run(JMSSession.java:5096)
weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:516)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:198)
weblogic.work.ExecuteThread.run(ExecuteThread.java:165)
}Yes, we have the same problem, stuck thread on weblogic.net.http.HttpsURLConnection.getResponseCode() (actually implemented by weblogic.net.http.SOAPHttpsURLConnection which also appears to be deprecated ???)
When we use the class javax.net.ssl.HttpsURLConnection the getResponseCode() method works perfectly well, so it looks like the weblogic implementation is to blame.
This is becoming rather urgent now because we're looking to migrate from OC4J to Weblogic and this issue has therefore become a blocker. -
Page stuck while load and leads to stuck thread
Hi
I am facing an issue from a long time.
When i run the page of my application, when it opens for first time it get stuck and get loading and take very much time and after that it leads to stuck thread issue. All the component of my page opens but there is a table which gets value from OIM APIs also loads for the fiirst time. As in coding i have done it will pass null value to the table and wont fetch data for the first time. it will only fetch when value will be passed in a text box and submit was hit.
So my page gets hanged or loading in the table section everytime when it loads. for that i have to login in weblogic server, run the workmanager, forcefully shut down the server instance and run the page again. But sometime that doesnt work too.
Please help its a bug trouble for me and creating a huge problem.
Thanks
SudeepI'm not sure I understand your issue - but if you want to delay the fetching of information into the table until you actually have a value to pass to it see:
http://blogs.oracle.com/shay/2010/08/delay_method_execution_when_us.html -
Detecting and Pinning a stuck thread
While using WLS 7.0 there exists a problem where some execute threads get stuck (i.e don't respond) and do not release the associated resources such as connections, files etc. How does one detect such a stuck thread ? Also, how can we go about pinning stuck threads?
"Haziq Pathan" <[email protected]> wrote in message news:3ffbfad4$[email protected]..
While using WLS 7.0 there exists a problem where some execute threads
get stuck (i.e don't respond) and do not release the associated resources
such as connections, files etc. How does one detect such a stuck thread ?Basically there is a server health monitor thread that periodically checks
threads and reports those who started execution and didn't finish it within
configured time.
Also, how can we go about pinning stuck threads?The best thing to do is to get a thread dump and see where the thread stuck,
and deliver a fix for the problem.
Hope this helps.
Regards,
Slava Imeshev -
Stuck Thread causing heap to grow steeply and resulting in crash
Hi,
We have an ADF application deployed in weblogic 10.3.2.
We have few PL/SQL's and SQL's which are complex and take around 5 minutes of time to complete at times. We also have the script running in database to kill long running queries so that it helps us to get rid of weblogic stuck threads to some extent.
Now when ever there is an SQL running for long time , we see a stuck thread casuing heap to grow very fast and casuing server to crash.
I know after a particular point of idle time thread is marked as stuck , but why does it filling the heap space - does stuck threads always causes heap to grow or is it just my specific case
any pointers on how to debug this issue to find the root cause.
Thanks,
NareshI think this setup needs profiling. To start-off, i would recommend collecting heap dump or jrf (if jrockit), instructions for them are available on net, as it varies with jvm versions and vendors.
Post that you should look for the thread which you believe is stuck and a possible cause using OQL in memory analyzer tool over the heap dumps.
Conceptually, till the time the objects (or data if you say) is being referenced, JVM cannot garbage collect it. So, I would say we need to figure out what are the contents of the heap, before coming to a conclusion that the threads which are stuck is causing the issue. Why I say so it that, I have seen in past that many-a-times, we suspect stuck threads to be an issue but it turns out to be that we are "knocking the wrong door".
Long running threads (aka STUCK thread in WLS), will normally cause memory issues, if the references/task that its trying to perform has a large local data (i.e. data with the thread scope), so it the time the thread is not released the data is alive (irrespective of the fact that the data is used or not).
Now, my first thought of this situation is that stuck thread is not an issue, considering that you are not able to replicate this in UAT post simulating stuck threads. Thus please collect some heap dumps (may be when you start observing memory increase) and then review them comparatively, I think that should be helpful.
AJ -
How does WebLogic clean up stuck threads?
Hi,
Using WebLogic 9.2.2 on Linux OS. In our logs, we are finding some stuck threads and it seems to be due to something WebLogic is doing. While we don't understand the cause (maybe the stack trace helps), does WebLogic do anything to remove stuck threads from its pool, or is there anything we can do to configure it to?
<Jul 21, 2009 2:50:21 PM UTC> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "605" seconds working on the request "Http Request: /RoutingEngine/pcFlow.do", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-69 "[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, priority=1, DAEMON> {
jrockit.net.SocketNativeIO.readBytesPinned(Unknown Source)
jrockit.net.SocketNativeIO.socketRead(Unknown Source)
java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)
java.net.SocketInputStream.read(SocketInputStream.java:107)
java.io.BufferedInputStream.fill(BufferedInputStream.java:189)
java.io.BufferedInputStream.read(BufferedInputStream.java:234)
^-- Holding lock: java.io.BufferedInputStream@3537712[thin lock]
weblogic.net.http.MessageHeader.isHTTP(MessageHeader.java:214)
weblogic.net.http.MessageHeader.parseHeader(MessageHeader.java:141)
weblogic.net.http.HttpClient.parseHTTP(HttpClient.java:473)
^-- Holding lock: java.io.BufferedInputStream@3537712[thin lock]
weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:332)
^-- Holding lock: java.io.BufferedInputStream@3537712[thin lock]
weblogic.net.http.SOAPHttpURLConnection.getInputStream(SOAPHttpURLConnection.java:36)
weblogic.wsee.util.dom.DOMParser.getWebLogicDocumentImpl(DOMParser.java:92)
weblogic.wsee.util.dom.DOMParser.getDocument(DOMParser.java:56)
weblogic.wsee.wsdl.WsdlReader.getDocument(WsdlReader.java:311)
weblogic.wsee.wsdl.WsdlReader.getDocument(WsdlReader.java:305)
weblogic.wsee.wsdl.WsdlSchema.parse(WsdlSchema.java:120)
weblogic.wsee.wsdl.WsdlSchemaImport.parse(WsdlSchemaImport.java:55)
weblogic.wsee.wsdl.WsdlSchema.parse(WsdlSchema.java:82)
weblogic.wsee.wsdl.WsdlSchema.parse(WsdlSchema.java:62)
weblogic.wsee.wsdl.WsdlTypes.parse(WsdlTypes.java:145)
weblogic.wsee.wsdl.WsdlDefinitions.parseChild(WsdlDefinitions.java:452)
weblogic.wsee.wsdl.WsdlExtensible.parse(WsdlExtensible.java:84)
weblogic.wsee.wsdl.WsdlDefinitions.parse(WsdlDefinitions.java:378)
weblogic.wsee.wsdl.WsdlDefinitions.parse(WsdlDefinitions.java:358)
weblogic.wsee.wsdl.WsdlFactory.parse(WsdlFactory.java:50)
weblogic.wsee.jaxrpc.ServiceImpl.loadWsdlDefinition(ServiceImpl.java:437)
weblogic.wsee.jaxrpc.ServiceImpl.<init>(ServiceImpl.java:95)
com.myco.regui.provisioning.client.ProvisioningAccountService_Impl.<init>(ProvisioningAccountService_Impl.java:21)
com.myco.regui.provisioning.client.ProvisioningAccountService_Impl.<init>(ProvisioningAccountService_Impl.java:17)
com.myco.regui.struts.accounts.AccountsForm.contactWPS(AccountsForm.java:208)
com.myco.regui.struts.accounts.AccountsForm.validateAccountid(AccountsForm.java:161)
com.myco.regui.struts.accounts.AccountsForm.validatePCAccountId(AccountsForm.java:144)
com.myco.regui.struts.accounts.AccountsForm._validatePCFields(AccountsForm.java:112)
com.myco.regui.struts.accounts.AccountsForm.validate(AccountsForm.java:83)
org.apache.struts.chain.commands.servlet.ValidateActionForm.validate(ValidateActionForm.java:57)
org.apache.struts.chain.commands.AbstractValidateActionForm.execute(AbstractValidateActionForm.java:94)
org.apache.struts.chain.commands.ActionCommandBase.execute(ActionCommandBase.java:48)
org.apache.commons.chain.impl.ChainBase.execute(ChainBase.java:176)
org.apache.commons.chain.generic.LookupCommand.execute(LookupCommand.java:303)
org.apache.commons.chain.impl.ChainBase.execute(ChainBase.java:176)
org.apache.struts.chain.ComposableRequestProcessor.process(ComposableRequestProcessor.java:272)
org.apache.struts.action.ActionServlet.process(ActionServlet.java:1903)
org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:462)
javax.servlet.http.HttpServlet.service(HttpServlet.java:736)
javax.servlet.http.HttpServlet.service(HttpServlet.java:851)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:198)
weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:41)
com.myco.regui.servlets.filters.TransactionFilter.doFilter(TransactionFilter.java:23)
.......You can configure these actions to occur when not all threads are stuck, but the number of stuck threads have exceeded a configured threshold:
* Shut down the Work Manager if it has stuck threads. A Work Manager that is shut down will refuse new work and reject existing work in
the queue by sending a rejection message. In a cluster, clustered clients will fail over to another cluster member.
* Shut down the application if there are stuck threads in the application. The application is shutdown by bringing it into admin mode. All
Work Managers belonging to the application are shut down, and behave as described above.
* Mark the server instance as failed and shut it down it down if there are stuck threads in the server. In a cluster, clustered clients that
are connected or attempting to connect will fail over to another cluster member.
Link : [http://e-docs.bea.com/wls/docs103/config_wls/overload.html] -
Oracle Service Bus "ThreadPool has stuck threads"
Hi all,
I have JMS Proxy service that listen to the queue, with small quantity of message every second, it works great.
But after put message with bigger quantity and faster insertion on queue, the proxy service cannot
pull the message anymore, the error message is "STUCK THREAD", it makes the weblogic console give warning status.
This is the Dump Thread Stacks :
===== FULL THREAD DUMP ===============
Tue Jun 8 11:30:24 2010
BEA JRockit(R) R27.6.1-20-104573-1.6.0_05-20081016-1236-linux-ia32
"Main Thread" id=1 idx=0x4 tid=7168 prio=5 alive, in native, waiting
-- Waiting for notification on: weblogic/t3/srvr/T3Srvr@0x1cd20688[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at weblogic/t3/srvr/T3Srvr.waitForDeath(T3Srvr.java:811)
^-- Lock released while waiting: weblogic/t3/srvr/T3Srvr@0x1cd20688[fat lock]
at weblogic/t3/srvr/T3Srvr.run(T3Srvr.java:459)
at weblogic/Server.main(Server.java:67)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"(Signal Handler)" id=2 idx=0x8 tid=7169 prio=5 alive, in native, daemon
"(GC Main Thread)" id=3 idx=0xc tid=7170 prio=5 alive, in native, native_waiting, daemon
"(GC Worker Thread 1)" id=? idx=0x10 tid=7171 prio=5 alive, in native, daemon
"(GC Worker Thread 2)" id=? idx=0x14 tid=7172 prio=5 alive, in native, daemon
"(GC Worker Thread 3)" id=? idx=0x18 tid=7173 prio=5 alive, in native, daemon
"(GC Worker Thread 4)" id=? idx=0x1c tid=7174 prio=5 alive, in native, daemon
"(GC Worker Thread 5)" id=? idx=0x20 tid=7175 prio=5 alive, in native, daemon
"(GC Worker Thread 6)" id=? idx=0x24 tid=7176 prio=5 alive, in native, daemon
"(GC Worker Thread 7)" id=? idx=0x28 tid=7177 prio=5 alive, in native, daemon
"(GC Worker Thread 8)" id=? idx=0x2c tid=7178 prio=5 alive, in native, daemon
"(Code Generation Thread 1)" id=4 idx=0x30 tid=7179 prio=5 alive, in native, native_waiting, daemon
"(Code Optimization Thread 1)" id=5 idx=0x34 tid=7180 prio=5 alive, in native, native_waiting, daemon
"(VM Periodic Task)" id=6 idx=0x38 tid=7181 prio=10 alive, in native, daemon
Is there any tuning that I need to do on Jrockit?
Really thanks in advance.
Regards,
Mr.KHi Eric,
Thank you for the link, I'll try for increasing it.
Hi Manoj,
hope it'll give you some additional information for helping me.
####<Jun 8, 2010 10:43:44 AM WIT> <Error> <WebLogicServer> <cspproduction> <AdminServerOSB> <[ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1275968624958> <BEA-000337> <[STUCK] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "653" seconds working on the request "weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl@39c79ac", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-50 "[STUCK] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, waiting, priority=1, DAEMON> {
-- Waiting for notification on: weblogic.rjvm.ResponseImpl@39cf5b1[fat lock]
java.lang.Object.wait(Object.java:???)
weblogic.rjvm.ResponseImpl.waitForData(ResponseImpl.java:80)
weblogic.rjvm.ResponseImpl.getTxContext(ResponseImpl.java:127)
weblogic.messaging.dispatcher.DispatcherProxy.unmarshalResponse(DispatcherProxy.java:236)
weblogic.messaging.dispatcher.DispatcherProxy.dispatchSyncTranFuture(DispatcherProxy.java:133)
weblogic.messaging.dispatcher.DispatcherWrapperState.dispatchSyncTran(DispatcherWrapperState.java:333)
weblogic.messaging.dispatcher.DispatcherWrapperState.dispatchSyncNoTran(DispatcherWrapperState.java:384)
weblogic.jms.dispatcher.DispatcherAdapter.dispatchSyncNoTran(DispatcherAdapter.java:59)
weblogic.jms.client.JMSProducer.toFEProducer(JMSProducer.java:1151)
weblogic.jms.client.JMSProducer.deliveryInternal(JMSProducer.java:679)
weblogic.jms.client.JMSProducer.sendInternal(JMSProducer.java:541)
weblogic.jms.client.JMSProducer.sendWithListener(JMSProducer.java:366)
weblogic.jms.client.JMSProducer.send(JMSProducer.java:353)
weblogic.jms.client.WLProducerImpl.send(WLProducerImpl.java:1050)
weblogic.jms.client.WLProducerImpl.send(WLProducerImpl.java:107)
weblogic.deployment.jms.WrappedMessageProducer.send(WrappedMessageProducer.java:235)
com.bea.wli.sb.transports.jms.JmsOutboundMessageContext$SendAction.sendMessage(JmsOutboundMessageContext.java:823)
com.bea.wli.sb.transports.jms.JmsOutboundMessageContext$SendAction.run(JmsOutboundMessageContext.java:745)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:350)
weblogic.security.service.SecurityManager.runAs(Unknown Source)
weblogic.security.Security.runAs(Security.java:60)
com.bea.wli.sb.transports.jms.JmsOutboundMessageContext.send(JmsOutboundMessageContext.java:502)
com.bea.wli.sb.transports.jms.JmsTransportProvider.sendMessageAsync(JmsTransportProvider.java:680)
sun.reflect.GeneratedMethodAccessor302.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
com.bea.wli.sb.transports.Util$1.invoke(Util.java:56)
$Proxy61.sendMessageAsync(Unknown Source)
com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageAsync(LoadBalanceFailoverListener.java:136)
com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageToServiceAsync(LoadBalanceFailoverListener.java:510)
com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageToService(LoadBalanceFailoverListener.java:477)
com.bea.wli.sb.transports.TransportManagerImpl.sendMessageToService(TransportManagerImpl.java:491)
com.bea.wli.sb.transports.TransportManagerImpl.sendMessageAsync(TransportManagerImpl.java:405)
com.bea.wli.sb.pipeline.PipelineContextImpl.doDispatch(PipelineContextImpl.java:508)
com.bea.wli.sb.pipeline.PipelineContextImpl.dispatch(PipelineContextImpl.java:486)
stages.publish.runtime.PublishRuntimeStep.processMessage(PublishRuntimeStep.java:83)
com.bea.wli.sb.stages.StageMetadataImpl$WrapperRuntimeStep.processMessage(StageMetadataImpl.java:343)
com.bea.wli.sb.stages.impl.SequenceRuntimeStep.processMessage(SequenceRuntimeStep.java:31)
com.bea.wli.sb.pipeline.PipelineStage.processMessage(PipelineStage.java:84)
com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:840)
com.bea.wli.sb.pipeline.Pipeline.processMessage(Pipeline.java:140)
com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:840)
com.bea.wli.sb.pipeline.PipelineNode.doRequest(PipelineNode.java:51)
com.bea.wli.sb.pipeline.Node.processMessage(Node.java:65)
com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:840)
com.bea.wli.sb.pipeline.Router.processMessage(Router.java:178)
com.bea.wli.sb.pipeline.MessageProcessor.processRequest(MessageProcessor.java:39)
com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:508)
com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:506)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:350)
weblogic.security.service.SecurityManager.runAs(Unknown Source)
com.bea.wli.sb.security.WLSSecurityContextService.runAs(WLSSecurityContextService.java:55)
com.bea.wli.sb.pipeline.RouterManager.processMessage(RouterManager.java:487)
com.bea.wli.sb.transports.TransportManagerImpl.receiveMessage(TransportManagerImpl.java:248)
com.bea.wli.sb.transports.jms.JmsInboundMDB.onMessage(JmsInboundMDB.java:79)
sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:???)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:27)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:277)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:187)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:153)
com.bea.core.repackaged.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:86)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:153)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:126)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:100)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:153)
com.bea.core.repackaged.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:148)
$Proxy79.onMessage(Unknown Source)
weblogic.ejb.container.internal.MDListener.execute(MDListener.java:403)
weblogic.ejb.container.internal.MDListener.transactionalOnMessage(MDListener.java:352)
weblogic.ejb.container.internal.MDListener.onMessage(MDListener.java:264)
weblogic.jms.client.JMSSession.onMessage(JMSSession.java:4526)
weblogic.jms.client.JMSSession.execute(JMSSession.java:4022)
weblogic.jms.client.JMSSession.executeMessage(JMSSession.java:3708)
weblogic.jms.client.JMSSession.access$000(JMSSession.java:114)
weblogic.jms.client.JMSSession$UseForRunnable.run(JMSSession.java:5058)
weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:516)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:198)
weblogic.work.ExecuteThread.run(ExecuteThread.java:165)
That is error message from Oracle service bus console, it stated there is JMS Proxy service that got stuck.
On previous post, I already attached the dump trace, it explained something about jrockit. But I still don't get the way for tuning it properly.
Regards,
Mr.K -
Mapping Stuck Threads to Proxy Services
Hello OSB Experts:
Is there a way I can map the information of a thread dump showing stuck threads to the proxy service that those threads are executing?
I am facing a situation that seems to be usual: one of several backends is performing poorly, producing stuck threads in the OSB app. servers. After a while, the performance of the whole OSB is affected because of the number of threads in stuck state.
I would like to be able to pinpoint the backend with problems in order to solve the problem before the health of the whole OSB is affected.
Can this be done without 3rd party tools?
Thanks
SebastiánNo, it is not possible to get the the proxy service name from the thread dump. The OSB stuff is handled under a weblogic wrapper thread.
One of the main reason for stuck threads is some of the proxy service calls taking a lot of time through the business services and there is no appropriate timeout set for the calls on the business service.
You have couple of options to find out the culprit. You can enable tracing on the proxy services and the business services and capture the metrics that should help you find a pattern for the performance. The numbers will help you find which backend is taking more time.
You can get nmore details on OSB Monitoring @ http://docs.oracle.com/cd/E23943_01/admin.1111/e15867/monitoring_ops.htm#OSBAG472
Secondly, always set some realistic timeout value (like 30 seconds/60 seconds depending on the sla of the back-end) on the business service. Leaving it to default (that is 0) is like configuring a indefinite timeout (in which case it uses the default weblogic connection timeout setting of 10/15 minutes).
This way the threads are not stuck waiting for the response and dont cause issues.
Additionally you use OSB alerts to notify you for service behavior discrepancies w.r.t response times, number of errors etc .. Refer http://docs.oracle.com/cd/E23943_01/admin.1111/e15867/monitoring_console.htm#CACDCDAH for more details.
Hope this information helps.
Thanks,
Patrick
It is considered good etiquette to reward answerers with points (as "helpful" - 5 pts - or "correct" - 10pts).+
https://forums.oracle.com/forums/ann.jspa?annID=893 -
"ThreadPool has stuck threads" - Warning while starting server soa 11.1.1.3
Hi,
When we are trying to restart the server its taking lot of time (nearly 2 hrs) and admin server status is showing as "Warning". when we check Monitoring -> Health for admin server we can see as "Thread Pool as stuck threads".
When clicked on "threadpool", can see "ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" and current request is for "weblogic.kernel.WorkManagerWrapper$1@6ce46ce4" thread.
Any solution for what to do.
Thanks,
RRFurther Updates on this :
I further debugged and found out few more things. This might help.
In Weblogic Admin console Home > Summary of Deployments >oracle-bam(11.1.1)
MDB application oracle-bam is NOT connected to messaging system with below error.
EJBs
MessageDispatcherBean - Error java.lang.IllegalArgumentException: Getting Deployment configuration...
connectionFactoryJNDIName - Red Cross with warning
destinationJNDIName - Red Cross with warning
resourceAdapterJNDIName - Red Cross with warning
Modules
Also please check out another similar thread and please update.
BAMAdapter Issue : java.rmi.ConnectException: Destination unreachable;
Thanks,
Sagar -
WebLogic 11g - Stuck Thread issue observed during MS-ADS authentication
Hi
We have weblogic server 10.3.5 installed on RHEL 5.6. For our application authentication, we are getting user credentials authenticated for application login page using Microsoft Active Directory Service (MS-ADS) configured under WebLogic Security Realm. We are getting the below error message now due to STUCK Thread observed at JVM level.
Could you please help me to fix this issue and my thanks in advance.
]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-6424 "[STUCK] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, suspended, waiting, priority=1, DAEMON> {
-- Waiting for notification on: netscape.ldap.LDAPResponseListener@3a8ecc70[fat lock]
java.lang.Object.wait(Object.java:???)
java.lang.Object.wait(Object.java:485)
netscape.ldap.LDAPMessageQueue.waitForMessage(LDAPMessageQueue.java:198)
netscape.ldap.LDAPMessageQueue.waitFirstMessage(LDAPMessageQueue.java:100)
netscape.ldap.LDAPConnection.sendRequest(LDAPConnection.java:1782)
^-- Holding lock: netscape.ldap.LDAPConnection@3a8ecc83[thin lock]
netscape.ldap.LDAPConnection.simpleBind(LDAPConnection.java:1752)
netscape.ldap.LDAPConnection.authenticate(LDAPConnection.java:1260)
netscape.ldap.LDAPConnection.authenticate(LDAPConnection.java:1273)
netscape.ldap.LDAPConnection.authenticate(LDAPConnection.java:1240)
netscape.ldap.LDAPConnection.referralRebind(LDAPConnection.java:4955)
netscape.ldap.LDAPConnection.performReferrals(LDAPConnection.java:5065)
netscape.ldap.LDAPConnection.checkSearchMsg(LDAPConnection.java:2632)
netscape.ldap.LDAPConnection.search(LDAPConnection.java:2500)
weblogic.security.providers.authentication.LDAPAtnDelegate.getDNForUser(LDAPAtnDelegate.java:3667)
weblogic.security.providers.authentication.LDAPAtnDelegate.authenticate(LDAPAtnDelegate.java:3513)
weblogic.security.providers.authentication.LDAPAtnLoginModuleImpl.login(LDAPAtnLoginModuleImpl.java:145)
com.bea.common.security.internal.service.LoginModuleWrapper$1.run(LoginModuleWrapper.java:110)
com.bea.common.security.internal.service.LoginModuleWrapper.login(LoginModuleWrapper.java:101)
sun.reflect.GeneratedMethodAccessor2808.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
javax.security.auth.login.LoginContext.invoke(LoginContext.java:720)
javax.security.auth.login.LoginContext.access$000(LoginContext.java:186)
javax.security.auth.login.LoginContext$4.run(LoginContext.java:683)
javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
javax.security.auth.login.LoginContext.login(LoginContext.java:566)
com.bea.common.security.internal.service.JAASLoginServiceImpl.login(JAASLoginServiceImpl.java:77)
sun.reflect.GeneratedMethodAccessor366.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
com.bea.common.security.internal.utils.Delegator$ProxyInvocationHandler.invoke(Delegator.java:57)
$Proxy28.login(Unknown Source)
weblogic.security.service.internal.WLSJAASLoginServiceImpl$ServiceImpl.login(WLSJAASLoginServiceImpl.java:81)
com.bea.common.security.internal.service.JAASAuthenticationServiceImpl.authenticate(JAASAuthenticationServiceImpl.java:74)
sun.reflect.GeneratedMethodAccessor364.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
com.bea.common.security.internal.utils.Delegator$ProxyInvocationHandler.invoke(Delegator.java:57)
$Proxy46.authenticate(Unknown Source)
weblogic.security.service.WLSJAASAuthenticationServiceWrapper.authenticate(WLSJAASAuthenticationServiceWrapper.java:29)
weblogic.security.service.PrincipalAuthenticator.authenticate(PrincipalAuthenticator.java:339)
weblogic.security.services.Authentication.doLogin(Authentication.java:127)
weblogic.security.services.Authentication.login(Authentication.java:69)
weblogic.security.services.Authentication.login(Authentication.java:51)
in.sis.ps.view.template.user.UserAdapter.doLogin(UserAdapter.java:681)
sun.reflect.GeneratedMethodAccessor2824.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
com.sun.el.parser.AstValue.invoke(Unknown Source)
com.sun.el.MethodExpressionImpl.invoke(Unknown Source)
org.apache.myfaces.trinidad.component.MethodExpressionMethodBinding.invoke(MethodExpressionMethodBinding.java:46)
com.sun.faces.application.ActionListenerImpl.processAction(ActionListenerImpl.java:85)
org.apache.myfaces.trinidad.component.UIXCommand.broadcast(UIXCommand.java:171)
oracle.adf.view.rich.component.fragment.ContextSwitchingComponent$1.run(ContextSwitchingComponent.java:130)
oracle.adf.view.rich.component.fragment.ContextSwitchingComponent._processPhase(ContextSwitchingComponent.java:447)
oracle.adf.view.rich.component.fragment.ContextSwitchingComponent.broadcast(ContextSwitchingComponent.java:122)
oracle.adf.view.rich.component.fragment.UIXInclude.broadcast(UIXInclude.java:106)
oracle.adf.view.rich.component.fragment.ContextSwitchingComponent$1.run(ContextSwitchingComponent.java:130)
oracle.adf.view.rich.component.fragment.ContextSwitchingComponent._processPhase(ContextSwitchingComponent.java:447)
oracle.adf.view.rich.component.fragment.ContextSwitchingComponent.broadcast(ContextSwitchingComponent.java:122)
oracle.adf.view.rich.component.fragment.UIXInclude.broadcast(UIXInclude.java:106)
javax.faces.component.UIViewRoot.broadcastEvents(UIViewRoot.java:720)
javax.faces.component.UIViewRoot.processApplication(UIViewRoot.java:1247)
oracle.adfinternal.view.faces.lifecycle.LifecycleImpl._invokeApplication(LifecycleImpl.java:964)
oracle.adfinternal.view.faces.lifecycle.LifecycleImpl._executePhase(LifecycleImpl.java:259)
oracle.adfinternal.view.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:181)
javax.faces.webapp.FacesServlet.service(FacesServlet.java:284)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:206)
weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.adf.model.servlet.ADFBindingFilter.doFilter(ADFBindingFilter.java:115)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.adfinternal.view.faces.webapp.rich.RegistrationFilter.doFilter(RegistrationFilter.java:65)
org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:459)
oracle.adfinternal.view.faces.activedata.AdsFilter.doFilter(AdsFilter.java:44)
org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:459)
org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl._doFilterImpl(TrinidadFilterImpl.java:253)
org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl.doFilter(TrinidadFilterImpl.java:148)
org.apache.myfaces.trinidad.webapp.TrinidadFilter.doFilter(TrinidadFilter.java:91)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:111)
oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:299)
oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:405)
oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:94)
oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:98)
oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:70)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:86)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:25)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3687)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3681)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:308)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:116)
weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2213)
weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2135)
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1420)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:203)
weblogic.work.ExecuteThread.run(ExecuteThread.java:170)
}Hi,
make sure to keep your Authentication provider in provider specific select group member search as limited where as default would be unlimited.
Once you made this changes then it will solve this issue.
Regards,
Kal -
How to resolve stuck threads in weblogic
JDeveloper: 11.3
Weblogic: 10.3
Hi,
System Admin is complaining about the stuck threads in our ADF Application. Can some one help me in resolving those in weblogic server or at code level.
Basically, I am looking for an approach to resolve them.
Thanks.If you get stuck threads you should also get a dump of all running threads, indicating which threads are stuck. This is the starting point for your investigation. Try to find out which threads are stuck and what they are supposed to do.
This should be you started.
Timo -
Weblogic server + stuck threads
Hi All,
We have java application deployed on weblogic server 10.1.3.4 which is integrated with BPEL (SOA Oracle 11g). The application creates a task through BPEL. We have configured StuckThread Max Time to 600 secs. Now after certain time (5-7 days) the servers goes into warning state due to stuck threads and finally it gets crashed
We noticed the OOM error in jrockit dump, could any one help me here as how to avoid this? your help is very much appreciated, thanksRegarding
Sample of your Weblogic logs showing the java.lang.OutOfMemoryError along with Stack Trace if available
Please find below
[ACTIVE] ExecuteThread: '20' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <532776f80ba738ab:-48a8d4c8:133068f0cee:-8000-00000000000001a5> <1322080967160> <BEA-080003> <RuntimeException thrown by rmi server: javax.management.remote.rmi.RMIConnectionImpl.invoke(Ljavax.management.ObjectName;Ljava.lang.String;Ljava.rmi.MarshalledObject;[Ljava.lang.String;Ljavax.security.auth.Subject;)
javax.management.RuntimeErrorException: java.lang.OutOfMemoryError: allocLargeObjectOrArray: [B, size 1335312.
javax.management.RuntimeErrorException: java.lang.OutOfMemoryError: allocLargeObjectOrArray: [B, size 1335312
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:858)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrowMaybeMBeanException(DefaultMBeanServerInterceptor.java:869)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:838)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$16.run(WLSMBeanServerInterceptorBase.java:449)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:447)
at weblogic.management.mbeanservers.internal.JMXContextInterceptor.invoke(JMXContextInterceptor.java:263)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$16.run(WLSMBeanServerInterceptorBase.java:449)
at weblogic.management.mbeanservers.internal.SecurityInterceptor.invoke(SecurityInterceptor.java:444)
at weblogic.management.jmx.mbeanserver.WLSMBeanServer.invoke(WLSMBeanServer.java:323)
at weblogic.management.mbeanservers.internal.JMXConnectorSubjectForwarder$11.run(JMXConnectorSubjectForwarder.java:661)
at weblogic.management.mbeanservers.internal.JMXConnectorSubjectForwarder.invoke(JMXConnectorSubjectForwarder.java:654)
at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
at javax.management.remote.rmi.RMIConnectionImpl_WLSkel.invoke(Unknown Source)
at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:668)
at weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:118)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:207)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:176)
Caused By: java.lang.OutOfMemoryError: allocLargeObjectOrArray: [B, size 1335312
at java.util.Arrays.copyOf(Arrays.java:2786)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1847)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1847)
at java.io.ObjectOutputStream$BlockDataOutputStream.flush(ObjectOutputStream.java:1792)
at java.io.ObjectOutputStream.flush(ObjectOutputStream.java:699)
at java.io.ObjectOutputStream.close(ObjectOutputStream.java:720)
at oracle.dms.jmx.app.ServerAggreMBeanImpl.getTableGroupInBytes(ServerAggreMBeanImpl.java:328)
at oracle.dms.jmx.app.ServerAggreMBeanWrapper.invokeOperation(ServerAggreMBeanWrapper.java:71)
at oracle.dms.jmx.app.AggreMBeanWrapper.invoke(AggreMBeanWrapper.java:167)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$16.run(WLSMBeanServerInterceptorBase.java:449)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:447)
at weblogic.management.mbeanservers.internal.JMXContextInterceptor.invoke(JMXContextInterceptor.java:263)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$16.run(WLSMBeanServerInterceptorBase.java:449)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:447)
at weblogic.management.mbeanservers.internal.SecurityInterceptor.invoke(SecurityInterceptor.java:444)
at weblogic.management.jmx.mbeanserver.WLSMBeanServer.invoke(WLSMBeanServer.java:323)
at weblogic.management.mbeanservers.internal.JMXConnectorSubjectForwarder$11$1.run(JMXConnectorSubjectForwarder.java:663)
at weblogic.management.mbeanservers.internal.JMXConnectorSubjectForwarder$11.run(JMXConnectorSubjectForwarder.java:661)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.management.mbeanservers.internal.JMXConnectorSubjectForwarder.invoke(JMXConnectorSubjectForwarder.java:654)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1367)
at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
at javax.management.remote.rmi.RMIConnectionImpl_WLSkel.invoke(Unknown Source)
at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:667)
at weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:522)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:146)
at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:518)
at weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:118)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:207)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:176)
Maybe you are looking for
-
Dears, Can anyone tell me about a report that gives the elements of the inventory equation inventory equation: stock value at the end of a period = stock value at the begining of the same period + In transactions value - out transactions value I
-
ASL, Sourcing rules and currencies.
<p> Hi, I would appreciate any help with the following: I need to source item's price and supplier from a BPA into requisition line. BPA's currency is EUR while functional currency is USD. I have defined the following: A. Approved BPA with currency E
-
Loading freight cost on inventory
Dear All I am implementing LE .Our client want freight charges to get uploaded on inventory during STO. Will any-one please help me to get rid of this problem , I want urgent solution for this . Best Regards Parveen Kumar Thakur
-
Firefox was my default, it went to boxes with statements,now it will not load,
Firefox stopped working yesterday for no reason, when I open the default page, all I get is 9 oblong boxes stacked one on top of the other with statements and a search button for each one, but nothing happens, which means I cannot access my computer
-
Did the iPad Photo Album interface (where there is a black background and you can pinch to expand albums) come from the Mac OS? I think I've been seeing that interface before the iPad came out but I cannot do it on my MacBook Pro. Can you help me cle