Fat Lock + Stuck Thread
Hi,
My application keeps on hanging intermittently and when i check the weblogic output i see fat lock and stuck thread errors. I hope someone can help me analyse and make more sense of the following log, Please
<Aug 19, 2013 12:00:06 AM EDT> <Notice> <Diagnostics> <BEA-320068> <Watch 'StuckThread' with severity 'Notice' on server 'EPMServer0' has triggered at Aug 19, 2013 12:00:06 AM EDT. Notification details:
WatchRuleType: Log
WatchRule: (SEVERITY = 'Error') AND ((MSGID = 'WL-000337') OR (MSGID = 'BEA-000337'))
WatchData: DATE = Aug 19, 2013 12:00:06 AM EDT SERVER = EPMServer0 MESSAGE = [STUCK] ExecuteThread: '27' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "649" seconds working on the request "Workmanager: default, Version: 0, Scheduled=true, Started=true, Started time: 649292 ms
POST /aps/SmartView HTTP/1.1
Accept: */*
Accept-Language: en-US
Accept-Encoding: gzip
ORA_EPM_SVCLIENT_CLIENTIP: 203.2.17.86
ORA_EPM_SVCLIENT_EXTENSIONS: AF52322C-D60B-41f1-A8C8-0C299FBB0AA4
Content-Encoding: gzip
User-Agent: HttpApp/1.0
Content-Length: 137
Cache-Control: no-cache
Cookie: JSESSIONID=qk79SRVbLc3870MPGQ8pY6Q1B7GZL6VW2TDKDZw1mgcB03shyGLc!786230261; ORA_EPMWS_User=veli.yolal%40au.ey.com; ORA_EPMWS_AccessibilityMode=false; ORA_EPMWS_ThemeSelection=BpmTadpole
Connection: Keep-Alive
X-Forwarded-For: 203.2.17.86
Proxy-Client-IP: 203.2.17.86
X-WebLogic-KeepAliveSecs: 30
X-WebLogic-Force-JVMID: 786230261
]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-19626 "[STUCK] ExecuteThread: '27' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, suspended, blocked, priority=1, DAEMON> {
-- Blocked trying to get lock: weblogic.jdbc.common.internal.GenericConnectionPool@30a0bb8b[fat lock]
weblogic.common.resourcepool.ResourcePoolImpl.reserveResourceInternal(ResourcePoolImpl.java:378)
weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:337)
weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:330)
weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:374)
weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:363)
weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:118)
weblogic.jdbc.common.internal.RmiDataSource.getPoolConnection(RmiDataSource.java:436)
weblogic.jdbc.common.internal.RmiDataSource.getConnectionInternal(RmiDataSource.java:504)
weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:498)
weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:491)
com.hyperion.hit.registry.datasource.SharedServicesDatabaseConnectionBuilder.getConnection(SharedServicesDatabaseConnectionBuilder.java:77)
com.hyperion.hit.registry.datasource.SharedServicesDatabaseConnectionBuilder.getConnection(SharedServicesDatabaseConnectionBuilder.java:26)
com.hyperion.hit.registry.RegistryUtils.createNewConnection(RegistryUtils.java:382)
com.hyperion.hit.registry.Registry.lookupComponents(Registry.java:1041)
com.essbase.server.framework.RegistryInterfacer.getClusterNames(Unknown Source)
com.essbase.server.framework.EssClusterHandler.getClusters(Unknown Source)
com.essbase.smartview.util.SvDataSources.listDataSources(Unknown Source)
com.essbase.smartview.admin.ApsServer.listServers(Unknown Source)
com.essbase.smartview.admin.ApsHandler.listServers(Unknown Source)
sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:???)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:27)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.java:267)
org.apache.commons.digester.CallMethodRule.end(CallMethodRule.java:499)
org.apache.commons.digester.Rule.end(Rule.java:253)
org.apache.commons.digester.Digester.endElement(Digester.java:1154)
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:592)
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1734)
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2686)
com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:648)
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:450)
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:765)
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:729)
com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:118)
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1198)
com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:516)
weblogic.xml.jaxp.WebLogicXMLReader.parse(WebLogicXMLReader.java:132)
weblogic.xml.jaxp.RegistryXMLReader.parse(RegistryXMLReader.java:152)
org.apache.commons.digester.Digester.parse(Digester.java:1783)
com.essbase.smartview.xml.XmlHandler.invoke(Unknown Source)
com.essbase.smartview.xml.XmlHandler.invoke(Unknown Source)
com.essbase.smartview.xml.XmlHandler.invoke(Unknown Source)
com.essbase.smartview.xml.EssbaseSmartView.doPost(Unknown Source)
javax.servlet.http.HttpServlet.service(HttpServlet.java:700)
javax.servlet.http.HttpServlet.service(HttpServlet.java:815)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:207)
weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:119)
oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:301)
oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:434)
oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:103)
oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:107)
oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:70)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:119)
oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:301)
oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:434)
oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:103)
oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:107)
oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:70)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:87)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3702)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3696)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:308)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:116)
weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2209)
weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2131)
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1457)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:250)
weblogic.work.ExecuteThread.run(ExecuteThread.java:213)
SUBSYSTEM = WebLogicServer USERID = <WLS Kernel> SEVERITY = Error THREAD = [ACTIVE] ExecuteThread: '34' for queue: 'weblogic.kernel.Default (self-tuning)' MSGID = BEA-000337 MACHINE = SGSINBMPFTAPP1 TXID = CONTEXTID = TIMESTAMP = 1376884806884
WatchAlarmType: AutomaticReset
WatchAlarmResetPeriod: 600000
>
<Aug 19, 2013 12:01:06 AM EDT> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '27' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "709" seconds working on the request "Workmanager: default, Version: 0, Scheduled=true, Started=true, Started time: 709304 ms
POST /aps/SmartView HTTP/1.1
Accept: */*
Accept-Language: en-US
Accept-Encoding: gzip
ORA_EPM_SVCLIENT_CLIENTIP: 203.2.17.86
ORA_EPM_SVCLIENT_EXTENSIONS: AF52322C-D60B-41f1-A8C8-0C299FBB0AA4
Content-Encoding: gzip
User-Agent: HttpApp/1.0
Content-Length: 137
Cache-Control: no-cache
Cookie: JSESSIONID=qk79SRVbLc3870MPGQ8pY6Q1B7GZL6VW2TDKDZw1mgcB03shyGLc!786230261; ORA_EPMWS_User=veli.yolal%40au.ey.com; ORA_EPMWS_AccessibilityMode=false; ORA_EPMWS_ThemeSelection=BpmTadpole
Connection: Keep-Alive
X-Forwarded-For: 203.2.17.86
Proxy-Client-IP: 203.2.17.86
X-WebLogic-KeepAliveSecs: 30
X-WebLogic-Force-JVMID: 786230261
]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-19626 "[STUCK] ExecuteThread: '27' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, suspended, blocked, priority=1, DAEMON> {
-- Blocked trying to get lock: weblogic.jdbc.common.internal.GenericConnectionPool@30a0bb8b[fat lock]
weblogic.common.resourcepool.ResourcePoolImpl.reserveResourceInternal(ResourcePoolImpl.java:378)
weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:337)
weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:330)
weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:374)
weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:363)
weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:118)
weblogic.jdbc.common.internal.RmiDataSource.getPoolConnection(RmiDataSource.java:436)
weblogic.jdbc.common.internal.RmiDataSource.getConnectionInternal(RmiDataSource.java:504)
weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:498)
weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:491)
com.hyperion.hit.registry.datasource.SharedServicesDatabaseConnectionBuilder.getConnection(SharedServicesDatabaseConnectionBuilder.java:77)
com.hyperion.hit.registry.datasource.SharedServicesDatabaseConnectionBuilder.getConnection(SharedServicesDatabaseConnectionBuilder.java:26)
com.hyperion.hit.registry.RegistryUtils.createNewConnection(RegistryUtils.java:382)
com.hyperion.hit.registry.Registry.lookupComponents(Registry.java:1041)
com.essbase.server.framework.RegistryInterfacer.getClusterNames(Unknown Source)
com.essbase.server.framework.EssClusterHandler.getClusters(Unknown Source)
com.essbase.smartview.util.SvDataSources.listDataSources(Unknown Source)
com.essbase.smartview.admin.ApsServer.listServers(Unknown Source)
com.essbase.smartview.admin.ApsHandler.listServers(Unknown Source)
sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:???)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:27)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.java:267)
org.apache.commons.digester.CallMethodRule.end(CallMethodRule.java:499)
org.apache.commons.digester.Rule.end(Rule.java:253)
org.apache.commons.digester.Digester.endElement(Digester.java:1154)
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:592)
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1734)
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2686)
com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:648)
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:450)
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:765)
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:729)
com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:118)
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1198)
com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:516)
weblogic.xml.jaxp.WebLogicXMLReader.parse(WebLogicXMLReader.java:132)
weblogic.xml.jaxp.RegistryXMLReader.parse(RegistryXMLReader.java:152)
org.apache.commons.digester.Digester.parse(Digester.java:1783)
com.essbase.smartview.xml.XmlHandler.invoke(Unknown Source)
com.essbase.smartview.xml.XmlHandler.invoke(Unknown Source)
com.essbase.smartview.xml.XmlHandler.invoke(Unknown Source)
com.essbase.smartview.xml.EssbaseSmartView.doPost(Unknown Source)
javax.servlet.http.HttpServlet.service(HttpServlet.java:700)
javax.servlet.http.HttpServlet.service(HttpServlet.java:815)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:207)
weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:119)
oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:301)
oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:434)
oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:103)
oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:107)
oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:70)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:119)
oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:301)
oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:434)
oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:103)
oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:107)
oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:70)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:87)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3702)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3696)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:308)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:116)
weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2209)
weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2131)
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1457)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:250)
weblogic.work.ExecuteThread.run(ExecuteThread.java:213)
>
<Aug 19, 2013 12:06:06 AM EDT> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '28' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "610" seconds working on the request "Workmanager: default, Version: 0, Scheduled=true, Started=true, Started time: 610316 ms
POST /HyperionPlanning/SmartView HTTP/1.1
Accept: */*
Accept-Language: en-US
Accept-Encoding: gzip
ORA_EPM_SVCLIENT_CLIENTIP: 203.2.17.86
ORA_EPM_SVCLIENT_EXTENSIONS: AF52322C-D60B-41f1-A8C8-0C299FBB0AA4
Content-Encoding: gzip
User-Agent: HttpApp/1.0
Content-Length: 196
Cache-Control: no-cache
Cookie: JSESSIONID=q8vvSRXLVvKTwD8M3mJhhJKHJ5YNGD0vLz6SgHyzc0TrhC6jV9pT!786230261; ORA_EPMWS_User=veli.yolal%40au.ey.com; ORA_EPMWS_AccessibilityMode=false; ORA_EPMWS_ThemeSelection=BpmTadpole
Connection: Keep-Alive
X-Forwarded-For: 203.2.17.86
Proxy-Client-IP: 203.2.17.86
X-WebLogic-KeepAliveSecs: 30
X-WebLogic-Force-JVMID: 786230261
]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-19652 "[STUCK] ExecuteThread: '28' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, suspended, blocked, priority=1, DAEMON> {
-- Blocked trying to get lock: com.hyperion.css.cache.Cache$ObjMonitor@31c747c2[fat lock]
com.hyperion.css.cache.Cache.getDeltaUpdates(Cache.java:89)
com.hyperion.css.cache.CacheManager.updateDeltaCache(CacheManager.java:964)
com.hyperion.css.facade.CSSFacade.processDeltaUpdate(CSSFacade.java:87)
com.hyperion.css.facade.CSSUserProvisioningFacade.getProvisionedApps(CSSUserProvisioningFacade.java:495)
com.hyperion.planning.security.HspCSS.listAppsForUser(HspCSS.java:127)
com.hyperion.planning.HspSmartViewProvider.listApplications(HspSmartViewProvider.java:1290)
sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:???)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:27)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.java:267)
org.apache.commons.digester.CallMethodRule.end(CallMethodRule.java:499)
org.apache.commons.digester.Rule.end(Rule.java:253)
org.apache.commons.digester.Digester.endElement(Digester.java:1154)
oracle.xml.parser.v2.NonValidatingParser.parseElement(NonValidatingParser.java:1510)
oracle.xml.parser.v2.NonValidatingParser.parseRootElement(NonValidatingParser.java:407)
oracle.xml.parser.v2.NonValidatingParser.parseDocument(NonValidatingParser.java:328)
oracle.xml.parser.v2.XMLParser.parse(XMLParser.java:217)
weblogic.xml.jaxp.RegistryXMLReader.parse(RegistryXMLReader.java:152)
org.apache.commons.digester.Digester.parse(Digester.java:1763)
HspSmartViewServlet.Handle(HspSmartViewServlet.java:223)
HspSmartViewServlet.Handle(HspSmartViewServlet.java:143)
HspSmartViewServlet.doPost(HspSmartViewServlet.java:137)
javax.servlet.http.HttpServlet.service(HttpServlet.java:700)
javax.servlet.http.HttpServlet.service(HttpServlet.java:815)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:207)
weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
HspValidationFilter.doFilter(HspValidationFilter.java:9)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:119)
oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:301)
oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:434)
oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:103)
oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:107)
oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:70)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:87)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:25)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3702)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3696)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:308)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:116)
weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2209)
weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2131)
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1457)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:250)
weblogic.work.ExecuteThread.run(ExecuteThread.java:213)
>
<Aug 19, 2013 12:07:06 AM EDT> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '28' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "670" seconds working on the request "Workmanager: default, Version: 0, Scheduled=true, Started=true, Started time: 670319 ms
POST /HyperionPlanning/SmartView HTTP/1.1
Accept: */*
Accept-Language: en-US
Accept-Encoding: gzip
ORA_EPM_SVCLIENT_CLIENTIP: 203.2.17.86
ORA_EPM_SVCLIENT_EXTENSIONS: AF52322C-D60B-41f1-A8C8-0C299FBB0AA4
Content-Encoding: gzip
User-Agent: HttpApp/1.0
Content-Length: 196
Cache-Control: no-cache
Cookie: JSESSIONID=q8vvSRXLVvKTwD8M3mJhhJKHJ5YNGD0vLz6SgHyzc0TrhC6jV9pT!786230261; ORA_EPMWS_User=veli.yolal%40au.ey.com; ORA_EPMWS_AccessibilityMode=false; ORA_EPMWS_ThemeSelection=BpmTadpole
Connection: Keep-Alive
X-Forwarded-For: 203.2.17.86
Proxy-Client-IP: 203.2.17.86
X-WebLogic-KeepAliveSecs: 30
X-WebLogic-Force-JVMID: 786230261
]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-19652 "[STUCK] ExecuteThread: '28' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, suspended, blocked, priority=1, DAEMON> {
-- Blocked trying to get lock: com.hyperion.css.cache.Cache$ObjMonitor@31c747c2[fat lock]
com.hyperion.css.cache.Cache.getDeltaUpdates(Cache.java:89)
com.hyperion.css.cache.CacheManager.updateDeltaCache(CacheManager.java:964)
com.hyperion.css.facade.CSSFacade.processDeltaUpdate(CSSFacade.java:87)
com.hyperion.css.facade.CSSUserProvisioningFacade.getProvisionedApps(CSSUserProvisioningFacade.java:495)
com.hyperion.planning.security.HspCSS.listAppsForUser(HspCSS.java:127)
com.hyperion.planning.HspSmartViewProvider.listApplications(HspSmartViewProvider.java:1290)
sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:???)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:27)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.java:267)
org.apache.commons.digester.CallMethodRule.end(CallMethodRule.java:499)
org.apache.commons.digester.Rule.end(Rule.java:253)
org.apache.commons.digester.Digester.endElement(Digester.java:1154)
oracle.xml.parser.v2.NonValidatingParser.parseElement(NonValidatingParser.java:1510)
oracle.xml.parser.v2.NonValidatingParser.parseRootElement(NonValidatingParser.java:407)
oracle.xml.parser.v2.NonValidatingParser.parseDocument(NonValidatingParser.java:328)
oracle.xml.parser.v2.XMLParser.parse(XMLParser.java:217)
weblogic.xml.jaxp.RegistryXMLReader.parse(RegistryXMLReader.java:152)
org.apache.commons.digester.Digester.parse(Digester.java:1763)
HspSmartViewServlet.Handle(HspSmartViewServlet.java:223)
HspSmartViewServlet.Handle(HspSmartViewServlet.java:143)
HspSmartViewServlet.doPost(HspSmartViewServlet.java:137)
javax.servlet.http.HttpServlet.service(HttpServlet.java:700)
javax.servlet.http.HttpServlet.service(HttpServlet.java:815)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:207)
weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
HspValidationFilter.doFilter(HspValidationFilter.java:9)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:119)
oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:301)
oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:434)
oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:103)
oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:107)
oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:70)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:87)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:25)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3702)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3696)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:308)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:116)
weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2209)
weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2131)
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1457)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:250)
weblogic.work.ExecuteThread.run(ExecuteThread.java:213)
>
<Aug 19, 2013 12:09:06 AM EDT> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "629" seconds working on the request "Workmanager: default, Version: 0, Scheduled=true, Started=true, Started time: 629117 ms
POST /aps/SmartView HTTP/1.1
Accept: */*
Accept-Language: en-US
Accept-Encoding: gzip
ORA_EPM_SVCLIENT_CLIENTIP: 10.236.1.121;192.168.1.5
ORA_EPM_SVCLIENT_EXTENSIONS: AF52322C-D60B-41f1-A8C8-0C299FBB0AA4
Content-Encoding: gzip
User-Agent: HttpApp/1.0
Content-Length: 136
Cache-Control: no-cache
Cookie: JSESSIONID=kCvXSRXLzL7yKgz1SkZTyWJzCmzrBTQ1W2F5fLvT9XqZvYln1n8l!786230261; ORA_EPMWS_User=Alvin-LQ.Zhang%40cn.ey.com; ORA_EPMWS_AccessibilityMode=false; ORA_EPMWS_ThemeSelection=BpmTadpole
Connection: Keep-Alive
X-Forwarded-For: 10.236.1.121
Proxy-Client-IP: 10.236.1.121
X-WebLogic-KeepAliveSecs: 30
X-WebLogic-Force-JVMID: 786230261
]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-19098 "[STUCK] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, suspended, blocked, priority=1, DAEMON> {
-- Blocked trying to get lock: weblogic.jdbc.common.internal.GenericConnectionPool@30a0bb8b[fat lock]
weblogic.common.resourcepool.ResourcePoolImpl.reserveResourceInternal(ResourcePoolImpl.java:378)
weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:337)
weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:330)
weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:374)
weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:363)
weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:118)
weblogic.jdbc.common.internal.RmiDataSource.getPoolConnection(RmiDataSource.java:436)
weblogic.jdbc.common.internal.RmiDataSource.getConnectionInternal(RmiDataSource.java:504)
weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:498)
weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:491)
com.hyperion.hit.registry.datasource.SharedServicesDatabaseConnectionBuilder.getConnection(SharedServicesDatabaseConnectionBuilder.java:77)
com.hyperion.hit.registry.datasource.SharedServicesDatabaseConnectionBuilder.getConnection(SharedServicesDatabaseConnectionBuilder.java:26)
com.hyperion.hit.registry.RegistryUtils.createNewConnection(RegistryUtils.java:382)
com.hyperion.hit.registry.Registry.lookupComponents(Registry.java:1041)
com.essbase.server.framework.RegistryInterfacer.getClusterNames(Unknown Source)
com.essbase.server.framework.EssClusterHandler.getClusters(Unknown Source)
com.essbase.smartview.util.SvDataSources.listDataSources(Unknown Source)
com.essbase.smartview.admin.ApsServer.listServers(Unknown Source)
com.essbase.smartview.admin.ApsHandler.listServers(Unknown Source)
sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:???)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:27)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.java:267)
org.apache.commons.digester.CallMethodRule.end(CallMethodRule.java:499)
org.apache.commons.digester.Rule.end(Rule.java:253)
org.apache.commons.digester.Digester.endElement(Digester.java:1154)
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:592)
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1734)
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2686)
com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:648)
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:450)
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:765)
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:729)
com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:118)
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1198)
com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:516)
weblogic.xml.jaxp.WebLogicXMLReader.parse(WebLogicXMLReader.java:132)
weblogic.xml.jaxp.RegistryXMLReader.parse(RegistryXMLReader.java:152)
org.apache.commons.digester.Digester.parse(Digester.java:1783)
com.essbase.smartview.xml.XmlHandler.invoke(Unknown Source)
com.essbase.smartview.xml.XmlHandler.invoke(Unknown Source)
com.essbase.smartview.xml.XmlHandler.invoke(Unknown Source)
com.essbase.smartview.xml.EssbaseSmartView.doPost(Unknown Source)
javax.servlet.http.HttpServlet.service(HttpServlet.java:700)
javax.servlet.http.HttpServlet.service(HttpServlet.java:815)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:207)
weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:119)
oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:301)
oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:434)
oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:103)
oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:107)
oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:70)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:119)
oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:301)
oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:434)
oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:103)
oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:107)
oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:70)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:87)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3702)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3696)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:308)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:116)
weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2209)
weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2131)
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1457)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:250)
weblogic.work.ExecuteThread.run(ExecuteThread.java:213)
>
<Aug 19, 2013 12:10:06 AM EDT> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '33' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "642" seconds working on the request "Workmanager: default, Version: 0, Scheduled=true, Started=true, Started time: 642309 ms
POST /aps/SmartView HTTP/1.1
Accept: */*
Accept-Language: en-US
Accept-Encoding: gzip
ORA_EPM_SVCLIENT_CLIENTIP: 10.236.1.121;192.168.1.5
ORA_EPM_SVCLIENT_EXTENSIONS: AF52322C-D60B-41f1-A8C8-0C299FBB0AA4
Content-Encoding: gzip
User-Agent: HttpApp/1.0
Content-Length: 135
Cache-Control: no-cache
Cookie: JSESSIONID=kCvXSRXLzL7yKgz1SkZTyWJzCmzrBTQ1W2F5fLvT9XqZvYln1n8l!786230261; ORA_EPMWS_User=Alvin-LQ.Zhang%40cn.ey.com; ORA_EPMWS_AccessibilityMode=false; ORA_EPMWS_ThemeSelection=BpmTadpole
Connection: Keep-Alive
X-Forwarded-For: 10.236.1.121
Proxy-Client-IP: 10.236.1.121
X-WebLogic-KeepAliveSecs: 30
X-WebLogic-Force-JVMID: 786230261
]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-19657 "[STUCK] ExecuteThread: '33' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, suspended, blocked, priority=1, DAEMON> {
-- Blocked trying to get lock: weblogic.jdbc.common.internal.GenericConnectionPool@30a0bb8b[fat lock]
weblogic.common.resourcepool.ResourcePoolImpl.reserveResourceInternal(ResourcePoolImpl.java:378)
weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:337)
weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:330)
weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:374)
weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:363)
weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:118)
weblogic.jdbc.common.internal.RmiDataSource.getPoolConnection(RmiDataSource.java:436)
weblogic.jdbc.common.internal.RmiDataSource.getConnectionInternal(RmiDataSource.java:504)
weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:498)
weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:491)
com.hyperion.hit.registry.datasource.SharedServicesDatabaseConnectionBuilder.getConnection(SharedServicesDatabaseConnectionBuilder.java:77)
com.hyperion.hit.registry.datasource.SharedServicesDatabaseConnectionBuilder.getConnection(SharedServicesDatabaseConnectionBuilder.java:26)
com.hyperion.hit.registry.RegistryUtils.createNewConnection(RegistryUtils.java:382)
com.hyperion.hit.registry.Registry.lookupComponents(Registry.java:1041)
com.essbase.server.framework.RegistryInterfacer.getClusterNames(Unknown Source)
com.essbase.server.framework.EssClusterHandler.getClusters(Unknown Source)
com.essbase.smartview.util.SvDataSources.listDataSources(Unknown Source)
com.essbase.smartview.admin.ApsServer.listServers(Unknown Source)
com.essbase.smartview.admin.ApsHandler.listServers(Unknown Source)
sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:???)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:27)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.java:267)
org.apache.commons.digester.CallMethodRule.end(CallMethodRule.java:499)
org.apache.commons.digester.Rule.end(Rule.java:253)
org.apache.commons.digester.Digester.endElement(Digester.java:1154)
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:592)
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1734)
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2686)
com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:648)
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:450)
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:765)
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:729)
com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:118)
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1198)
com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:516)
weblogic.xml.jaxp.WebLogicXMLReader.parse(WebLogicXMLReader.java:132)
weblogic.xml.jaxp.RegistryXMLReader.parse(RegistryXMLReader.java:152)
org.apache.commons.digester.Digester.parse(Digester.java:1783)
com.essbase.smartview.xml.XmlHandler.invoke(Unknown Source)
com.essbase.smartview.xml.XmlHandler.invoke(Unknown Source)
com.essbase.smartview.xml.XmlHandler.invoke(Unknown Source)
com.essbase.smartview.xml.EssbaseSmartView.doPost(Unknown Source)
javax.servlet.http.HttpServlet.service(HttpServlet.java:700)
javax.servlet.http.HttpServlet.service(HttpServlet.java:815)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:207)
weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:119)
oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:301)
oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:434)
oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:103)
oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:107)
oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:70)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:119)
oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:301)
oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:434)
oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:103)
oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:107)
oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:70)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:87)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3702)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3696)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:308)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:116)
weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2209)
weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2131)
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1457)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:250)
weblogic.work.ExecuteThread.run(ExecuteThread.java:213)
>
<Aug 19, 2013 12:10:06 AM EDT> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "689" seconds working on the request "Workmanager: default, Version: 0, Scheduled=true, Started=true, Started time: 689128 ms
POST /aps/SmartView HTTP/1.1
Accept: */*
Accept-Language: en-US
Accept-Encoding: gzip
ORA_EPM_SVCLIENT_CLIENTIP: 10.236.1.121;192.168.1.5
ORA_EPM_SVCLIENT_EXTENSIONS: AF52322C-D60B-41f1-A8C8-0C299FBB0AA4
Content-Encoding: gzip
User-Agent: HttpApp/1.0
Content-Length: 136
Cache-Control: no-cache
Cookie: JSESSIONID=kCvXSRXLzL7yKgz1SkZTyWJzCmzrBTQ1W2F5fLvT9XqZvYln1n8l!786230261; ORA_EPMWS_User=Alvin-LQ.Zhang%40cn.ey.com; ORA_EPMWS_AccessibilityMode=false; ORA_EPMWS_ThemeSelection=BpmTadpole
Connection: Keep-Alive
X-Forwarded-For: 10.236.1.121
Proxy-Client-IP: 10.236.1.121
X-WebLogic-KeepAliveSecs: 30
X-WebLogic-Force-JVMID: 786230261
]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-19098 "[STUCK] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, suspended, blocked, priority=1, DAEMON> {
-- Blocked trying to get lock: weblogic.jdbc.common.internal.GenericConnectionPool@30a0bb8b[fat lock]
weblogic.common.resourcepool.ResourcePoolImpl.reserveResourceInternal(ResourcePoolImpl.java:378)
weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:337)
weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:330)
weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:374)
weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:363)
weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:118)
weblogic.jdbc.common.internal.RmiDataSource.getPoolConnection(RmiDataSource.java:436)
weblogic.jdbc.common.internal.RmiDataSource.getConnectionInternal(RmiDataSource.java:504)
weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:498)
weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:491)
com.hyperion.hit.registry.datasource.SharedServicesDatabaseConnectionBuilder.getConnection(SharedServicesDatabaseConnectionBuilder.java:77)
com.hyperion.hit.registry.datasource.SharedServicesDatabaseConnectionBuilder.getConnection(SharedServicesDatabaseConnectionBuilder.java:26)
com.hyperion.hit.registry.RegistryUtils.createNewConnection(RegistryUtils.java:382)
com.hyperion.hit.registry.Registry.lookupComponents(Registry.java:1041)
com.essbase.server.framework.RegistryInterfacer.getClusterNames(Unknown Source)
com.essbase.server.framework.EssClusterHandler.getClusters(Unknown Source)
com.essbase.smartview.util.SvDataSources.listDataSources(Unknown Source)
com.essbase.smartview.admin.ApsServer.listServers(Unknown Source)
com.essbase.smartview.admin.ApsHandler.listServers(Unknown Source)
sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:???)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:27)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.java:267)
org.apache.commons.digester.CallMethodRule.end(CallMethodRule.java:499)
org.apache.commons.digester.Rule.end(Rule.java:253)
org.apache.commons.digester.Digester.endElement(Digester.java:1154)
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:592)
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1734)
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2686)
com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:648)
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:450)
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:765)
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:729)
com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:118)
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1198)
com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:516)
weblogic.xml.jaxp.WebLogicXMLReader.parse(WebLogicXMLReader.java:132)
weblogic.xml.jaxp.RegistryXMLReader.parse(RegistryXMLReader.java:152)
org.apache.commons.digester.Digester.parse(Digester.java:1783)
com.essbase.smartview.xml.XmlHandler.invoke(Unknown Source)
com.essbase.smartview.xml.XmlHandler.invoke(Unknown Source)
com.essbase.smartview.xml.XmlHandler.invoke(Unknown Source)
com.essbase.smartview.xml.EssbaseSmartView.doPost(Unknown Source)
javax.servlet.http.HttpServlet.service(HttpServlet.java:700)
javax.servlet.http.HttpServlet.service(HttpServlet.java:815)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:207)
weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:119)
oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:301)
oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:434)
oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:103)
oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:107)
oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:70)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:119)
oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:301)
oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:434)
oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:103)
oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:107)
oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:70)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:87)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3702)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3696)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:308)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:116)
weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2209)
weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2131)
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1457)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:250)
weblogic.work.ExecuteThread.run(ExecuteThread.java:213)
>
<Aug 19, 2013 12:11:06 AM EDT> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '33' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "702" seconds working on the request "Workmanager: default, Version: 0, Scheduled=true, Started=true, Started time: 702315 ms
POST /aps/SmartView HTTP/1.1
Accept: */*
Accept-Language: en-US
Accept-Encoding: gzip
ORA_EPM_SVCLIENT_CLIENTIP: 10.236.1.121;192.168.1.5
ORA_EPM_SVCLIENT_EXTENSIONS: AF52322C-D60B-41f1-A8C8-0C299FBB0AA4
Content-Encoding: gzip
User-Agent: HttpApp/1.0
Content-Length: 135
Cache-Control: no-cache
Cookie: JSESSIONID=kCvXSRXLzL7yKgz1SkZTyWJzCmzrBTQ1W2F5fLvT9XqZvYln1n8l!786230261; ORA_EPMWS_User=Alvin-LQ.Zhang%40cn.ey.com; ORA_EPMWS_AccessibilityMode=false; ORA_EPMWS_ThemeSelection=BpmTadpole
Connection: Keep-Alive
X-Forwarded-For: 10.236.1.121
Proxy-Client-IP: 10.236.1.121
X-WebLogic-KeepAliveSecs: 30
X-WebLogic-Force-JVMID: 786230261
]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-19657 "[STUCK] ExecuteThread: '33' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, suspended, blocked, priority=1, DAEMON> {
-- Blocked trying to get lock: weblogic.jdbc.common.internal.GenericConnectionPool@30a0bb8b[fat lock]
weblogic.common.resourcepool.ResourcePoolImpl.reserveResourceInternal(ResourcePoolImpl.java:378)
weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:337)
weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:330)
weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:374)
weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:363)
weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:118)
weblogic.jdbc.common.internal.RmiDataSource.getPoolConnection(RmiDataSource.java:436)
weblogic.jdbc.common.internal.RmiDataSource.getConnectionInternal(RmiDataSource.java:504)
weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:498)
weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:491)
com.hyperion.hit.registry.datasource.SharedServicesDatabaseConnectionBuilder.getConnection(SharedServicesDatabaseConnectionBuilder.java:77)
com.hyperion.hit.registry.datasource.SharedServicesDatabaseConnectionBuilder.getConnection(SharedServicesDatabaseConnectionBuilder.java:26)
com.hyperion.hit.registry.RegistryUtils.createNewConnection(RegistryUtils.java:382)
com.hyperion.hit.registry.Registry.lookupComponents(Registry.java:1041)
com.essbase.server.framework.RegistryInterfacer.getClusterNames(Unknown Source)
com.essbase.server.framework.EssClusterHandler.getClusters(Unknown Source)
com.essbase.smartview.util.SvDataSources.listDataSources(Unknown Source)
com.essbase.smartview.admin.ApsServer.listServers(Unknown Source)
com.essbase.smartview.admin.ApsHandler.listServers(Unknown Source)
sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:???)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:27)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.java:267)
org.apache.commons.digester.CallMethodRule.end(CallMethodRule.java:499)
org.apache.commons.digester.Rule.end(Rule.java:253)
org.apache.commons.digester.Digester.endElement(Digester.java:1154)
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:592)
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1734)
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2686)
com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:648)
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:450)
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:765)
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:729)
com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:118)
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1198)
com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:516)
weblogic.xml.jaxp.WebLogicXMLReader.parse(WebLogicXMLReader.java:132)
weblogic.xml.jaxp.RegistryXMLReader.parse(RegistryXMLReader.java:152)
org.apache.commons.digester.Digester.parse(Digester.java:1783)
com.essbase.smartview.xml.XmlHandler.invoke(Unknown Source)
com.essbase.smartview.xml.XmlHandler.invoke(Unknown Source)
com.essbase.smartview.xml.XmlHandler.invoke(Unknown Source)
com.essbase.smartview.xml.EssbaseSmartView.doPost(Unknown Source)
javax.servlet.http.HttpServlet.service(HttpServlet.java:700)
javax.servlet.http.HttpServlet.service(HttpServlet.java:815)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:207)
weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:119)
oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:301)
oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:434)
oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:103)
oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:107)
oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:70)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:119)
oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:301)
oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:434)
oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:103)
oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:107)
oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:70)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:87)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3702)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3696)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:308)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:116)
weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2209)
weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2131)
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1457)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:250)
weblogic.work.ExecuteThread.run(ExecuteThread.java:213)
>
<Aug 19, 2013 12:11:06 AM EDT> <Notice> <Diagnostics> <BEA-320068> <Watch 'StuckThread' with severity 'Notice' on server 'EPMServer0' has triggered at Aug 19, 2013 12:11:06 AM EDT. Notification details:
WatchRuleType: Log
WatchRule: (SEVERITY = 'Error') AND ((MSGID = 'WL-000337') OR (MSGID = 'BEA-000337'))
WatchData: DATE = Aug 19, 2013 12:11:06 AM EDT SERVER = EPMServer0 MESSAGE = [STUCK] ExecuteThread: '33' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "702" seconds working on the request "Workmanager: default, Version: 0, Scheduled=true, Started=true, Started time: 702315 ms
POST /aps/SmartView HTTP/1.1
Accept: */*
Accept-Language: en-US
Accept-Encoding: gzip
ORA_EPM_SVCLIENT_CLIENTIP: 10.236.1.121;192.168.1.5
ORA_EPM_SVCLIENT_EXTENSIONS: AF52322C-D60B-41f1-A8C8-0C299FBB0AA4
Content-Encoding: gzip
User-Agent: HttpApp/1.0
Content-Length: 135
Cache-Control: no-cache
Cookie: JSESSIONID=kCvXSRXLzL7yKgz1SkZTyWJzCmzrBTQ1W2F5fLvT9XqZvYln1n8l!786230261; ORA_EPMWS_User=Alvin-LQ.Zhang%40cn.ey.com; ORA_EPMWS_AccessibilityMode=false; ORA_EPMWS_ThemeSelection=BpmTadpole
Connection: Keep-Alive
X-Forwarded-For: 10.236.1.121
Proxy-Client-IP: 10.236.1.121
X-WebLogic-KeepAliveSecs: 30
X-WebLogic-Force-JVMID: 786230261
]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-19657 "[STUCK] ExecuteThread: '33' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, suspended, blocked, priority=1, DAEMON> {
-- Blocked trying to get lock: weblogic.jdbc.common.internal.GenericConnectionPool@30a0bb8b[fat lock]
weblogic.common.resourcepool.ResourcePoolImpl.reserveResourceInternal(ResourcePoolImpl.java:378)
weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:337)
weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:330)
weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:374)
weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:363)
weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:118)
weblogic.jdbc.common.internal.RmiDataSource.getPoolConnection(RmiDataSource.java:436)
weblogic.jdbc.common.internal.RmiDataSource.getConnectionInternal(RmiDataSource.java:504)
weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:498)
weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:491)
com.hyperion.hit.registry.datasource.SharedServicesDatabaseConnectionBuilder.getConnection(SharedServicesDatabaseConnectionBuilder.java:77)
com.hyperion.hit.registry.datas
Hi
I see many threads blocked trying to get a database connection "weblogic.jdbc.common.internal.ConnectionPool.reserve(" ...Isuggest you to check connection pools to see if they are overloaded.
Similar Messages
-
Stuck Threads on weblogic 10.3.2 Holding lock HttpURLConnection
We are having issue with STUCK THREADS on weblogic 10.3.2. Basically my application makes a lot of web service calls over HTTPS and once in a while I see few STUCK THREADS and most of the time they clear but sometimes I have to bounce my server to clean.
I have a thread dump, according to this its holding lock at weblogic.net.http.HttpURLConnection.getInputStream
Any help to resolve this problem is highly appreciated.
ThreadDump:
<May 24, 2011 9:59:08 AM EDT> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "604" seconds working on the request "weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl@b607d75", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-75 "[STUCK] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, priority=1, DAEMON> {
jrockit.net.SocketNativeIO.readBytesPinned(SocketNativeIO.java:???)
jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:31)
java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)
java.net.SocketInputStream.read(SocketInputStream.java:107)
weblogic.utils.io.ChunkedInputStream.read(ChunkedInputStream.java:149)
java.io.InputStream.read(InputStream.java:85)
com.certicom.tls.record.ReadHandler.readFragment(Unknown Source)
com.certicom.tls.record.ReadHandler.readRecord(Unknown Source)
com.certicom.tls.record.ReadHandler.read(Unknown Source)
^-- Holding lock: com.certicom.tls.record.ReadHandler@bec563c[thin lock]
com.certicom.io.InputSSLIOStreamWrapper.read(Unknown Source)
java.io.BufferedInputStream.fill(BufferedInputStream.java:189)
java.io.BufferedInputStream.read(BufferedInputStream.java:236)
^-- Holding lock: java.io.BufferedInputStream@bec5d9f[thin lock]
weblogic.net.http.MessageHeader.isHTTP(MessageHeader.java:214)
weblogic.net.http.MessageHeader.parseHeader(MessageHeader.java:141)
weblogic.net.http.HttpClient.parseHTTP(HttpClient.java:452)
^-- Holding lock: weblogic.net.http.HttpsClient@bec5202[thin lock]
weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:328)
weblogic.net.http.SOAPHttpsURLConnection.getInputStream(SOAPHttpsURLConnection.java:37)
^-- Holding lock: weblogic.net.http.SOAPHttpsURLConnection@bd3b209[thin lock]
weblogic.net.http.HttpURLConnection.getResponseCode(HttpURLConnection.java:939)
com.sun.xml.ws.transport.http.client.HttpClientTransport.readResponseCodeAndMessage(HttpClientTransport.java:213)
com.sun.xml.ws.transport.http.client.HttpTransportPipe.process(HttpTransportPipe.java:116)
com.sun.xml.ws.transport.http.client.HttpTransportPipe.processRequest(HttpTransportPipe.java:101)
com.sun.xml.ws.api.pipe.Fiber.__doRun(Fiber.java:580)
com.sun.xml.ws.api.pipe.Fiber._doRun(Fiber.java:553)
com.sun.xml.ws.api.pipe.Fiber.doRun(Fiber.java:535)
com.sun.xml.ws.api.pipe.Fiber.runSync(Fiber.java:433)
^-- Holding lock: com.sun.xml.ws.api.pipe.Fiber@bd3afa8[thin lock]
com.sun.xml.ws.client.Stub.process(Stub.java:238)
com.sun.xml.ws.client.sei.SEIStub.doProcess(SEIStub.java:135)
com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:102)
com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:89)
com.sun.xml.ws.client.sei.SEIStub.invoke(SEIStub.java:116)
$Proxy107.jobSubmit(Unknown Source)
com.somejob.invokeService(JobOperation.java:53)
com.somejobws.operations.ticket.execute(ticket.java:30)
com.somejobutil.Somehelper.process(Somehelper.java:174)
com.somejobutil.Somehelper.processJob(Somehelper.java:147)
com.somejobutil.Somehelper.processMDB(Somehelper.java:91)
com.somejobejb.ProcessMDB.onMessage(ProcessMDB.java:28)
sun.reflect.GeneratedMethodAccessor109.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:102)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:86)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:102)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:151)
$Proxy67.onMessage(Unknown Source)
weblogic.ejb.container.internal.MDListener.execute(MDListener.java:403)
weblogic.ejb.container.internal.MDListener.transactionalOnMessage(MDListener.java:352)
weblogic.ejb.container.internal.MDListener.onMessage(MDListener.java:264)
weblogic.jms.client.JMSSession.onMessage(JMSSession.java:4564)
weblogic.jms.client.JMSSession.execute(JMSSession.java:4060)
weblogic.jms.client.JMSSession.executeMessage(JMSSession.java:3746)
weblogic.jms.client.JMSSession.access$000(JMSSession.java:114)
weblogic.jms.client.JMSSession$UseForRunnable.run(JMSSession.java:5096)
weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:516)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:198)
weblogic.work.ExecuteThread.run(ExecuteThread.java:165)
}Yes, we have the same problem, stuck thread on weblogic.net.http.HttpsURLConnection.getResponseCode() (actually implemented by weblogic.net.http.SOAPHttpsURLConnection which also appears to be deprecated ???)
When we use the class javax.net.ssl.HttpsURLConnection the getResponseCode() method works perfectly well, so it looks like the weblogic implementation is to blame.
This is becoming rather urgent now because we're looking to migrate from OC4J to Weblogic and this issue has therefore become a blocker. -
SOA 11g: Intergation with BAM 11g failed with ThreadPool has stuck threads
Hi,
I have installed BAM and SOA server on local host. Now while integrating BAM with BAMAdapter it's going into long running state and finally failing with below error.
<Notice> <Diagnostics> <mars.as.local> <soa_server1> <[ACTIVE] ExecuteThread: '14' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <c82d73b0a2a7776f:37a0afd7:130d1278d97:-8000-0000000000028a7f> <1309264042380> <BEA-320068> <Watch 'StuckThread' with severity 'Notice' on server 'soa_server1' has triggered at Jun 28, 2011 1:27:22 PM BST. Notification details:
WatchRuleType: Log
WatchRule: (SEVERITY = 'Error') AND ((MSGID = 'WL-000337') OR (MSGID = 'BEA-000337'))
WatchData: DATE = Jun 28, 2011 1:27:22 PM BST SERVER = soa_server1 MESSAGE = *[STUCK] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "638" seconds working on the request "weblogic.servlet.internal.ServletRequestImpl@16e92176[*
*POST /soa-infra/services/default/BAMInsert/bpelprocess1_client_ep HTTP/1.1*
Connection: TE
TE: trailers, deflate, gzip, compress
User-Agent: Oracle HTTPClient Version 10h
SOAPAction: "process"
Accept-Encoding: gzip, x-gzip, compress, x-compress
ECID-Context: 1.c82d73b0a2a7776f:2ced6156:130d1186ef6:-8000-00000000000046b2;kYjE0ZJOoOTLkKPOoLRKlSODoITT_G
Content-type: text/xml; charset=UTF-8
Content-Length: 270
*]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:*
*Thread-139 "[STUCK] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, suspended, sleeping, priority=1, DAEMON> {*
*java.lang.Thread.sleep(Thread.java:???)*
oracle.bam.common.remoting.BamEjbClient.getSession(BamEjbClient.java:973)
oracle.bam.common.remoting.BamEjbClient.getADCSession(BamEjbClient.java:350)
oracle.bam.adc.api.util.Context.<init>(Context.java:270)
oracle.bam.adapter.adc.CachedConnection.<init>(CachedConnection.java:134)
oracle.bam.adapter.adc.ADCManagedConnectionFactory.getCachedConnection(ADCManagedConnectionFactory.java:490)
^-- Holding lock: java.lang.String@13416588[fat lock]
What could be root cause of this? Can anyone please help to find the solution?
Even though both servers on same host why BAM connection is getting to much time and Putting server in warning state like :
+[STUCK] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'+
*Do i need to modify any config file at Admin or BAM level? if Yes, what parameter Do i need to change?*
Please guide.
Thanks,
Sagar.Further Updates on this :
I further debugged and found out few more things. This might help.
In Weblogic Admin console Home > Summary of Deployments >oracle-bam(11.1.1)
MDB application oracle-bam is NOT connected to messaging system with below error.
EJBs
MessageDispatcherBean - Error java.lang.IllegalArgumentException: Getting Deployment configuration...
connectionFactoryJNDIName - Red Cross with warning
destinationJNDIName - Red Cross with warning
resourceAdapterJNDIName - Red Cross with warning
Modules
Also please check out another similar thread and please update.
BAMAdapter Issue : java.rmi.ConnectException: Destination unreachable;
Thanks,
Sagar -
Oracle Service Bus "ThreadPool has stuck threads"
Hi all,
I have JMS Proxy service that listen to the queue, with small quantity of message every second, it works great.
But after put message with bigger quantity and faster insertion on queue, the proxy service cannot
pull the message anymore, the error message is "STUCK THREAD", it makes the weblogic console give warning status.
This is the Dump Thread Stacks :
===== FULL THREAD DUMP ===============
Tue Jun 8 11:30:24 2010
BEA JRockit(R) R27.6.1-20-104573-1.6.0_05-20081016-1236-linux-ia32
"Main Thread" id=1 idx=0x4 tid=7168 prio=5 alive, in native, waiting
-- Waiting for notification on: weblogic/t3/srvr/T3Srvr@0x1cd20688[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at weblogic/t3/srvr/T3Srvr.waitForDeath(T3Srvr.java:811)
^-- Lock released while waiting: weblogic/t3/srvr/T3Srvr@0x1cd20688[fat lock]
at weblogic/t3/srvr/T3Srvr.run(T3Srvr.java:459)
at weblogic/Server.main(Server.java:67)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"(Signal Handler)" id=2 idx=0x8 tid=7169 prio=5 alive, in native, daemon
"(GC Main Thread)" id=3 idx=0xc tid=7170 prio=5 alive, in native, native_waiting, daemon
"(GC Worker Thread 1)" id=? idx=0x10 tid=7171 prio=5 alive, in native, daemon
"(GC Worker Thread 2)" id=? idx=0x14 tid=7172 prio=5 alive, in native, daemon
"(GC Worker Thread 3)" id=? idx=0x18 tid=7173 prio=5 alive, in native, daemon
"(GC Worker Thread 4)" id=? idx=0x1c tid=7174 prio=5 alive, in native, daemon
"(GC Worker Thread 5)" id=? idx=0x20 tid=7175 prio=5 alive, in native, daemon
"(GC Worker Thread 6)" id=? idx=0x24 tid=7176 prio=5 alive, in native, daemon
"(GC Worker Thread 7)" id=? idx=0x28 tid=7177 prio=5 alive, in native, daemon
"(GC Worker Thread 8)" id=? idx=0x2c tid=7178 prio=5 alive, in native, daemon
"(Code Generation Thread 1)" id=4 idx=0x30 tid=7179 prio=5 alive, in native, native_waiting, daemon
"(Code Optimization Thread 1)" id=5 idx=0x34 tid=7180 prio=5 alive, in native, native_waiting, daemon
"(VM Periodic Task)" id=6 idx=0x38 tid=7181 prio=10 alive, in native, daemon
Is there any tuning that I need to do on Jrockit?
Really thanks in advance.
Regards,
Mr.KHi Eric,
Thank you for the link, I'll try for increasing it.
Hi Manoj,
hope it'll give you some additional information for helping me.
####<Jun 8, 2010 10:43:44 AM WIT> <Error> <WebLogicServer> <cspproduction> <AdminServerOSB> <[ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1275968624958> <BEA-000337> <[STUCK] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "653" seconds working on the request "weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl@39c79ac", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-50 "[STUCK] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, waiting, priority=1, DAEMON> {
-- Waiting for notification on: weblogic.rjvm.ResponseImpl@39cf5b1[fat lock]
java.lang.Object.wait(Object.java:???)
weblogic.rjvm.ResponseImpl.waitForData(ResponseImpl.java:80)
weblogic.rjvm.ResponseImpl.getTxContext(ResponseImpl.java:127)
weblogic.messaging.dispatcher.DispatcherProxy.unmarshalResponse(DispatcherProxy.java:236)
weblogic.messaging.dispatcher.DispatcherProxy.dispatchSyncTranFuture(DispatcherProxy.java:133)
weblogic.messaging.dispatcher.DispatcherWrapperState.dispatchSyncTran(DispatcherWrapperState.java:333)
weblogic.messaging.dispatcher.DispatcherWrapperState.dispatchSyncNoTran(DispatcherWrapperState.java:384)
weblogic.jms.dispatcher.DispatcherAdapter.dispatchSyncNoTran(DispatcherAdapter.java:59)
weblogic.jms.client.JMSProducer.toFEProducer(JMSProducer.java:1151)
weblogic.jms.client.JMSProducer.deliveryInternal(JMSProducer.java:679)
weblogic.jms.client.JMSProducer.sendInternal(JMSProducer.java:541)
weblogic.jms.client.JMSProducer.sendWithListener(JMSProducer.java:366)
weblogic.jms.client.JMSProducer.send(JMSProducer.java:353)
weblogic.jms.client.WLProducerImpl.send(WLProducerImpl.java:1050)
weblogic.jms.client.WLProducerImpl.send(WLProducerImpl.java:107)
weblogic.deployment.jms.WrappedMessageProducer.send(WrappedMessageProducer.java:235)
com.bea.wli.sb.transports.jms.JmsOutboundMessageContext$SendAction.sendMessage(JmsOutboundMessageContext.java:823)
com.bea.wli.sb.transports.jms.JmsOutboundMessageContext$SendAction.run(JmsOutboundMessageContext.java:745)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:350)
weblogic.security.service.SecurityManager.runAs(Unknown Source)
weblogic.security.Security.runAs(Security.java:60)
com.bea.wli.sb.transports.jms.JmsOutboundMessageContext.send(JmsOutboundMessageContext.java:502)
com.bea.wli.sb.transports.jms.JmsTransportProvider.sendMessageAsync(JmsTransportProvider.java:680)
sun.reflect.GeneratedMethodAccessor302.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
com.bea.wli.sb.transports.Util$1.invoke(Util.java:56)
$Proxy61.sendMessageAsync(Unknown Source)
com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageAsync(LoadBalanceFailoverListener.java:136)
com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageToServiceAsync(LoadBalanceFailoverListener.java:510)
com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageToService(LoadBalanceFailoverListener.java:477)
com.bea.wli.sb.transports.TransportManagerImpl.sendMessageToService(TransportManagerImpl.java:491)
com.bea.wli.sb.transports.TransportManagerImpl.sendMessageAsync(TransportManagerImpl.java:405)
com.bea.wli.sb.pipeline.PipelineContextImpl.doDispatch(PipelineContextImpl.java:508)
com.bea.wli.sb.pipeline.PipelineContextImpl.dispatch(PipelineContextImpl.java:486)
stages.publish.runtime.PublishRuntimeStep.processMessage(PublishRuntimeStep.java:83)
com.bea.wli.sb.stages.StageMetadataImpl$WrapperRuntimeStep.processMessage(StageMetadataImpl.java:343)
com.bea.wli.sb.stages.impl.SequenceRuntimeStep.processMessage(SequenceRuntimeStep.java:31)
com.bea.wli.sb.pipeline.PipelineStage.processMessage(PipelineStage.java:84)
com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:840)
com.bea.wli.sb.pipeline.Pipeline.processMessage(Pipeline.java:140)
com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:840)
com.bea.wli.sb.pipeline.PipelineNode.doRequest(PipelineNode.java:51)
com.bea.wli.sb.pipeline.Node.processMessage(Node.java:65)
com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:840)
com.bea.wli.sb.pipeline.Router.processMessage(Router.java:178)
com.bea.wli.sb.pipeline.MessageProcessor.processRequest(MessageProcessor.java:39)
com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:508)
com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:506)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:350)
weblogic.security.service.SecurityManager.runAs(Unknown Source)
com.bea.wli.sb.security.WLSSecurityContextService.runAs(WLSSecurityContextService.java:55)
com.bea.wli.sb.pipeline.RouterManager.processMessage(RouterManager.java:487)
com.bea.wli.sb.transports.TransportManagerImpl.receiveMessage(TransportManagerImpl.java:248)
com.bea.wli.sb.transports.jms.JmsInboundMDB.onMessage(JmsInboundMDB.java:79)
sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:???)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:27)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:277)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:187)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:153)
com.bea.core.repackaged.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:86)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:153)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:126)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:100)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:153)
com.bea.core.repackaged.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:148)
$Proxy79.onMessage(Unknown Source)
weblogic.ejb.container.internal.MDListener.execute(MDListener.java:403)
weblogic.ejb.container.internal.MDListener.transactionalOnMessage(MDListener.java:352)
weblogic.ejb.container.internal.MDListener.onMessage(MDListener.java:264)
weblogic.jms.client.JMSSession.onMessage(JMSSession.java:4526)
weblogic.jms.client.JMSSession.execute(JMSSession.java:4022)
weblogic.jms.client.JMSSession.executeMessage(JMSSession.java:3708)
weblogic.jms.client.JMSSession.access$000(JMSSession.java:114)
weblogic.jms.client.JMSSession$UseForRunnable.run(JMSSession.java:5058)
weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:516)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:198)
weblogic.work.ExecuteThread.run(ExecuteThread.java:165)
That is error message from Oracle service bus console, it stated there is JMS Proxy service that got stuck.
On previous post, I already attached the dump trace, it explained something about jrockit. But I still don't get the way for tuning it properly.
Regards,
Mr.K -
Stuck Thread - JMS Business proxy Service (ALSB 2.5)
Hi All,
We are having a jms proxy service which uses a JMS Business Service to send messages to a jms queue on weblogic 8.1 sp6. The alsb domain runs on weblogic 9.2.The jms proxy service is made single threaded by configuring a suitable work manager.
Now and then, we get a stuck thread issue in production and the proxy service stops consuming the messaging off the jms queue in wls 9.2. These are the errors in the logs:
<[ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1235437298516> <BEA-000337> <[STUCK] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "695" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@5321caf", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-70 "[STUCK] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, waiting, priority=1, DAEMON> {
-- Waiting for notification on: weblogic.rjvm.ResponseImpl@51961a5[fat lock]
jrockit.vm.Threads.waitForSignal(Native Method)
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Native Method)
weblogic.rjvm.ResponseImpl.waitForData(ResponseImpl.java:73)
^-- Lock released while waiting: weblogic.rjvm.ResponseImpl@51961a5[fat lock]
weblogic.rjvm.ResponseImpl.getTxContext(ResponseImpl.java:100)
weblogic.rjvm.BasicOutboundRequest.sendReceive(BasicOutboundRequest.java:109)
weblogic.rmi.internal.BasicRemoteRef.invoke(BasicRemoteRef.java:223)
weblogic.jms.dispatcher.DispatcherImpl_816_WLStub.dispatchSyncTranFuture(Unknown Source)
weblogic.jms.dispatcher.DispatcherInteropAdapter.dispatchSyncTranFuture(DispatcherInteropAdapter.java:80)
weblogic.messaging.dispatcher.DispatcherWrapperState.dispatchSyncTran(DispatcherWrapperState.java:230)
weblogic.jms.dispatcher.DispatcherAdapter.dispatchSyncTran(DispatcherAdapter.java:51)
weblogic.jms.client.JMSProducer.toFEProducer(JMSProducer.java:1047)
weblogic.jms.client.JMSProducer.deliveryInternal(JMSProducer.java:670)
weblogic.jms.client.JMSProducer.sendInternal(JMSProducer.java:455)
weblogic.jms.client.JMSProducer.send(JMSProducer.java:352)
weblogic.jms.client.WLProducerImpl.send(WLProducerImpl.java:1009)
weblogic.deployment.jms.WrappedMessageProducer.send(WrappedMessageProducer.java:157)
com.bea.wli.sb.transports.jms.JmsOutboundMessageContext$SendAction.run(JmsOutboundMessageContext.java:795)
com.bea.wli.sb.transports.jms.JmsOutboundMessageContext.send(JmsOutboundMessageContext.java:500)
com.bea.wli.sb.transports.jms.JmsTransportProvider.sendMessageAsync(JmsTransportProvider.java:612)
jrockit.reflect.CompiledMethodInvoker.invoke0(Unknown Source)
jrockit.reflect.CompiledMethodInvoker.invoke(Unknown Source)
java.lang.reflect.Method.invoke(Unknown Source)
com.bea.wli.sb.transports.Util$1.invoke(Util.java:241)
$Proxy39.sendMessageAsync(Unknown Source)
com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageAsync(LoadBalanceFailoverListener.java:137)
com.bea.wli.sb.transports.TransportManagerImpl.sendMessageToService(TransportManagerImpl.java:437)
com.bea.wli.sb.transports.TransportManagerImpl.sendMessageAsync(TransportManagerImpl.java:328)
com.bea.wli.sb.pipeline.PipelineContextImpl.doDispatch(PipelineContextImpl.java:503)
com.bea.wli.sb.pipeline.PipelineContextImpl.dispatch(PipelineContextImpl.java:415)
stages.publish.runtime.PublishRuntimeStep.processMessage(PublishRuntimeStep.java:106)
com.bea.wli.sb.stages.StageMetadataImpl$WrapperRuntimeStep.processMessage(StageMetadataImpl.java:327)
com.bea.wli.sb.stages.impl.SequenceRuntimeStep.processMessage(SequenceRuntimeStep.java:33)
com.bea.wli.sb.pipeline.PipelineStage.processMessage(PipelineStage.java:100)
com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:771)
com.bea.wli.sb.pipeline.Pipeline.processMessage(Pipeline.java:73)
com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:771)
com.bea.wli.sb.pipeline.PipelineNode.doRequest(PipelineNode.java:47)
com.bea.wli.sb.pipeline.Node.processMessage(Node.java:57)
com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:771)
com.bea.wli.sb.pipeline.Router.processMessage(Router.java:141)
com.bea.wli.sb.pipeline.MessageProcessor.processRequest(MessageProcessor.java:75)
com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:646)
com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:645)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:147)
com.bea.wli.sb.pipeline.RouterManager.processMessage(RouterManager.java:627)
com.bea.wli.sb.transports.TransportManagerImpl.receiveMessage(TransportManagerImpl.java:264)
com.bea.wli.sb.transports.jms.JmsInboundMDB.onMessage(JmsInboundMDB.java:108)
weblogic.ejb.container.internal.MDListener.execute(MDListener.java:429)
weblogic.ejb.container.internal.MDListener.transactionalOnMessage(MDListener.java:335)
weblogic.ejb.container.internal.MDListener.onMessage(MDListener.java:292)
weblogic.jms.client.JMSSession.onMessage(JMSSession.java:4060)
weblogic.jms.client.JMSSession.execute(JMSSession.java:4012)
weblogic.jms.client.JMSSession$UseForRunnable.run(JMSSession.java:4467)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
jrockit.vm.RNI.c2java(Native Method)
Any one know the root cause of this issue or ways to prevent this?
Also Is there a way to make the proxy service to reconsume messages off the queue without a server restart?
Thanks,
AtheekHi atheek, did you find the solution??
I already got the same/identic issue, do you mind giving me an aid?
Really thanks
Regards,
Mr.K -
STUCK Threads.... Web Logic goes to overloaded state
Hi,
We are facing production problem with struck threads and WL server is going to overloaded state very frequently. When this happens, we have to restart the server to void this issue. Since we dont know when this problem occurs, our support team has to keep monitor the server all the time.
The environment details are:
JRocket: jrockit160_R28_0_1
WL: WebLogic 10.3.0.0
OS: Red Hat Enterprise Linux AS release 4 (Nahant Update 8)
We are extensively thread using the thread pool in application. These threads are being used to write the logs into DB, file system, monitor the timeout of external connections, etc. The issue has been seen various places like log4j, httpClient, etc. Here am sending the thread dump of an issue.
If you could help what is the an issue, it would be great help.
Thanks for your help in advance
Thanks
Sree Hari
Log 1:
"[STUCK] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'" RUNNABLE native
jrockit.net.SocketNativeIO.readBytesPinned(Native Method)
jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:32)
java.net.SocketInputStream.socketRead0(SocketInputStream.java)
java.net.SocketInputStream.read(SocketInputStream.java:129)
com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:789)
com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:746)
com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read(BufferedInputStream.java:237)
org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78)
org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106)
org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116)
org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973)
org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)
org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
com.bestbuy.bbym.beast.cap.core.http.HttpSender.sendRequestAndFetchResponse(HttpSender.java:67)
com.bestbuy.bbym.beast.cap.core.http.HttpAdapter.processRequest(HttpAdapter.java:164)
com.bestbuy.bbym.beast.cap.core.ProxyHelper.getResponse(ProxyHelper.java:324)
com.bestbuy.bbym.beast.cap.core.multirequest.PostRequest.execute(PostRequest.java:237)
com.bestbuy.bbym.beast.cap.util.MultiRequestUtil.executeNextCommand(MultiRequestUtil.java:185)
com.bestbuy.bbym.beast.cap.core.multirequest.MultiRequestProxy.invoke(MultiRequestProxy.java:87)
com.bestbuy.bbym.beast.cap.core.RequestProcessor.process(RequestProcessor.java:68)
com.bestbuy.bbym.beast.cap.core.controller.Controller.handleRequest(Controller.java:737)
com.bestbuy.bbym.beast.cap.core.controller.SyncServiceHandlingController.handleSyncRequest(SyncServiceHandlingController.java:28)
com.bestbuy.bbym.beast.cap.core.router.RequestRouter.routeRequest(RequestRouter.java:71)
com.bestbuy.bbym.beast.cap.core.controller.CAPWsController.processRequest(CAPWsController.java:68)
sun.reflect.GeneratedMethodAccessor364.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.apache.axis.providers.java.RPCProvider.invokeMethod(RPCProvider.java:397)
org.apache.axis.providers.java.RPCProvider.processMessage(RPCProvider.java:186)
org.apache.axis.providers.java.JavaProvider.invoke(JavaProvider.java:323)
org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
org.apache.axis.handlers.soap.SOAPService.invoke(SOAPService.java:454)
org.apache.axis.server.AxisServer.invoke(AxisServer.java:281)
org.apache.axis.transport.http.AxisServlet.doPost(AxisServlet.java:699)
javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
org.apache.axis.transport.http.AxisServletBase.service(AxisServletBase.java:327)
javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:292)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:175)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3495)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
weblogic.security.service.SecurityManager.runAs(Unknown Source)
weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2180)
weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2086)
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1406)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
Log 2:
"[STUCK] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" id=15 idx=0x54 tid=20089 prio=1 alive, blocked, native_blocked, daemon
-- Blocked trying to get lock: org/apache/log4j/Logger@0xc501e0a8[fat lock]
at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)
at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1402)[optimized]
at jrockit/vm/Locks.lockFat(Locks.java:1503)[optimized]
at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1045)[optimized]
at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:996)[optimized]
at org/apache/log4j/Category.callAppenders(Category.java:204)[optimized]
at org/apache/log4j/Category.forcedLog(Category.java:391)[inlined]
at org/apache/log4j/Category.info(Category.java:666)[optimized]
at com/bestbuy/bbym/beast/cap/util/logging/LoggingHelper.logDBInfo(LoggingHelper.java:228)[inlined]
at com/bestbuy/bbym/beast/cap/core/controller/Controller.handleRequest(Controller.java:838)[optimized]
at com/bestbuy/bbym/beast/cap/core/controller/SyncServiceHandlingController.handleSyncRequest(SyncServiceHandlingController.java:28)[inlined]
at com/bestbuy/bbym/beast/cap/core/router/RequestRouter.routeRequest(RequestRouter.java:71)[optimized]
at com/bestbuy/bbym/beast/cap/core/controller/CAPWsController.processRequest(CAPWsController.java:68)
at sun/reflect/GeneratedMethodAccessor120.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source)
at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)[optimized]
at java/lang/reflect/Method.invoke(Method.java:597)[optimized]
at org/apache/axis/providers/java/RPCProvider.invokeMethod(RPCProvider.java:397)
at org/apache/axis/providers/java/RPCProvider.processMessage(RPCProvider.java:186)
at org/apache/axis/providers/java/JavaProvider.invoke(JavaProvider.java:323)
at org/apache/axis/strategies/InvocationStrategy.visit(InvocationStrategy.java:32)
at org/apache/axis/SimpleChain.doVisiting(SimpleChain.java:118)
at org/apache/axis/SimpleChain.invoke(SimpleChain.java:83)
at org/apache/axis/handlers/soap/SOAPService.invoke(SOAPService.java:454)
at org/apache/axis/server/AxisServer.invoke(AxisServer.java:281)[optimized]
at org/apache/axis/transport/http/AxisServlet.doPost(AxisServlet.java:699)[optimized]
at javax/servlet/http/HttpServlet.service(HttpServlet.java:727)[optimized]
at org/apache/axis/transport/http/AxisServletBase.service(AxisServletBase.java:327)
at javax/servlet/http/HttpServlet.service(HttpServlet.java:820)
at weblogic/servlet/internal/StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
at weblogic/servlet/internal/StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)[inlined]
at weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:292)[optimized]
at weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:175)[inlined]
at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3495)[optimized]
at weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)[inlined]
at weblogic/security/service/SecurityManager.runAs(Lweblogic/security/acl/internal/AuthenticatedSubject;Lweblogic/security/acl/internal/AuthenticatedSubject;Ljava/security/PrivilegedAction;)Ljava/lang/Object;(Unknown Source)[inlined]
at weblogic/servlet/internal/WebAppServletContext.securedExecute(WebAppServletContext.java:2180)[optimized]
at weblogic/servlet/internal/WebAppServletContext.execute(WebAppServletContext.java:2086)
at weblogic/servlet/internal/ServletRequestImpl.run(ServletRequestImpl.java:1406)
at weblogic/work/ExecuteThread.execute(ExecuteThread.java:201)[optimized]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:173)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"weblogic.time.TimeEventGenerator" id=16 idx=0x58 tid=20090 prio=9 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/time/common/internal/TimeTable@0xa5889270[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at weblogic/time/common/internal/TimeTable.snooze(TimeTable.java:286)
^-- Lock released while waiting: weblogic/time/common/internal/TimeTable@0xa5889270[fat lock]
at weblogic/time/common/internal/TimeEventGenerator.run(TimeEventGenerator.java:117)
at java/lang/Thread.run(Thread.java:619)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"weblogic.timers.TimerThread" id=17 idx=0x5c tid=20091 prio=9 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/timers/internal/TimerThread@0xa5887ae0[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at weblogic/timers/internal/TimerThread$Thread.run(TimerThread.java:267)
^-- Lock released while waiting: weblogic/timers/internal/TimerThread@0xa5887ae0[fat lock]
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"[STUCK] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" id=18 idx=0x60 tid=20092 prio=1 alive, blocked, native_blocked, daemon
-- Blocked trying to get lock: org/apache/log4j/Logger@0xc501e0a8[fat lock]
at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)
at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1402)[optimized]
at jrockit/vm/Locks.lockFat(Locks.java:1503)[optimized]
at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1045)[optimized]
at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:996)[optimized]
at org/apache/log4j/Category.callAppenders(Category.java:204)[optimized]
at org/apache/log4j/Category.forcedLog(Category.java:391)[inlined]
at org/apache/log4j/Category.info(Category.java:666)[optimized]
at com/bestbuy/bbym/beast/cap/util/logging/LoggingHelper.logDBInfo(LoggingHelper.java:228)[inlined]
at com/bestbuy/bbym/beast/cap/core/controller/Controller.handleRequest(Controller.java:838)[optimized]
at com/bestbuy/bbym/beast/cap/core/controller/SyncServiceHandlingController.handleSyncRequest(SyncServiceHandlingController.java:28)[inlined]
at com/bestbuy/bbym/beast/cap/core/router/RequestRouter.routeRequest(RequestRouter.java:71)[optimized]
at com/bestbuy/bbym/beast/cap/core/controller/CAPWsController.processRequest(CAPWsController.java:68)
at sun/reflect/GeneratedMethodAccessor120.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source)
at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)[optimized]
at java/lang/reflect/Method.invoke(Method.java:597)[optimized]
at org/apache/axis/providers/java/RPCProvider.invokeMethod(RPCProvider.java:397)
at org/apache/axis/providers/java/RPCProvider.processMessage(RPCProvider.java:186)
at org/apache/axis/providers/java/JavaProvider.invoke(JavaProvider.java:323)
at org/apache/axis/strategies/InvocationStrategy.visit(InvocationStrategy.java:32)
at org/apache/axis/SimpleChain.doVisiting(SimpleChain.java:118)
at org/apache/axis/SimpleChain.invoke(SimpleChain.java:83)
at org/apache/axis/handlers/soap/SOAPService.invoke(SOAPService.java:454)
at org/apache/axis/server/AxisServer.invoke(AxisServer.java:281)[optimized]
at org/apache/axis/transport/http/AxisServlet.doPost(AxisServlet.java:699)[optimized]
at javax/servlet/http/HttpServlet.service(HttpServlet.java:727)[optimized]
at org/apache/axis/transport/http/AxisServletBase.service(AxisServletBase.java:327)
at javax/servlet/http/HttpServlet.service(HttpServlet.java:820)
at weblogic/servlet/internal/StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
at weblogic/servlet/internal/StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)[inlined]
at weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:292)[optimized]
at weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:175)[inlined]
at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3495)[optimized]
at weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)[inlined]
at weblogic/security/service/SecurityManager.runAs(Lweblogic/security/acl/internal/AuthenticatedSubject;Lweblogic/security/acl/internal/AuthenticatedSubject;Ljava/security/PrivilegedAction;)Ljava/lang/Object;(Unknown Source)[inlined]
at weblogic/servlet/internal/WebAppServletContext.securedExecute(WebAppServletContext.java:2180)[optimized]
at weblogic/servlet/internal/WebAppServletContext.execute(WebAppServletContext.java:2086)
at weblogic/servlet/internal/ServletRequestImpl.run(ServletRequestImpl.java:1406)
at weblogic/work/ExecuteThread.execute(ExecuteThread.java:201)[optimized]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:173)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of traceThis is your own logging environment?
Well this is what you get when everything is synchronized.
You have to debug the logging in your environment, for example, by using JRockit Mission Control in particular a flight recording
Example of using the flight recording tool can be found here: middlewaremagic.com/weblogic/?p=6930 (Good luck!) -
Stuck thread in adf application with weblogic
hi every body
I am working in JDev 11.1.1.4 and weblogic 10.3.4
when I deploy my Application in my production weblogic
after some time (it does not constant it is variable) I got warnning on the server
when I check the server I got some stuck thread
any idea or notes how can I know what is the reason of these stuck threads?
is there any objects have thread unsafe issue, resource contention or race condition in ADF?
BR,
Alaaactully I do that guys
but what I get does not have any related to my application
even those http request does not request the same page
this is the stuck thread when I dump
"[STUCK] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" id=61 idx=0xe8 tid=13369 prio=1 alive, waiting, native_blocked, daemon
-- Waiting for notification on: oracle/adfinternal/controller/util/RequestLock@0x88e06b58[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
at java/lang/Object.wait(Object.java:485)[inlined]
at oracle/adfinternal/controller/util/RequestLock.lock(RequestLock.java:42)[inlined]
at oracle/adfinternal/controller/state/RootViewPortContextImpl.lockViewPortRequestLock(RootViewPortContextImpl.java:604)[optimized]
^-- Lock released while waiting: oracle/adfinternal/controller/util/RequestLock@0x88e06b58[fat lock]
at oracle/adfinternal/controller/state/ControllerState.initializeRequest(ControllerState.java:833)[inlined]
at oracle/adfinternal/controller/state/ControllerState.initializeRequest(ControllerState.java:754)[inlined]
at oracle/adfinternal/controller/application/AdfcConfigurator.beginRequest(AdfcConfigurator.java:50)[optimized]
at org/apache/myfaces/trinidadinternal/config/GlobalConfiguratorImpl._startConfiguratorServiceRequest(GlobalConfiguratorImpl.java:562)[inlined]
at org/apache/myfaces/trinidadinternal/config/GlobalConfiguratorImpl.beginRequest(GlobalConfiguratorImpl.java:212)[optimized]
at org/apache/myfaces/trinidadinternal/webapp/TrinidadFilterImpl.doFilter(TrinidadFilterImpl.java:155)[optimized]
at org/apache/myfaces/trinidad/webapp/TrinidadFilter.doFilter(TrinidadFilter.java:92)[optimized]
at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[optimized]
at oracle/adf/library/webapp/LibraryFilter.doFilter(LibraryFilter.java:175)[optimized]
at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[optimized]
at oracle/security/jps/ee/http/JpsAbsFilter$1.run(JpsAbsFilter.java:111)[optimized]
at jrockit/vm/AccessController.doPrivileged(AccessController.java:254)[inlined]
at oracle/security/jps/util/JpsSubject.doAsPrivileged(JpsSubject.java:313)[inlined]
at oracle/security/jps/ee/util/JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:413)[inlined]
at oracle/security/jps/ee/http/JpsAbsFilter.runJaasMode(JpsAbsFilter.java:94)[inlined]
at oracle/security/jps/ee/http/JpsAbsFilter.doFilter(JpsAbsFilter.java:161)[optimized]
at oracle/security/jps/ee/http/JpsFilter.doFilter(JpsFilter.java:71)[optimized]
at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[optimized]
at oracle/dms/servlet/DMSServletFilter.doFilter(DMSServletFilter.java:136)[optimized]
at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[optimized]
at weblogic/servlet/internal/RequestEventsFilter.doFilter(RequestEventsFilter.java:27)[optimized]
at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[inlined]
at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3715)[inlined]
at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3681)[optimized]
at weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)[optimized]
at weblogic/security/service/SecurityManager.runAs(SecurityManager.java:120)[inlined]
at weblogic/servlet/internal/WebAppServletContext.securedExecute(WebAppServletContext.java:2277)[inlined]
at weblogic/servlet/internal/WebAppServletContext.execute(WebAppServletContext.java:2183)[optimized]
at weblogic/servlet/internal/ServletRequestImpl.run(ServletRequestImpl.java:1454)[optimized]
at weblogic/work/ExecuteThread.execute(ExecuteThread.java:207)[optimized]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:176)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
any another ideas?
BR,
Alaa -
WebLogic 11g - Stuck Thread issue observed during MS-ADS authentication
Hi
We have weblogic server 10.3.5 installed on RHEL 5.6. For our application authentication, we are getting user credentials authenticated for application login page using Microsoft Active Directory Service (MS-ADS) configured under WebLogic Security Realm. We are getting the below error message now due to STUCK Thread observed at JVM level.
Could you please help me to fix this issue and my thanks in advance.
]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-6424 "[STUCK] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, suspended, waiting, priority=1, DAEMON> {
-- Waiting for notification on: netscape.ldap.LDAPResponseListener@3a8ecc70[fat lock]
java.lang.Object.wait(Object.java:???)
java.lang.Object.wait(Object.java:485)
netscape.ldap.LDAPMessageQueue.waitForMessage(LDAPMessageQueue.java:198)
netscape.ldap.LDAPMessageQueue.waitFirstMessage(LDAPMessageQueue.java:100)
netscape.ldap.LDAPConnection.sendRequest(LDAPConnection.java:1782)
^-- Holding lock: netscape.ldap.LDAPConnection@3a8ecc83[thin lock]
netscape.ldap.LDAPConnection.simpleBind(LDAPConnection.java:1752)
netscape.ldap.LDAPConnection.authenticate(LDAPConnection.java:1260)
netscape.ldap.LDAPConnection.authenticate(LDAPConnection.java:1273)
netscape.ldap.LDAPConnection.authenticate(LDAPConnection.java:1240)
netscape.ldap.LDAPConnection.referralRebind(LDAPConnection.java:4955)
netscape.ldap.LDAPConnection.performReferrals(LDAPConnection.java:5065)
netscape.ldap.LDAPConnection.checkSearchMsg(LDAPConnection.java:2632)
netscape.ldap.LDAPConnection.search(LDAPConnection.java:2500)
weblogic.security.providers.authentication.LDAPAtnDelegate.getDNForUser(LDAPAtnDelegate.java:3667)
weblogic.security.providers.authentication.LDAPAtnDelegate.authenticate(LDAPAtnDelegate.java:3513)
weblogic.security.providers.authentication.LDAPAtnLoginModuleImpl.login(LDAPAtnLoginModuleImpl.java:145)
com.bea.common.security.internal.service.LoginModuleWrapper$1.run(LoginModuleWrapper.java:110)
com.bea.common.security.internal.service.LoginModuleWrapper.login(LoginModuleWrapper.java:101)
sun.reflect.GeneratedMethodAccessor2808.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
javax.security.auth.login.LoginContext.invoke(LoginContext.java:720)
javax.security.auth.login.LoginContext.access$000(LoginContext.java:186)
javax.security.auth.login.LoginContext$4.run(LoginContext.java:683)
javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
javax.security.auth.login.LoginContext.login(LoginContext.java:566)
com.bea.common.security.internal.service.JAASLoginServiceImpl.login(JAASLoginServiceImpl.java:77)
sun.reflect.GeneratedMethodAccessor366.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
com.bea.common.security.internal.utils.Delegator$ProxyInvocationHandler.invoke(Delegator.java:57)
$Proxy28.login(Unknown Source)
weblogic.security.service.internal.WLSJAASLoginServiceImpl$ServiceImpl.login(WLSJAASLoginServiceImpl.java:81)
com.bea.common.security.internal.service.JAASAuthenticationServiceImpl.authenticate(JAASAuthenticationServiceImpl.java:74)
sun.reflect.GeneratedMethodAccessor364.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
com.bea.common.security.internal.utils.Delegator$ProxyInvocationHandler.invoke(Delegator.java:57)
$Proxy46.authenticate(Unknown Source)
weblogic.security.service.WLSJAASAuthenticationServiceWrapper.authenticate(WLSJAASAuthenticationServiceWrapper.java:29)
weblogic.security.service.PrincipalAuthenticator.authenticate(PrincipalAuthenticator.java:339)
weblogic.security.services.Authentication.doLogin(Authentication.java:127)
weblogic.security.services.Authentication.login(Authentication.java:69)
weblogic.security.services.Authentication.login(Authentication.java:51)
in.sis.ps.view.template.user.UserAdapter.doLogin(UserAdapter.java:681)
sun.reflect.GeneratedMethodAccessor2824.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
com.sun.el.parser.AstValue.invoke(Unknown Source)
com.sun.el.MethodExpressionImpl.invoke(Unknown Source)
org.apache.myfaces.trinidad.component.MethodExpressionMethodBinding.invoke(MethodExpressionMethodBinding.java:46)
com.sun.faces.application.ActionListenerImpl.processAction(ActionListenerImpl.java:85)
org.apache.myfaces.trinidad.component.UIXCommand.broadcast(UIXCommand.java:171)
oracle.adf.view.rich.component.fragment.ContextSwitchingComponent$1.run(ContextSwitchingComponent.java:130)
oracle.adf.view.rich.component.fragment.ContextSwitchingComponent._processPhase(ContextSwitchingComponent.java:447)
oracle.adf.view.rich.component.fragment.ContextSwitchingComponent.broadcast(ContextSwitchingComponent.java:122)
oracle.adf.view.rich.component.fragment.UIXInclude.broadcast(UIXInclude.java:106)
oracle.adf.view.rich.component.fragment.ContextSwitchingComponent$1.run(ContextSwitchingComponent.java:130)
oracle.adf.view.rich.component.fragment.ContextSwitchingComponent._processPhase(ContextSwitchingComponent.java:447)
oracle.adf.view.rich.component.fragment.ContextSwitchingComponent.broadcast(ContextSwitchingComponent.java:122)
oracle.adf.view.rich.component.fragment.UIXInclude.broadcast(UIXInclude.java:106)
javax.faces.component.UIViewRoot.broadcastEvents(UIViewRoot.java:720)
javax.faces.component.UIViewRoot.processApplication(UIViewRoot.java:1247)
oracle.adfinternal.view.faces.lifecycle.LifecycleImpl._invokeApplication(LifecycleImpl.java:964)
oracle.adfinternal.view.faces.lifecycle.LifecycleImpl._executePhase(LifecycleImpl.java:259)
oracle.adfinternal.view.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:181)
javax.faces.webapp.FacesServlet.service(FacesServlet.java:284)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:206)
weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.adf.model.servlet.ADFBindingFilter.doFilter(ADFBindingFilter.java:115)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.adfinternal.view.faces.webapp.rich.RegistrationFilter.doFilter(RegistrationFilter.java:65)
org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:459)
oracle.adfinternal.view.faces.activedata.AdsFilter.doFilter(AdsFilter.java:44)
org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:459)
org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl._doFilterImpl(TrinidadFilterImpl.java:253)
org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl.doFilter(TrinidadFilterImpl.java:148)
org.apache.myfaces.trinidad.webapp.TrinidadFilter.doFilter(TrinidadFilter.java:91)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:111)
oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:299)
oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:405)
oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:94)
oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:98)
oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:70)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:86)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:25)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3687)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3681)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:308)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:116)
weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2213)
weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2135)
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1420)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:203)
weblogic.work.ExecuteThread.run(ExecuteThread.java:170)
}Hi,
make sure to keep your Authentication provider in provider specific select group member search as limited where as default would be unlimited.
Once you made this changes then it will solve this issue.
Regards,
Kal -
PAPI client calls - stuck threads with albpm57 engine
PAPI client running on weblogic portal 10.3, connecting to ALBPM 5.7 engine (on weblogic 8.1 cluster).
threads get stuck in WLP servers (where PAPI client runs), and the call is for getActiveInstances from default view. We have around 15000 active instances, and the papi cache size is configured for 25000.
"[STUCK] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" id=17 idx=0x7c tid=6615 prio=1 alive, in native, waiting, daemon
-- Waiting for notification on: fuego/papi/impl/lock/SimpleLock@0x1f443568[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)[optimized]
at java/lang/Object.wait(Object.java:485)
at fuego/papi/impl/lock/SimpleLock.readLock(SimpleLock.java:50)
^-- Lock released while waiting: fuego/papi/impl/lock/SimpleLock@0x1f443568[fat lock]
at fuego/papi/impl/InstanceCache$Entry.readLock(InstanceCache.java:708)
at fuego/papi/impl/InstanceCache.getInstancesByFilter(InstanceCache.java:505)
at fuego/papi/impl/InstanceCache.getInstancesByFilter(InstanceCache.java:231)
at fuego/papi/impl/ProcessServiceImpl.getInstancesByFilter(ProcessServiceImpl.java:1084)
at fuego/papi/impl/ProcessServiceNESessionImpl.getInstancesByFilter(ProcessServiceNESessionImpl.java:754)
Please help how to resolve this.
thanks in advance.
Edited by: 811778 on May 17, 2013 5:05 PMsupport team suggested to use,
-Dfuego.papi.lockManagerClass=fuego.papi.impl.lock.ParanoidLockManager for each java process, acting as PAPI client. This class available in fuegopapi-client.jar
additional comments from support team,
Implementing this property does not generate side-effects. This property detects and kills/recovers deadlocks in the PAPI instance cache -
Blocked trying to get lock: java.lang.Class@[fat lock]
Joe,
Can you please help me understand what is going on with this thread dump? There is no dead lock; but I have stuck thread. Is it waiting an Oracle prepared statement?
===== FULL THREAD DUMP ===============
Wed Sep 4 16:12:27 2013
Oracle JRockit(R) R28.0.1-21-133393-1.6.0_20-20100512-2126-linux-x86_64
"Main Thread" id=1 idx=0x4 tid=26484 prio=5 alive, waiting, native_blocked
-- Waiting for notification on: weblogic/t3/srvr/T3Srvr@0x1398a8228[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at weblogic/t3/srvr/T3Srvr.waitForDeath(T3Srvr.java:902)
^-- Lock released while waiting: weblogic/t3/srvr/T3Srvr@0x1398a8228[fat lock]
at weblogic/t3/srvr/T3Srvr.run(T3Srvr.java:465)
at weblogic/Server.main(Server.java:67)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"(OC Main Thread)" id=2 idx=0x8 tid=26487 prio=5 alive, native_waiting, daemon
"(GC Worker Thread 1)" id=? idx=0xc tid=26488 prio=5 alive, daemon
"(GC Worker Thread 2)" id=? idx=0x10 tid=26489 prio=5 alive, daemon
"(GC Worker Thread 3)" id=? idx=0x14 tid=26490 prio=5 alive, daemon
"(GC Worker Thread 4)" id=? idx=0x18 tid=26491 prio=5 alive, daemon
"(Code Generation Thread 1)" id=3 idx=0x1c tid=26494 prio=5 alive, native_waiting, daemon
"(Code Optimization Thread 1)" id=4 idx=0x20 tid=26495 prio=5 alive, native_waiting, daemon
"(VM Periodic Task)" id=5 idx=0x24 tid=26496 prio=10 alive, native_blocked, daemon
"(Attach Listener)" id=6 idx=0x28 tid=26497 prio=5 alive, native_blocked, daemon
"Finalizer" id=7 idx=0x2c tid=26498 prio=8 alive, native_waiting, daemon
at jrockit/memory/Finalizer.waitForFinalizees(J[Ljava/lang/Object;)I(Native Method)
at jrockit/memory/Finalizer.access$700(Finalizer.java:12)
at jrockit/memory/Finalizer$4.run(Finalizer.java:189)
at java/lang/Thread.run(Thread.java:619)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"Reference Handler" id=8 idx=0x30 tid=26499 prio=10 alive, native_waiting, daemon
at java/lang/ref/Reference.waitForActivatedQueue(J)Ljava/lang/ref/Reference;(Native Method)
at java/lang/ref/Reference.access$100(Reference.java:11)
at java/lang/ref/Reference$ReferenceHandler.run(Reference.java:82)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"(Sensor Event Thread)" id=9 idx=0x34 tid=26500 prio=5 alive, native_blocked, daemon
"VM JFR Buffer Thread" id=10 idx=0x38 tid=26501 prio=5 alive, in native, daemon
"Timer-0" id=13 idx=0x3c tid=26502 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: java/util/TaskQueue@0x1399c29d0[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at java/util/TimerThread.mainLoop(Timer.java:483)
^-- Lock released while waiting: java/util/TaskQueue@0x1399c29d0[fat lock]
at java/util/TimerThread.run(Timer.java:462)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"Timer-1" id=14 idx=0x40 tid=26504 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: java/util/TaskQueue@0x1399c2a58[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/util/TimerThread.mainLoop(Timer.java:509)
^-- Lock released while waiting: java/util/TaskQueue@0x1399c2a58[fat lock]
at java/util/TimerThread.run(Timer.java:462)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" id=15 idx=0x44 tid=26505 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/work/ExecuteThread@0x1399c3848[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at jrockit/vm/Locks.wait(Locks.java:1964)[inlined]
at java/lang/Object.wait(Object.java:485)[inlined]
at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:157)[optimized]
^-- Lock released while waiting: weblogic/work/ExecuteThread@0x1399c3848[fat lock]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:178)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"JFR request timer" id=16 idx=0x48 tid=26506 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: java/util/TaskQueue@0x1399c2ae0[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at java/util/TimerThread.mainLoop(Timer.java:483)
^-- Lock released while waiting: java/util/TaskQueue@0x1399c2ae0[fat lock]
at java/util/TimerThread.run(Timer.java:462)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"weblogic.time.TimeEventGenerator" id=18 idx=0x4c tid=26507 prio=9 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/time/common/internal/TimeTable@0x1399c5070[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at weblogic/time/common/internal/TimeTable.snooze(TimeTable.java:286)
^-- Lock released while waiting: weblogic/time/common/internal/TimeTable@0x1399c5070[fat lock]
at weblogic/time/common/internal/TimeEventGenerator.run(TimeEventGenerator.java:117)
at java/lang/Thread.run(Thread.java:619)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"JMAPI event thread" id=19 idx=0x50 tid=26508 prio=5 alive, in native, daemon
"weblogic.timers.TimerThread" id=20 idx=0x54 tid=26509 prio=9 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/timers/internal/TimerThread@0x1399c2c08[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at weblogic/timers/internal/TimerThread$Thread.run(TimerThread.java:262)
^-- Lock released while waiting: weblogic/timers/internal/TimerThread@0x1399c2c08[fat lock]
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"[STANDBY] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" id=21 idx=0x58 tid=26510 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/work/ExecuteThread@0x1399c3930[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at jrockit/vm/Locks.wait(Locks.java:1964)[inlined]
at java/lang/Object.wait(Object.java:485)[inlined]
at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:157)[optimized]
^-- Lock released while waiting: weblogic/work/ExecuteThread@0x1399c3930[fat lock]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:178)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"NM Reaper" id=24 idx=0x5c tid=26512 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: java/util/TaskQueue@0x1399c5938[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/util/TimerThread.mainLoop(Timer.java:509)
^-- Lock released while waiting: java/util/TaskQueue@0x1399c5938[fat lock]
at java/util/TimerThread.run(Timer.java:462)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" id=25 idx=0x60 tid=26514 prio=5 alive, blocked, native_blocked, daemon
-- Blocked trying to get lock: java/lang/String@0x13898d7f0[fat lock]
at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)
at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1402)[optimized]
at jrockit/vm/Locks.lockFat(Locks.java:1503)[optimized]
at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1045)[optimized]
at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:996)[optimized]
at jrockit/vm/Locks.monitorEnter(Locks.java:2170)[optimized]
at weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:153)
at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)
at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)
at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" id=26 idx=0x64 tid=26515 prio=5 alive, in native, daemon
at jrockit/ext/epoll/EPoll.epollWait0(ILjava/nio/ByteBuffer;II)I(Native Method)
at jrockit/ext/epoll/EPoll.epollWait(EPoll.java:115)
at weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:156)
^-- Holding lock: java/lang/String@0x13898d7f0[fat lock]
at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)
at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)
at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" id=27 idx=0x68 tid=26516 prio=5 alive, blocked, native_blocked, daemon
-- Blocked trying to get lock: java/lang/String@0x13898d7f0[fat lock]
at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)
at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1402)[optimized]
at jrockit/vm/Locks.lockFat(Locks.java:1503)[optimized]
at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1045)[optimized]
at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:996)[optimized]
at jrockit/vm/Locks.monitorEnter(Locks.java:2170)[optimized]
at weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:153)
at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)
at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)
at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '3' for queue: 'weblogic.socket.Muxer'" id=28 idx=0x6c tid=26517 prio=5 alive, blocked, native_blocked, daemon
-- Blocked trying to get lock: java/lang/String@0x13898d7f0[fat lock]
at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)
at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1402)[inlined]
at jrockit/vm/Locks.fatLockReacquire(Locks.java:1640)[optimized]
at jrockit/vm/Locks.lockFat(Locks.java:1508)[optimized]
at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1045)[optimized]
at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:996)[optimized]
at jrockit/vm/Locks.monitorEnter(Locks.java:2170)[optimized]
at weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:153)
at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)
at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)
at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"VDE Transaction Processor Thread" id=31 idx=0x70 tid=26559 prio=2 alive, waiting, native_blocked, daemon
-- Waiting for notification on: com/octetstring/vde/backend/standard/TransactionProcessor@0x1399c3ab8[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at com/octetstring/vde/backend/standard/TransactionProcessor.waitChange(TransactionProcessor.java:367)
^-- Lock released while waiting: com/octetstring/vde/backend/standard/TransactionProcessor@0x1399c3ab8[fat lock]
at com/octetstring/vde/backend/standard/TransactionProcessor.run(TransactionProcessor.java:212)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"UCP-worker-thread-16" id=485 idx=0x74 tid=1649 prio=5 alive, parked, native_blocked, daemon
-- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x106f68908
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2221)[inlined]
at jrockit/proxy/sun/misc/Unsafe.park(Unsafe.java:592)[inlined]
at java/util/concurrent/locks/LockSupport.park(LockSupport.java:158)[inlined]
at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)[optimized]
at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)[optimized]
at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)[optimized]
at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java/lang/Thread.run(Thread.java:619)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"DoSManager" id=34 idx=0x78 tid=26561 prio=6 alive, sleeping, native_waiting, daemon
at java/lang/Thread.sleep(J)V(Native Method)
at com/octetstring/vde/DoSManager.run(DoSManager.java:433)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"Timer-2" id=35 idx=0x7c tid=26562 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: java/util/TaskQueue@0x1399c5df8[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/util/TimerThread.mainLoop(Timer.java:509)
^-- Lock released while waiting: java/util/TaskQueue@0x1399c5df8[fat lock]
at java/util/TimerThread.run(Timer.java:462)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"OracleTimeoutPollingThread" id=36 idx=0x80 tid=26585 prio=10 alive, sleeping, native_waiting, daemon
at java/lang/Thread.sleep(J)V(Native Method)
at oracle/jdbc/driver/OracleTimeoutPollingThread.run(OracleTimeoutPollingThread.java:155)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"Thread-14" id=37 idx=0x84 tid=26586 prio=5 alive, parked, native_blocked, daemon
-- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x1399c6018
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2221)[inlined]
at jrockit/proxy/sun/misc/Unsafe.park(Unsafe.java:592)[inlined]
at java/util/concurrent/locks/LockSupport.park(LockSupport.java:158)[inlined]
at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)[optimized]
at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)[optimized]
at weblogic/utils/concurrent/JDK15ConcurrentBlockingQueue.take(JDK15ConcurrentBlockingQueue.java:89)
at weblogic/store/internal/PersistentStoreImpl.getOutstandingWork(PersistentStoreImpl.java:650)
at weblogic/store/internal/PersistentStoreImpl.run(PersistentStoreImpl.java:707)
at weblogic/store/internal/PersistentStoreImpl$2.run(PersistentStoreImpl.java:464)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"JPS FileChangeNotifier<clean-only>" id=38 idx=0x88 tid=26587 prio=5 alive, parked, native_blocked, daemon
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2221)
at jrockit/proxy/sun/misc/Unsafe.park(Unsafe.java:592)[inlined]
at java/util/concurrent/locks/LockSupport.parkNanos(LockSupport.java:198)[inlined]
at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)[optimized]
at java/util/concurrent/DelayQueue.take(DelayQueue.java:164)[optimized]
at java/util/concurrent/ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)[inlined]
at java/util/concurrent/ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)[optimized]
at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)[optimized]
at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java/lang/Thread.run(Thread.java:619)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"LogFlusher" id=39 idx=0x8c tid=26588 prio=5 alive, sleeping, native_waiting, daemon
at java/lang/Thread.sleep(J)V(Native Method)
at oracle/core/ojdl/BufferedLogWriter$Flusher.run(BufferedLogWriter.java:409)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"LogFlusher" id=40 idx=0x90 tid=26589 prio=5 alive, sleeping, native_waiting, daemon
at java/lang/Thread.sleep(J)V(Native Method)
at oracle/core/ojdl/BufferedLogWriter$Flusher.run(BufferedLogWriter.java:409)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"LogFlusher" id=41 idx=0x94 tid=26590 prio=5 alive, sleeping, native_waiting, daemon
at java/lang/Thread.sleep(J)V(Native Method)
at oracle/core/ojdl/BufferedLogWriter$Flusher.run(BufferedLogWriter.java:409)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"[STANDBY] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'" id=265 idx=0x98 tid=28296 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/work/ExecuteThread@0x106f29288[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at jrockit/vm/Locks.wait(Locks.java:1964)[inlined]
at java/lang/Object.wait(Object.java:485)[inlined]
at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:157)[optimized]
^-- Lock released while waiting: weblogic/work/ExecuteThread@0x106f29288[fat lock]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:178)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"Thread-71" id=266 idx=0x9c tid=28297 prio=10 alive, sleeping, native_waiting, daemon
at java/lang/Thread.sleep(J)V(Native Method)
at oracle/wsm/policyaccess/impl/cache/PolicyCacheImpl$BackTracingSynchronizationThread.run(PolicyCacheImpl.java:759)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"Thread-70" id=264 idx=0xa0 tid=28295 prio=10 alive, parked, native_blocked, daemon
-- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x106f3df50
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2221)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at java/util/concurrent/locks/LockSupport.park(LockSupport.java:158)
at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at oracle/mds/internal/notifications/EventDispatcherThread.run(EventDispatcherThread.java:54)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"oracle.dfw.impl.incident - ADR Executor (created: Fri Aug 30 01:12:40 GMT 2013)" id=45 idx=0xa4 tid=26601 prio=5 alive, parked, native_blocked, daemon
-- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x1399c6698
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2221)[inlined]
at jrockit/proxy/sun/misc/Unsafe.park(Unsafe.java:592)[inlined]
at java/util/concurrent/locks/LockSupport.park(LockSupport.java:158)[inlined]
at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)[optimized]
at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)[optimized]
at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)[optimized]
at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java/lang/Thread.run(Thread.java:619)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"weblogic.work.j2ee.J2EEWorkManager$WorkWithListener@37a58fc8" id=394 idx=0xa8 tid=28767 prio=10 alive, in native, daemon
at jrockit/net/SocketNativeIO.readBytesPinned(Ljava/io/FileDescriptor;[BIII)I(Native Method)
at jrockit/net/SocketNativeIO.socketRead(SocketNativeIO.java:32)[inlined]
at java/net/SocketInputStream.socketRead0(Ljava/io/FileDescriptor;[BIII)I(SocketInputStream.java)[inlined]
at java/net/SocketInputStream.read(SocketInputStream.java:129)[optimized]
at oracle/net/nt/MetricsEnabledInputStream.read(TcpNTAdapter.java:564)[optimized]
at oracle/net/ns/Packet.receive(Packet.java:295)[inlined]
at oracle/net/ns/DataPacket.receive(DataPacket.java:94)[optimized]
at oracle/net/ns/NetInputStream.getNextPacket(NetInputStream.java:176)[optimized]
at oracle/net/ns/NetInputStream.read(NetInputStream.java:121)[inlined]
at oracle/net/ns/NetInputStream.read(NetInputStream.java:96)[inlined]
at oracle/net/ns/NetInputStream.read(NetInputStream.java:81)[optimized]
at oracle/jdbc/driver/T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124)[optimized]
at oracle/jdbc/driver/T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80)[inlined]
at oracle/jdbc/driver/T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1087)[optimized]
at oracle/jdbc/driver/T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1062)[inlined]
at oracle/jdbc/driver/T4C8Oall.receive(T4C8Oall.java:625)[optimized]
at oracle/jdbc/driver/T4CCallableStatement.doOall8(T4CCallableStatement.java:201)[inlined]
at oracle/jdbc/driver/T4CCallableStatement.executeForRows(T4CCallableStatement.java:1036)[optimized]
at oracle/jdbc/driver/OracleStatement.doExecuteWithTimeout(OracleStatement.java:1379)[optimized]
at oracle/jdbc/driver/OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3568)[optimized]
at oracle/jdbc/driver/OraclePreparedStatement.execute(OraclePreparedStatement.java:3739)[optimized]
^-- Holding lock: oracle/jdbc/driver/T4CConnection@0x125a827f8[recursive]
at oracle/jdbc/driver/OracleCallableStatement.execute(OracleCallableStatement.java:8150)[optimized]
^-- Holding lock: oracle/jdbc/driver/T4CConnection@0x125a827f8[thin lock]
at oracle/jdbc/driver/OraclePreparedStatementWrapper.execute(OraclePreparedStatementWrapper.java:1535)[optimized]
at weblogic/jdbc/wrapper/PreparedStatement.execute(PreparedStatement.java:99)[optimized]
at oracle/integration/platform/blocks/event/saq/SAQBusinessEventBus.readEvent(SAQBusinessEventBus.java:1232)[optimized]
at oracle/integration/platform/blocks/event/saq/SAQBusinessEventBus.handleSingleQueueEvent(SAQBusinessEventBus.java:687)[optimized]
at oracle/integration/platform/blocks/event/saq/SAQBusinessEventBus.handleQueueEvents(SAQBusinessEventBus.java:660)
at oracle/integration/platform/blocks/event/saq/SAQBusinessEventBus.access$000(SAQBusinessEventBus.java:71)
at oracle/integration/platform/blocks/event/saq/SAQBusinessEventBus$1.run(SAQBusinessEventBus.java:319)
at oracle/integration/platform/blocks/executor/WorkManagerExecutor$1.run(WorkManagerExecutor.java:105)
at weblogic/work/j2ee/J2EEWorkManager$WorkWithListener.run(J2EEWorkManager.java:183)
at weblogic/work/DaemonWorkThread.run(DaemonWorkThread.java:30)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"oracle.dfw.incident.IncidentCreatorThread" id=47 idx=0xac tid=26603 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: java/util/ArrayList@0x1399c6780[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at oracle/dfw/impl/incident/DiagnosticsDataExtractorImpl$IncidentCreatorThread.run(DiagnosticsDataExtractorImpl.java:1597)
^-- Lock released while waiting: java/util/ArrayList@0x1399c6780[fat lock]
at java/lang/Thread.run(Thread.java:619)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"oracle.dfw.incident.FloodController" id=49 idx=0xb4 tid=26606 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: java/lang/Object@0x1399c6868[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at oracle/dfw/impl/incident/FloodController.run(FloodController.java:136)
^-- Lock released while waiting: java/lang/Object@0x1399c6868[fat lock]
at java/lang/Thread.run(Thread.java:619)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"UCP-worker-thread-16" id=487 idx=0xb8 tid=1681 prio=5 alive, parked, native_blocked, daemon
-- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x106f89660
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2221)[inlined]
at jrockit/proxy/sun/misc/Unsafe.park(Unsafe.java:592)[inlined]
at java/util/concurrent/locks/LockSupport.park(LockSupport.java:158)[inlined]
at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)[optimized]
at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)[optimized]
at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)[optimized]
at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java/lang/Thread.run(Thread.java:619)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"Thread-72" id=267 idx=0xbc tid=28298 prio=10 alive, sleeping, native_waiting, daemon
at java/lang/Thread.sleep(J)V(Native Method)
at oracle/wsm/policyaccess/impl/cache/PolicyCacheImpl$PolicyCacheRefreshThread.run(PolicyCacheImpl.java:915)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"weblogic.work.j2ee.J2EEWorkManager$WorkWithListener@37a58d77" id=393 idx=0xc0 tid=28766 prio=10 alive, in native, daemon
at jrockit/net/SocketNativeIO.readBytesPinned(Ljava/io/FileDescriptor;[BIII)I(Native Method)
at jrockit/net/SocketNativeIO.socketRead(SocketNativeIO.java:32)[inlined]
at java/net/SocketInputStream.socketRead0(Ljava/io/FileDescriptor;[BIII)I(SocketInputStream.java)[inlined]
at java/net/SocketInputStream.read(SocketInputStream.java:129)[optimized]
at oracle/net/nt/MetricsEnabledInputStream.read(TcpNTAdapter.java:564)[optimized]
at oracle/net/ns/Packet.receive(Packet.java:295)[inlined]
at oracle/net/ns/DataPacket.receive(DataPacket.java:94)[optimized]
at oracle/net/ns/NetInputStream.getNextPacket(NetInputStream.java:176)[optimized]
at oracle/net/ns/NetInputStream.read(NetInputStream.java:121)[inlined]
at oracle/net/ns/NetInputStream.read(NetInputStream.java:96)[inlined]
at oracle/net/ns/NetInputStream.read(NetInputStream.java:81)[optimized]
at oracle/jdbc/driver/T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124)[optimized]
at oracle/jdbc/driver/T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80)[inlined]
at oracle/jdbc/driver/T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1087)[optimized]
at oracle/jdbc/driver/T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1062)[inlined]
at oracle/jdbc/driver/T4C8Oall.receive(T4C8Oall.java:625)[optimized]
at oracle/jdbc/driver/T4CCallableStatement.doOall8(T4CCallableStatement.java:201)[inlined]
at oracle/jdbc/driver/T4CCallableStatement.executeForRows(T4CCallableStatement.java:1036)[optimized]
at oracle/jdbc/driver/OracleStatement.doExecuteWithTimeout(OracleStatement.java:1379)[optimized]
at oracle/jdbc/driver/OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3568)[optimized]
at oracle/jdbc/driver/OraclePreparedStatement.execute(OraclePreparedStatement.java:3739)[optimized]
^-- Holding lock: oracle/jdbc/driver/T4CConnection@0x12e33e6b8[recursive]
at oracle/jdbc/driver/OracleCallableStatement.execute(OracleCallableStatement.java:8150)[optimized]
^-- Holding lock: oracle/jdbc/driver/T4CConnection@0x12e33e6b8[thin lock]
at oracle/jdbc/driver/OraclePreparedStatementWrapper.execute(OraclePreparedStatementWrapper.java:1535)[optimized]
at weblogic/jdbc/wrapper/PreparedStatement.execute(PreparedStatement.java:99)[optimized]
at oracle/integration/platform/blocks/event/saq/SAQBusinessEventBus.readOAOODelivery(SAQBusinessEventBus.java:1302)[optimized]
at oracle/integration/platform/blocks/event/saq/SAQBusinessEventBus.handleSingleOAOODelivery(SAQBusinessEventBus.java:750)[optimized]
at oracle/integration/platform/blocks/event/saq/SAQBusinessEventBus.handleQueueEvents(SAQBusinessEventBus.java:669)
at oracle/integration/platform/blocks/event/saq/SAQBusinessEventBus.access$000(SAQBusinessEventBus.java:71)
at oracle/integration/platform/blocks/event/saq/SAQBusinessEventBus$1.run(SAQBusinessEventBus.java:319)
at oracle/integration/platform/blocks/executor/WorkManagerExecutor$1.run(WorkManagerExecutor.java:105)
at weblogic/work/j2ee/J2EEWorkManager$WorkWithListener.run(J2EEWorkManager.java:183)
at weblogic/work/DaemonWorkThread.run(DaemonWorkThread.java:30)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"weblogic.work.j2ee.J2EEWorkManager$WorkWithListener@37a4ecb8" id=392 idx=0xc4 tid=28765 prio=10 alive, in native, daemon
at jrockit/net/SocketNativeIO.readBytesPinned(Ljava/io/FileDescriptor;[BIII)I(Native Method)
at jrockit/net/SocketNativeIO.socketRead(SocketNativeIO.java:32)[inlined]
at java/net/SocketInputStream.socketRead0(Ljava/io/FileDescriptor;[BIII)I(SocketInputStream.java)[inlined]
at java/net/SocketInputStream.read(SocketInputStream.java:129)[optimized]
at oracle/net/nt/MetricsEnabledInputStream.read(TcpNTAdapter.java:564)[optimized]
at oracle/net/ns/Packet.receive(Packet.java:295)[inlined]
at oracle/net/ns/DataPacket.receive(DataPacket.java:94)[optimized]
at oracle/net/ns/NetInputStream.getNextPacket(NetInputStream.java:176)[optimized]
at oracle/net/ns/NetInputStream.read(NetInputStream.java:121)[inlined]
at oracle/net/ns/NetInputStream.read(NetInputStream.java:96)[inlined]
at oracle/net/ns/NetInputStream.read(NetInputStream.java:81)[optimized]
at oracle/jdbc/driver/T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124)[optimized]
at oracle/jdbc/driver/T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80)[inlined]
at oracle/jdbc/driver/T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1087)[optimized]
at oracle/jdbc/driver/T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1062)[inlined]
at oracle/jdbc/driver/T4C8Oall.receive(T4C8Oall.java:625)[optimized]
at oracle/jdbc/driver/T4CCallableStatement.doOall8(T4CCallableStatement.java:201)[inlined]
at oracle/jdbc/driver/T4CCallableStatement.executeForRows(T4CCallableStatement.java:1036)[optimized]
at oracle/jdbc/driver/OracleStatement.doExecuteWithTimeout(OracleStatement.java:1379)[optimized]
at oracle/jdbc/driver/OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3568)[optimized]
at oracle/jdbc/driver/OraclePreparedStatement.execute(OraclePreparedStatement.java:3739)[optimized]
^-- Holding lock: oracle/jdbc/driver/T4CConnection@0x125a80ae8[recursive]
at oracle/jdbc/driver/OracleCallableStatement.execute(OracleCallableStatement.java:8150)[optimized]
^-- Holding lock: oracle/jdbc/driver/T4CConnection@0x125a80ae8[thin lock]
at oracle/jdbc/driver/OraclePreparedStatementWrapper.execute(OraclePreparedStatementWrapper.java:1535)[optimized]
at weblogic/jdbc/wrapper/PreparedStatement.execute(PreparedStatement.java:99)[optimized]
at oracle/integration/platform/blocks/event/saq/SAQBusinessEventBus.readEvent(SAQBusinessEventBus.java:1232)[optimized]
at oracle/integration/platform/blocks/event/saq/SAQBusinessEventBus.handleSingleQueueEvent(SAQBusinessEventBus.java:687)[optimized]
at oracle/integration/platform/blocks/event/saq/SAQBusinessEventBus.handleQueueEvents(SAQBusinessEventBus.java:660)
at oracle/integration/platform/blocks/event/saq/SAQBusinessEventBus.access$000(SAQBusinessEventBus.java:71)
at oracle/integration/platform/blocks/event/saq/SAQBusinessEventBus$1.run(SAQBusinessEventBus.java:319)
at oracle/integration/platform/blocks/executor/WorkManagerExecutor$1.run(WorkManagerExecutor.java:105)
at weblogic/work/j2ee/J2EEWorkManager$WorkWithListener.run(J2EEWorkManager.java:183)
at weblogic/work/DaemonWorkThread.run(DaemonWorkThread.java:30)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"UCP-worker-thread-16" id=488 idx=0xc8 tid=1682 prio=5 alive, parked, native_blocked, daemon
-- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x102a76870
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2221)[inlined]
at jrockit/proxy/sun/misc/Unsafe.park(Unsafe.java:592)[inlined]
at java/util/concurrent/locks/LockSupport.park(LockSupport.java:158)[inlined]
at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)[optimized]
at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)[optimized]
at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)[optimized]
at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java/lang/Thread.run(Thread.java:619)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"weblogic.work.j2ee.J2EEWorkManager$WorkWithListener@37afc2f4" id=396 idx=0xcc tid=28769 prio=10 alive, sleeping, native_waiting, daemon
at java/lang/Thread.sleep(J)V(Native Method)
at oracle/tip/mediator/common/listener/DBLocker.enqueueLockedMessages(DBLocker.java:213)[optimized]
at oracle/tip/mediator/common/listener/DBLocker.run(DBLocker.java:84)
at oracle/integration/platform/blocks/executor/WorkManagerExecutor$1.run(WorkManagerExecutor.java:105)
at weblogic/work/j2ee/J2EEWorkManager$WorkWithListener.run(J2EEWorkManager.java:183)
at weblogic/work/DaemonWorkThread.run(DaemonWorkThread.java:30)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"LogFlusher" id=56 idx=0xd0 tid=26614 prio=5 alive, sleeping, native_waiting, daemon
at java/lang/Thread.sleep(J)V(Native Method)
at oracle/core/ojdl/BufferedLogWriter$Flusher.run(BufferedLogWriter.java:409)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"weblogic.work.j2ee.J2EEWorkManager$WorkWithListener@37afb4b1" id=395 idx=0xd4 tid=28768 prio=10 alive, parked, native_blocked, daemon
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2221)
at jrockit/proxy/sun/misc/Unsafe.park(Unsafe.java:592)[inlined]
at java/util/concurrent/locks/LockSupport.parkNanos(LockSupport.java:198)[inlined]
at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)[optimized]
at java/util/concurrent/LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424)[optimized]
at oracle/tip/mediator/common/listener/AbstractWorker.run(AbstractWorker.java:64)
at oracle/integration/platform/blocks/executor/WorkManagerExecutor$1.run(WorkManagerExecutor.java:105)
at weblogic/work/j2ee/J2EEWorkManager$WorkWithListener.run(J2EEWorkManager.java:183)
You need to open a support case, and it's for the application, not WebLogic. You have threads deadlocked
in application code.
Circular (deadlocked) lock chains
=================================
Chain 5:
"[STUCK] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)'" id=577 idx=0x560 tid=5192 waiting for java/util/concurrent/locks/ReentrantLock$NonfairSync@0x3895ee610 held by:
"[STUCK] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)'" id=433 idx=0x278 tid=30215 waiting for java/util/concurrent/locks/ReentrantLock$NonfairSync@0x38a904880 held by:
"[STUCK] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)'" id=577 idx=0x560 tid=5192 -
Hello All,
We are getting stuck thread in weblogic under high load.
following is the thread dump from weblogic 10
"[STANDBY] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)'" id=83 idx=0x134 tid=5376 prio=5 alive, in native, waiting, daemon
-- Waiting for notification on: weblogic/work/ExecuteThread@0x06DEC880[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:474)
at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)
^-- Lock released while waiting: weblogic/work/ExecuteThread@0x06DEC880[fat lock]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"[STANDBY] ExecuteThread: '14' for queue: 'weblogic.kernel.Default (self-tuning)'" id=84 idx=0x138 tid=1688 prio=5 alive, in native, waiting, daemon
-- Waiting for notification on: weblogic/work/ExecuteThread@0x06E50070[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:474)
at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)
^-- Lock released while waiting: weblogic/work/ExecuteThread@0x06E50070[fat lock]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"[STUCK] ExecuteThread: '15' for queue: 'weblogic.kernel.Default (self-tuning)'" id=85 idx=0x13c tid=3984 prio=1 alive, daemon
at jrockit/vm/Allocator.allocArray(Allocator.java:392)
at java/lang/AbstractStringBuilder.<init>(AbstractStringBuilder.java:44)
at java/lang/StringBuffer.<init>(StringBuffer.java:80)
at com/vignette/exolab/castor/jdo/oql/ParseTreeWalker.addWhereClause(ParseTreeWalker.java:994)
at com/vignette/exolab/castor/jdo/oql/ParseTreeWalker.createQueryExpression(ParseTreeWalker.java:816)
at com/vignette/exolab/castor/jdo/oql/ParseTreeWalker.<init>(ParseTreeWalker.java:139)
at com/vignette/exolab/castor/jdo/engine/OQLQueryImpl.create(OQLQueryImpl.java:274)
at com/vignette/exolab/castor/jdo/engine/DatabaseImpl.getOQLQuery(DatabaseImpl.java:502)
at com/vignette/as/server/persist/dao/JDODAOPersistor.findByQuery(JDODAOPersistor.java:395)
at com/vignette/as/server/persist/dao/JDODAOPersistor.findByIds(JDODAOPersistor.java:360)
at com/vignette/as/server/persist/ExtensibleApplicationObjectPersistor.findByIdsHelper(ExtensibleApplicationObjectPersistor.java:159)
at com/vignette/as/server/persist/ExtensibleApplicationObjectPersistor.findByIds(ExtensibleApplicationObjectPersistor.java:90)
at com/vignette/as/server/logic/channel/ChannelOpsLogic.findById(ChannelOpsLogic.java:194)
at com/vignette/as/server/logic/channel/ChannelOpsLogicGen.findById(ChannelOpsLogicGen.java:1553)
at com/vignette/as/client/javabean/Channel.findById(Channel.java:247)
at com/vignette/as/server/logic/channel/ChannelOpsLogic.getSubchannels(ChannelOpsLogic.java:563)
at com/vignette/as/server/logic/channel/ChannelOpsLogicGen.getSubchannels(ChannelOpsLogicGen.java:3237)
at com/vignette/as/client/javabean/ChannelBase.getSubchannels(ChannelBase.java:480)
at com/vignette/as/client/javabean/ChannelBase.getAllSubchannels(ChannelBase.java:375)
at com/astrazeneca/portal/dpm/ChannelWrapper.getSubchannels(ChannelWrapper.java:203)
at jsp_servlet/_jsp/_grid/__xmlgeneration.subChannelRec(__xmlgeneration.java:97)
at jsp_servlet/_jsp/_grid/__xmlgeneration.subChannelRec(__xmlgeneration.java:146)
at jsp_servlet/_jsp/_grid/__xmlgeneration._jspService(__xmlgeneration.java:430)
at weblogic/servlet/jsp/JspBase.service(JspBase.java:34)
at weblogic/servlet/internal/StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:226)
at weblogic/servlet/internal/StubSecurityHelper.invokeServlet(StubSecurityHelper.java:124)
at weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:283)
at weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:175)
at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3395)
at weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic/security/service/SecurityManager.runAs(Lweblogic/security/acl/internal/AuthenticatedSubject;Lweblogic/security/acl/internal/AuthenticatedSubject;Ljava/security/PrivilegedAction;)Ljava/lang/Object;(Unknown Source)
at weblogic/servlet/internal/WebAppServletContext.securedExecute(WebAppServletContext.java:2140)
at weblogic/servlet/internal/WebAppServletContext.execute(WebAppServletContext.java:2046)
at weblogic/servlet/internal/ServletRequestImpl.run(ServletRequestImpl.java:1366)
at weblogic/work/ExecuteThread.execute(ExecuteThread.java:200)
at weblogic/work/ExecuteThread.run(ExecuteThread.java:172)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"[STANDBY] ExecuteThread: '16' for queue: 'weblogic.kernel.Default (self-tuning)'" id=86 idx=0x140 tid=1192 prio=5 alive, in native, waiting, daemon
-- Waiting for notification on: weblogic/work/ExecuteThread@0x06E50AF0[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:474)
at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)
^-- Lock released while waiting: weblogic/work/ExecuteThread@0x06E50AF0[fat lock]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"VAP Topic Consumer Thread - 0" id=87 idx=0x144 tid=6040 prio=5 alive, in native, parked, daemon
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2416)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at java/util/concurrent/locks/LockSupport.park(LockSupport.java:118)
at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1767)
at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
at com/epicentric/topic/internal/MessageQueue.remove(MessageQueue.java:67)
at com/epicentric/topic/internal/Consumer.run(Consumer.java:72)
at java/lang/Thread.run(Thread.java:595)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"VAP Topic Consumer Thread - 1" id=88 idx=0x148 tid=4752 prio=5 alive, in native, parked, daemon
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2416)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at java/util/concurrent/locks/LockSupport.park(LockSupport.java:118)
at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1767)
at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
at com/epicentric/topic/internal/MessageQueue.remove(MessageQueue.java:67)
at com/epicentric/topic/internal/Consumer.run(Consumer.java:72)
at java/lang/Thread.run(Thread.java:595)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"VAP Topic Consumer Thread - 2" id=89 idx=0x14c tid=4504 prio=5 alive, in native, parked, daemon
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2416)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at java/util/concurrent/locks/LockSupport.park(LockSupport.java:118)
at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1767)
at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
at com/epicentric/topic/internal/MessageQueue.remove(MessageQueue.java:67)
at com/epicentric/topic/internal/Consumer.run(Consumer.java:72)
at java/lang/Thread.run(Thread.java:595)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"VAP Topic Consumer Thread - 3" id=90 idx=0x150 tid=5572 prio=5 alive, in native, parked, daemon
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2416)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at java/util/concurrent/locks/LockSupport.park(LockSupport.java:118)
at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1767)
at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
at com/epicentric/topic/internal/MessageQueue.remove(MessageQueue.java:67)
at com/epicentric/topic/internal/Consumer.run(Consumer.java:72)
at java/lang/Thread.run(Thread.java:595)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"VAP Topic Consumer Thread - 4" id=91 idx=0x154 tid=4340 prio=5 alive, in native, parked, daemon
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2416)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at java/util/concurrent/locks/LockSupport.park(LockSupport.java:118)
at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1767)
at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
at com/epicentric/topic/internal/MessageQueue.remove(MessageQueue.java:67)
at com/epicentric/topic/internal/Consumer.run(Consumer.java:72)
at java/lang/Thread.run(Thread.java:595)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"CS_Local_Monitor_0" id=92 idx=0x158 tid=3864 prio=5 alive, in native, sleeping, native_waiting, daemon
at java/lang/Thread.sleep(J)V(Native Method)
at com/vignette/config/client/cachefile/ConfigSpaceLocal$Monitor.run(ConfigSpaceLocal.java:1364)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"Thread-41" id=93 idx=0x15c tid=4884 prio=1 alive, in native, waiting, daemon
-- Waiting for notification on: java/util/ArrayList@0x0A899918[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:474)
at com/vignette/logging/LocaleAppender$Dispatcher.run(LocaleAppender.java:393)
^-- Lock released while waiting: java/util/ArrayList@0x0A899918[fat lock]
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"Thread-42" id=94 idx=0x160 tid=4968 prio=1 alive, in native, waiting, daemon
-- Waiting for notification on: java/util/ArrayList@0x06BE2238[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:474)
at com/vignette/logging/LocaleAppender$Dispatcher.run(LocaleAppender.java:393)
^-- Lock released while waiting: java/util/ArrayList@0x06BE2238[fat lock]
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"TaskScheduler thread" id=98 idx=0x164 tid=1580 prio=5 alive, in native, sleeping, native_waiting, daemon
at java/lang/Thread.sleep(J)V(Native Method)
at com/epicentric/task/TaskScheduler.sleepUntilRuntime(TaskScheduler.java:226)
at com/epicentric/task/TaskScheduler.run(TaskScheduler.java:83)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"TaskQueue Lease Manager" id=100 idx=0x168 tid=3908 prio=5 alive, in native, sleeping, native_waiting, daemon
at java/lang/Thread.sleep(J)V(Native Method)
at com/epicentric/task/internal/LeaseManager.waitForNextExecutionTime(LeaseManager.java:69)
at com/epicentric/task/internal/LeaseManager.run(LeaseManager.java:52)
at java/lang/Thread.run(Thread.java:595)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"Thread System Expiration and Ostracizing Thread" id=101 idx=0x16c tid=4200 prio=5 alive, in native, sleeping, native_waiting, daemon
at java/lang/Thread.sleep(J)V(Native Method)
at com/epicentric/threadpool/standard/StandardThreadPool$ExpirationThread.run(StandardThreadPool.java:238)
at java/lang/Thread.run(Thread.java:595)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"Task Executor" id=102 idx=0x170 tid=3592 prio=1 alive, in native, sleeping, native_waiting, daemon
at java/lang/Thread.sleep(J)V(Native Method)
at com/epicentric/task/internal/TaskExecutor.waitForNextExecutionTime(TaskExecutor.java:205)
at com/epicentric/task/internal/TaskExecutor.run(TaskExecutor.java:186)
at java/lang/Thread.run(Thread.java:595)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"Task Metric Gatherer" id=103 idx=0x174 tid=5988 prio=1 alive, in native, sleeping, native_waiting, daemon
at java/lang/Thread.sleep(J)V(Native Method)
at com/epicentric/task/internal/TaskExecutor$1.run(TaskExecutor.java:142)
at java/lang/Thread.run(Thread.java:595)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"CS_Local_Monitor_0" id=142 idx=0x178 tid=5196 prio=5 alive, in native, sleeping, native_waiting, daemon
at java/lang/Thread.sleep(J)V(Native Method)
at com/vignette/config/client/cachefile/ConfigSpaceLocal$Monitor.run(ConfigSpaceLocal.java:1364)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"Thread-54" id=130 idx=0x17c tid=648 prio=1 alive, in native, waiting, daemon
-- Waiting for notification on: java/util/ArrayList@0x0AD3C798[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:474)
at com/vignette/logging/LocaleAppender$Dispatcher.run(LocaleAppender.java:393)
^-- Lock released while waiting: java/util/ArrayList@0x0AD3C798[fat lock]
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"Timer-10" id=113 idx=0x180 tid=1092 prio=5 alive, in native, waiting, daemon
-- Waiting for notification on: java/util/TaskQueue@0x00B22F40[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/util/TimerThread.mainLoop(Timer.java:509)
^-- Lock released while waiting: java/util/TaskQueue@0x00B22F40[fat lock]
at java/util/TimerThread.run(Timer.java:462)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"JNDI Monitor" id=116 idx=0x184 tid=1856 prio=5 alive, in native, waiting, daemon
-- Waiting for notification on: com/epicentric/jndi/Monitor@0x00B12D60[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at com/epicentric/jndi/Monitor.run(Monitor.java:100)
^-- Lock released while waiting: com/epicentric/jndi/Monitor@0x00B12D60[fat lock]
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"Thread-47" id=117 idx=0x188 tid=816 prio=5 alive, in native, daemon
at jrockit/net/SocketNativeIO.readBytesPinned(Ljava/io/FileDescriptor;[BIII)I(Native Method)
at jrockit/net/SocketNativeIO.socketRead(SocketNativeIO.java:31)
at java/net/SocketInputStream.socketRead0(Ljava/io/FileDescriptor;[BIII)I(SocketInputStream.java)
at java/net/SocketInputStream.read(SocketInputStream.java:129)
at java/io/BufferedInputStream.fill(BufferedInputStream.java:218)
at java/io/BufferedInputStream.read1(BufferedInputStream.java:256)
at java/io/BufferedInputStream.read(BufferedInputStream.java:313)
^-- Holding lock: java/io/BufferedInputStream@0x00B3EA48[thin lock]
at com/sun/jndi/ldap/Connection.run(Connection.java:784)
at java/lang/Thread.run(Thread.java:595)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"TaskScheduler thread" id=119 idx=0x18c tid=4520 prio=5 alive, in native, sleeping, native_waiting, daemon
at java/lang/Thread.sleep(J)V(Native Method)
at com/epicentric/task/TaskScheduler.run(TaskScheduler.java:94)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"OracleTimeoutPollingThread" id=124 idx=0x190 tid=2628 prio=10 alive, in native, sleeping, native_waiting, daemon
at java/lang/Thread.sleep(J)V(Native Method)
at oracle/jdbc/driver/OracleTimeoutPollingThread.run(OracleTimeoutPollingThread.java:158)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"Thread-55" id=131 idx=0x194 tid=5700 prio=1 alive, in native, waiting, daemon
-- Waiting for notification on: java/util/ArrayList@0x0973F378[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:474)
at com/vignette/logging/LocaleAppender$Dispatcher.run(LocaleAppender.java:393)
^-- Lock released while waiting: java/util/ArrayList@0x0973F378[fat lock]
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"Thread-56" id=132 idx=0x198 tid=5596 prio=5 alive, in native, sleeping, native_waiting, daemon
at java/lang/Thread.sleep(J)V(Native Method)
at org/apache/commons/pool/impl/GenericObjectPool$Evictor.run(GenericObjectPool.java:1122)
at java/lang/Thread.run(Thread.java:595)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"CacheSyncThread" id=133 idx=0x19c tid=708 prio=5 alive, in native, sleeping, native_waiting, daemon
at java/lang/Thread.sleep(J)V(Native Method)
at com/vignette/as/server/caching/CacheSynchronization.run(CacheSynchronization.java:135)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"Thread-58" id=135 idx=0x1a0 tid=5652 prio=5 alive, in native, sleeping, native_waiting, daemon
at java/lang/Thread.sleep(J)V(Native Method)
at org/apache/commons/pool/impl/GenericObjectPool$Evictor.run(GenericObjectPool.java:1122)
at java/lang/Thread.run(Thread.java:595)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"Thread-61" id=143 idx=0x1a4 tid=4196 prio=1 alive, in native, waiting, daemon
-- Waiting for notification on: java/util/ArrayList@0x0BEF1160[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:474)
at com/vignette/logging/LocaleAppender$Dispatcher.run(LocaleAppender.java:393)
^-- Lock released while waiting: java/util/ArrayList@0x0BEF1160[fat lock]
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"Thread-62" id=144 idx=0x1a8 tid=5456 prio=1 alive, in native, waiting, daemon
-- Waiting for notification on: java/util/ArrayList@0x0D650770[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:474)
at com/vignette/logging/LocaleAppender$Dispatcher.run(LocaleAppender.java:393)
^-- Lock released while waiting: java/util/ArrayList@0x0D650770[fat lock]
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"Thread-63" id=145 idx=0x1ac tid=1056 prio=5 alive, in native, sleeping, native_waiting, daemon
at java/lang/Thread.sleep(J)V(Native Method)
at org/apache/commons/pool/impl/GenericObjectPool$Evictor.run(GenericObjectPool.java:1122)
at java/lang/Thread.run(Thread.java:595)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"CacheSyncThread" id=146 idx=0x1b0 tid=3496 prio=5 alive, in native, sleeping, native_waiting, daemon
at java/lang/Thread.sleep(J)V(Native Method)
at com/vignette/as/server/caching/CacheSynchronization.run(CacheSynchronization.java:135)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"Thread-64" id=147 idx=0x1b4 tid=4132 prio=5 alive, in native, sleeping, native_waiting, daemon
at java/lang/Thread.sleep(J)V(Native Method)
at org/apache/commons/pool/impl/GenericObjectPool$Evictor.run(GenericObjectPool.java:1122)
at java/lang/Thread.run(Thread.java:595)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"CS_Local_Monitor_0" id=221 idx=0x1b8 tid=6104 prio=5 alive, in native, sleeping, native_waiting, daemon
at java/lang/Thread.sleep(J)V(Native Method)
at com/vignette/config/client/cachefile/ConfigSpaceLocal$Monitor.run(ConfigSpaceLocal.java:1364)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"Thread-84" id=222 idx=0x1bc tid=3500 prio=1 alive, in native, waiting, daemon
-- Waiting for notification on: java/util/ArrayList@0x1673EAD8[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:474)
at com/vignette/logging/LocaleAppender$Dispatcher.run(LocaleAppender.java:393)
^-- Lock released while waiting: java/util/ArrayList@0x1673EAD8[fat lock]
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"Thread-85" id=223 idx=0x1c0 tid=5492 prio=1 alive, in native, waiting, daemon
-- Waiting for notification on: java/util/ArrayList@0x1932F128[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:474)
at com/vignette/logging/LocaleAppender$Dispatcher.run(LocaleAppender.java:393)
^-- Lock released while waiting: java/util/ArrayList@0x1932F128[fat lock]
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"[STANDBY] ExecuteThread: '17' for queueHi,
We are facing a similar stuck thread problem with the following stack trace...
Any help on this is much appreciated.
Thanks,
Sen
Thread-86 "[STUCK] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, waiting, priority=1, DAEMON> {
-- Waiting for notification on: netscape.ldap.LDAPSearchListener@2accce6[fat lock]
java.lang.Object.wait(Object.java:???)
java.lang.Object.wait(Object.java:485)
netscape.ldap.LDAPMessageQueue.waitForMessage(LDAPMessageQueue.java:198)
netscape.ldap.LDAPMessageQueue.waitFirstMessage(LDAPMessageQueue.java:100)
netscape.ldap.LDAPConnection.sendRequest(LDAPConnection.java:1782)
^-- Holding lock: netscape.ldap.LDAPConnection@138555c[thin lock]
netscape.ldap.LDAPConnection.search(LDAPConnection.java:2500)
weblogic.security.providers.authentication.LDAPAtnDelegate.getDNForUser(LDAPAtnDelegate.java:3317)
weblogic.security.providers.authentication.LDAPAtnDelegate.listMemberGroups(LDAPAtnDelegate.java:2091)
weblogic.security.providers.authentication.LDAPAtnLoginModuleImpl.addAllGroups(LDAPAtnLoginModuleImpl.java:419)
weblogic.security.providers.authentication.LDAPAtnLoginModuleImpl.addGroups(LDAPAtnLoginModuleImpl.java:411)
weblogic.security.providers.authentication.LDAPAtnLoginModuleImpl.login(LDAPAtnLoginModuleImpl.java:136)
com.bea.common.security.internal.service.LoginModuleWrapper$1.run(LoginModuleWrapper.java:110)
com.bea.common.security.internal.service.LoginModuleWrapper.login(LoginModuleWrapper.java:101)
sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:???)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:27)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
javax.security.auth.login.LoginContext.invoke(LoginContext.java:720)
javax.security.auth.login.LoginContext.access$000(LoginContext.java:186)
javax.security.auth.login.LoginContext$4.run(LoginContext.java:683)
javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
javax.security.auth.login.LoginContext.login(LoginContext.java:566)
com.bea.common.security.internal.service.JAASLoginServiceImpl.login(JAASLoginServiceImpl.java:77)
sun.reflect.GeneratedMethodAccessor2726.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
com.bea.common.security.internal.utils.Delegator$ProxyInvocationHandler.invoke(Delegator.java:57)
$Proxy17.login(Unknown Source)
weblogic.security.service.internal.WLSJAASLoginServiceImpl$ServiceImpl.login(Unknown Source)
com.bea.common.security.internal.service.JAASAuthenticationServiceImpl.authenticate(JAASAuthenticationServiceImpl.java:74)
sun.reflect.GeneratedMethodAccessor2727.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
com.bea.common.security.internal.utils.Delegator$ProxyInvocationHandler.invoke(Delegator.java:57)
$Proxy37.authenticate(Unknown Source)
weblogic.security.service.WLSJAASAuthenticationServiceWrapper.authenticate(Unknown Source)
weblogic.security.service.PrincipalAuthenticator.authenticate(Unknown Source)
weblogic.servlet.security.internal.SecurityModule.checkAuthenticate(SecurityModule.java:213)
weblogic.servlet.security.ServletAuthentication.login(ServletAuthentication.java:391)
weblogic.servlet.security.ServletAuthentication.weak(ServletAuthentication.java:371)
com.bea.p13n.security.Authentication.login(Authentication.java:118)
com.ppluk.moneyout.portal.auth.LoginServlet.doGet(LoginServlet.java:40)
javax.servlet.http.HttpServlet.service(HttpServlet.java:700)
javax.servlet.http.HttpServlet.service(HttpServlet.java:815)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:198)
weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:41)
com.bea.portal.tools.servlet.http.HttpContextFilter.doFilter(HttpContextFilter.java:50)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:41)
com.bea.p13n.servlets.PortalServletFilter.doFilter(PortalServletFilter.java:241)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:41)
com.bea.jsptools.servlet.PagedResultServiceFilter.doFilter(PagedResultServiceFilter.java:61)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:41)
weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:25)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:41)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3468)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:308)
weblogic.security.service.SecurityManager.runAs(Unknown Source)
weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2116)
weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2038)
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1372)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:198)
weblogic.work.ExecuteThread.run(ExecuteThread.java:165)
} -
Waiting for notification on: weblogic/t3/srvr/T3Srvr@0xac909c80[fat lock]
i am trying to deploy a soa composite and i end up getting the following error.....
looks like the first phase of the deployment is successfull and the setStatus has a thread block , please suggest
THREAD DUMP
"Main Thread" id=1 idx=0x4 tid=1099 prio=5 alive, waiting, native_blocked
-- Waiting for notification on: weblogic/t3/srvr/T3Srvr@0xac909c80[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at weblogic/t3/srvr/T3Srvr.waitForDeath(T3Srvr.java:979)
^-- Lock released while waiting: weblogic/t3/srvr/T3Srvr@0xac909c80[fat lock]
at weblogic/t3/srvr/T3Srvr.run(T3Srvr.java:488)
at weblogic/Server.main(Server.java:71)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"(Signal Handler)" id=2 idx=0x8 tid=1100 prio=5 alive, native_blocked, daemon
"(OC Main Thread)" id=3 idx=0xc tid=1101 prio=5 alive, native_waiting, daemon
"(VM Periodic Task)" id=6 idx=0x38 tid=1112 prio=10 alive, native_blocked, daemon
Blocked lock chains
===================
Chain 2:
"ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" id=26 idx=0x74 tid=1179 waiting for java/lang/String@0xac909020 held by:
"ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" id=27 idx=0x78 tid=1180 in chain 1
Chain 3:
"ExecuteThread: '3' for queue: 'weblogic.socket.Muxer'" id=28 idx=0x7c tid=1181 waiting for java/lang/String@0xac909020 held by:
"ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" id=27 idx=0x78 tid=1180 in chain 1
Open lock chains
================
Chain 1:
"ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" id=25 idx=0x70 tid=1178 waiting for java/lang/String@0xac909020 held by:
"ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" id=27 idx=0x78 tid=1180 (active)
===== END OF THREAD DUMP ===============You may refer -
SOA composite deployment exception [STUCK] ExecuteThread: '4' for queue:
Regards,
Anuj -
Getting Stuck threads ,application is going down
Hi All ,
Need your expertise on the below issue . from the past 2 days we are getting stuck thread issues . can you please provide insight into the below issue please ?
===== FULL THREAD DUMP ===============
Wed Jan 8 09:28:23 2014
Oracle JRockit(R) R28.2.3-13-149708-1.6.0_31-20120327-1523-linux-x86_64
"Main Thread" id=1 idx=0x4 tid=24390 prio=5 alive, waiting, native_blocked
-- Waiting for notification on: weblogic/t3/srvr/T3Srvr@0xf18da770[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at weblogic/t3/srvr/T3Srvr.waitForDeath(T3Srvr.java:981)
^-- Lock released while waiting: weblogic/t3/srvr/T3Srvr@0xf18da770[fat lock]
at weblogic/t3/srvr/T3Srvr.run(T3Srvr.java:490)
at weblogic/Server.main(Server.java:71)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"(Signal Handler)" id=2 idx=0x8 tid=24391 prio=5 alive, native_blocked, daemon
"(OC Main Thread)" id=3 idx=0xc tid=24392 prio=5 alive, native_waiting, daemon
"(GC Worker Thread 1)" id=? idx=0x10 tid=24393 prio=5 alive, daemon
"(GC Worker Thread 2)" id=? idx=0x14 tid=24394 prio=5 alive, daemon
"(GC Worker Thread 3)" id=? idx=0x18 tid=24395 prio=5 alive, daemon
"(GC Worker Thread 4)" id=? idx=0x1c tid=24396 prio=5 alive, daemon
"(GC Worker Thread 5)" id=? idx=0x20 tid=24397 prio=5 alive, daemon
"(GC Worker Thread 6)" id=? idx=0x24 tid=24398 prio=5 alive, daemon
"(GC Worker Thread 7)" id=? idx=0x28 tid=24399 prio=5 alive, daemon
"(GC Worker Thread 8)" id=? idx=0x2c tid=24400 prio=5 alive, daemon
"(GC Worker Thread 9)" id=? idx=0x30 tid=24401 prio=5 alive, daemon
"(GC Worker Thread 10)" id=? idx=0x34 tid=24402 prio=5 alive, daemon
"(GC Worker Thread 11)" id=? idx=0x38 tid=24403 prio=5 alive, daemon
"(GC Worker Thread 12)" id=? idx=0x3c tid=24404 prio=5 alive, daemon
"(GC Worker Thread 13)" id=? idx=0x40 tid=24405 prio=5 alive, daemon
"(GC Worker Thread 14)" id=? idx=0x44 tid=24406 prio=5 alive, daemon
"(GC Worker Thread 15)" id=? idx=0x48 tid=24407 prio=5 alive, daemon
"(GC Worker Thread 16)" id=? idx=0x4c tid=24408 prio=5 alive, daemon
"(GC Worker Thread 17)" id=? idx=0x50 tid=24409 prio=5 alive, daemon
"(GC Worker Thread 18)" id=? idx=0x54 tid=24410 prio=5 alive, daemon
"(Code Generation Thread 1)" id=4 idx=0x58 tid=24411 prio=5 alive, native_waiting, daemon
"(Code Optimization Thread 1)" id=5 idx=0x5c tid=24412 prio=5 alive, native_waiting, daemon
"(VM Periodic Task)" id=6 idx=0x60 tid=24413 prio=10 alive, native_blocked, daemon
"Finalizer" id=7 idx=0x64 tid=24414 prio=8 alive, native_waiting, daemon
at jrockit/memory/Finalizer.waitForFinalizees(J[Ljava/lang/Object;)I(Native Method)
at jrockit/memory/Finalizer.access$700(Finalizer.java:12)
at jrockit/memory/Finalizer$4.run(Finalizer.java:189)
at java/lang/Thread.run(Thread.java:662)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"Reference Handler" id=8 idx=0x68 tid=24415 prio=10 alive, native_waiting, daemon
at java/lang/ref/Reference.waitForActivatedQueue(J)Ljava/lang/ref/Reference;(Native Method)
at java/lang/ref/Reference.access$100(Reference.java:11)
at java/lang/ref/Reference$ReferenceHandler.run(Reference.java:82)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"(Sensor Event Thread)" id=9 idx=0x6c tid=24416 prio=5 alive, native_blocked, daemon
"VM JFR Buffer Thread" id=10 idx=0x70 tid=24417 prio=5 alive, in native, daemon
"Timer-0" id=13 idx=0x74 tid=24418 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: java/util/TaskQueue@0xf190d120[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at java/util/TimerThread.mainLoop(Timer.java:483)
^-- Lock released while waiting: java/util/TaskQueue@0xf190d120[fat lock]
at java/util/TimerThread.run(Timer.java:462)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"Timer-1" id=14 idx=0x78 tid=24419 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: java/util/TaskQueue@0xf190d188[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/util/TimerThread.mainLoop(Timer.java:509)
^-- Lock released while waiting: java/util/TaskQueue@0xf190d188[fat lock]
at java/util/TimerThread.run(Timer.java:462)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"[STUCK] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" id=15 idx=0x7c tid=24420 prio=1 alive, blocked, native_blocked, daemon
-- Blocked trying to get lock: oracle/jdbc/driver/T4CConnection@0xe4b45e08[thin lock]
at jrockit/vm/Threads.sleep(I)V(Native Method)
at jrockit/vm/Locks.waitForThinRelease(Locks.java:955)[optimized]
at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1083)
at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1005)
at jrockit/vm/Locks.monitorEnter(Locks.java:2179)
at oracle/jdbc/driver/PhysicalConnection.rollback(PhysicalConnection.java:3896)
at weblogic/jdbc/wrapper/Connection.forcedCleanup(Connection.java:153)
at weblogic/common/resourcepool/ResourcePoolImpl.timeoutInactiveResources(ResourcePoolImpl.java:1953)
at weblogic/common/resourcepool/ResourcePoolImpl.access$1500(ResourcePoolImpl.java:41)
at weblogic/common/resourcepool/ResourcePoolImpl$ResourcePoolMaintanenceTask.timerExpired(ResourcePoolImpl.java:2675)
at weblogic/timers/internal/TimerImpl.run(TimerImpl.java:273)
at weblogic/work/SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528)
at weblogic/work/ExecuteThread.execute(ExecuteThread.java:209)
at weblogic/work/ExecuteThread.run(ExecuteThread.java:178)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"JFR request timer" id=16 idx=0x80 tid=24421 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: java/util/TaskQueue@0xf190d208[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at java/util/TimerThread.mainLoop(Timer.java:483)
^-- Lock released while waiting: java/util/TaskQueue@0xf190d208[fat lock]
at java/util/TimerThread.run(Timer.java:462)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"weblogic.time.TimeEventGenerator" id=18 idx=0x84 tid=24422 prio=9 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/time/common/internal/TimeTable@0xf190e690[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at weblogic/time/common/internal/TimeTable.snooze(TimeTable.java:286)
^-- Lock released while waiting: weblogic/time/common/internal/TimeTable@0xf190e690[fat lock]
at weblogic/time/common/internal/TimeEventGenerator.run(TimeEventGenerator.java:117)
at java/lang/Thread.run(Thread.java:662)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"JMAPI event thread" id=19 idx=0x88 tid=24423 prio=5 alive, in native, daemon
"weblogic.timers.TimerThread" id=20 idx=0x8c tid=24424 prio=9 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/timers/internal/TimerThread@0xf190d310[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at weblogic/timers/internal/TimerThread$Thread.run(TimerThread.java:262)
^-- Lock released while waiting: weblogic/timers/internal/TimerThread@0xf190d310[fat lock]
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" id=21 idx=0x90 tid=24425 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/work/ExecuteThread@0xf190dfe8[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:162)
^-- Lock released while waiting: weblogic/work/ExecuteThread@0xf190dfe8[fat lock]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:183)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" id=24 idx=0x94 tid=24426 prio=5 alive, blocked, native_blocked, daemon
-- Blocked trying to get lock: java/lang/String@0xf18a6480[fat lock]
at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)
at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1411)[optimized]
at jrockit/vm/Locks.lockFat(Locks.java:1512)[optimized]
at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1054)[optimized]
at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1005)[optimized]
at jrockit/vm/Locks.monitorEnter(Locks.java:2179)[optimized]
at weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:153)
at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)
at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)
at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" id=25 idx=0x98 tid=24427 prio=5 alive, in native, daemon
at jrockit/ext/epoll/EPoll.epollWait0(ILjava/nio/ByteBuffer;II)I(Native Method)
at jrockit/ext/epoll/EPoll.epollWait(EPoll.java:115)
at weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:156)
^-- Holding lock: java/lang/String@0xf18a6480[fat lock]
at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)
at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)
at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" id=26 idx=0x9c tid=24428 prio=5 alive, blocked, native_blocked, daemon
-- Blocked trying to get lock: java/lang/String@0xf18a6480[fat lock]
at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)
at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1411)[optimized]
at jrockit/vm/Locks.lockFat(Locks.java:1512)[optimized]
at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1054)[optimized]
at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1005)[optimized]
at jrockit/vm/Locks.monitorEnter(Locks.java:2179)[optimized]
at weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:153)
at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)
at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)
at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '3' for queue: 'weblogic.socket.Muxer'" id=27 idx=0xa0 tid=24429 prio=5 alive, blocked, native_blocked, daemon
-- Blocked trying to get lock: java/lang/String@0xf18a6480[fat lock]
at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)
at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1411)[optimized]
at jrockit/vm/Locks.lockFat(Locks.java:1512)[optimized]
at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1054)[optimized]
at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1005)[optimized]
at jrockit/vm/Locks.monitorEnter(Locks.java:2179)[optimized]
at weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:153)
at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)
at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)
at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"VDE Transaction Processor Thread" id=30 idx=0xa4 tid=24430 prio=2 alive, waiting, native_blocked, daemon
-- Waiting for notification on: com/octetstring/vde/backend/standard/TransactionProcessor@0xf190e0d0[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at com/octetstring/vde/backend/standard/TransactionProcessor.waitChange(TransactionProcessor.java:367)
^-- Lock released while waiting: com/octetstring/vde/backend/standard/TransactionProcessor@0xf190e0d0[fat lock]
at com/octetstring/vde/backend/standard/TransactionProcessor.run(TransactionProcessor.java:212)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"Query_Execution_78" id=248 idx=0xa8 tid=28978 prio=5 alive, parked, native_blocked, daemon
-- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0xe2f645d8
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2230)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at java/util/concurrent/locks/LockSupport.park(LockSupport.java:156)
at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java/lang/Thread.run(Thread.java:662)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"DoSManager" id=33 idx=0xac tid=24432 prio=6 alive, sleeping, native_waiting, daemon
at java/lang/Thread.sleep(J)V(Native Method)
at com/octetstring/vde/DoSManager.run(DoSManager.java:433)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"Thread-12" id=34 idx=0xb0 tid=24433 prio=5 alive, parked, native_blocked
-- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0xf48abb88
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2230)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at java/util/concurrent/locks/LockSupport.park(LockSupport.java:156)
at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at weblogic/utils/concurrent/JDK15ConcurrentBlockingQueue.take(JDK15ConcurrentBlockingQueue.java:89)
at weblogic/store/internal/PersistentStoreImpl.getOutstandingWork(PersistentStoreImpl.java:672)
at weblogic/store/internal/PersistentStoreImpl.run(PersistentStoreImpl.java:721)
at java/lang/Thread.run(Thread.java:662)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"Timer-2" id=35 idx=0xb4 tid=24434 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: java/util/TaskQueue@0xf5cf4fc0[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/util/TimerThread.mainLoop(Timer.java:509)
^-- Lock released while waiting: java/util/TaskQueue@0xf5cf4fc0[fat lock]
at java/util/TimerThread.run(Timer.java:462)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"Timer-3" id=37 idx=0xb8 tid=24435 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: java/util/TaskQueue@0xf5c14d00[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/util/TimerThread.mainLoop(Timer.java:509)
^-- Lock released while waiting: java/util/TaskQueue@0xf5c14d00[fat lock]
at java/util/TimerThread.run(Timer.java:462)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"DBQueueList" id=36 idx=0xbc tid=24436 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: java/lang/Object@0xf5c24d00[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at com/demantra/applicationServer/metaDataObjects/dbQueue/DBQueueList.run(DBQueueList.java:256)
^-- Lock released while waiting: java/lang/Object@0xf5c24d00[fat lock]
at java/lang/Thread.run(Thread.java:662)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"DemantraWorkflowLogCleaner" id=38 idx=0xc0 tid=24437 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: com/demantra/workflow/util/WorkflowServer$LogCleaner@0xe4abf950[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at com/demantra/workflow/util/WorkflowServer$LogCleaner.run(WorkflowServer.java:1379)
^-- Lock released while waiting: com/demantra/workflow/util/WorkflowServer$LogCleaner@0xe4abf950[fat lock]
at java/lang/Thread.run(Thread.java:662)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"WFProcess_0" id=39 idx=0xc4 tid=24438 prio=5 alive, in native, daemon
at jrockit/net/SocketNativeIO.readBytesPinned(Ljava/io/FileDescriptor;[BIII)I(Native Method)
at jrockit/net/SocketNativeIO.socketRead(SocketNativeIO.java:32)
at java/net/SocketInputStream.socketRead0(Ljava/io/FileDescriptor;[BIII)I(SocketInputStream.java)
at java/net/SocketInputStream.read(SocketInputStream.java:129)
at oracle/net/ns/Packet.receive(Packet.java:293)
at oracle/net/ns/DataPacket.receive(DataPacket.java:104)
at oracle/net/ns/NetInputStream.getNextPacket(NetInputStream.java:315)[optimized]
at oracle/net/ns/NetInputStream.read(NetInputStream.java:260)
at oracle/net/ns/NetInputStream.read(NetInputStream.java:185)
at oracle/net/ns/NetInputStream.read(NetInputStream.java:102)
at oracle/jdbc/driver/T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124)
at oracle/jdbc/driver/T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80)
at oracle/jdbc/driver/T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1136)
at oracle/jdbc/driver/T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1113)
at oracle/jdbc/driver/T4CTTIfun.receive(T4CTTIfun.java:288)
at oracle/jdbc/driver/T4CTTIfun.doRPC(T4CTTIfun.java:191)
at oracle/jdbc/driver/T4C8Oall.doOALL(T4C8Oall.java:523)
at oracle/jdbc/driver/T4CStatement.doOall8(T4CStatement.java:193)
at oracle/jdbc/driver/T4CStatement.executeForRows(T4CStatement.java:998)
at oracle/jdbc/driver/OracleStatement.doExecuteWithTimeout(OracleStatement.java:1315)
at oracle/jdbc/driver/OracleStatement.executeInternal(OracleStatement.java:1890)
at oracle/jdbc/driver/OracleStatement.execute(OracleStatement.java:1855)
^-- Holding lock: oracle/jdbc/driver/T4CConnection@0xe4b45e08[thin lock]
at oracle/jdbc/driver/OracleStatementWrapper.execute(OracleStatementWrapper.java:304)
at weblogic/jdbc/wrapper/Statement.execute(Statement.java:466)
at org/apache/commons/dbcp/DelegatingStatement.execute(DelegatingStatement.java:264)
at com/demantra/applicationServer/services/DBServicesCommon.executeSelect(DBServicesCommon.java:1464)
at com/demantra/applicationServer/services/DBServices.executeSelect(DBServices.java:663)
at com/demantra/applicationServer/services/DBServicesCommon.createMaterializedViewAs(DBServicesCommon.java:1143)
at com/demantra/applicationServer/services/DBServices.createMaterializedViewAs(DBServices.java:429)
at com/demantra/applicationServer/services/queryrun/ExportDataHandler.exportData(ExportDataHandler.java:53)
at com/demantra/applicationServer/services/queryrun/ExportRunner.exportData(ExportRunner.java:62)
at com/demantra/applicationServer/metaDataObjects/transfer/profiles/QueryProfile.exportProfile(QueryProfile.java:1002)
^-- Holding lock: com/demantra/applicationServer/metaDataObjects/transfer/profiles/QueryProfile@0xe4aa5cd8[biased lock]
at com/demantra/applicationServer/metaDataObjects/transfer/profiles/IntegrationProfile.exportData(IntegrationProfile.java:331)
at com/demantra/applicationServer/metaDataObjects/transfer/Transfer.exportData(Transfer.java:370)
at com/demantra/workflow/step/TransferStep.doExecute(TransferStep.java:266)
at com/demantra/workflow/step/LinkedStep.execute(LinkedStep.java:914)
at com/demantra/workflow/process/WFProcess.mainLoop(WFProcess.java:1259)
at com/demantra/workflow/process/WFProcess.run(WFProcess.java:654)
at java/lang/Thread.run(Thread.java:662)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"WFProcess_1" id=40 idx=0xc8 tid=24439 prio=5 alive, sleeping, native_waiting, daemon
at java/lang/Thread.sleep(J)V(Native Method)
at com/demantra/workflow/process/WFProcess.checkStepEndConditions(WFProcess.java:1405)
at com/demantra/workflow/process/WFProcess.mainLoop(WFProcess.java:1284)
at com/demantra/workflow/process/WFProcess.run(WFProcess.java:654)
at java/lang/Thread.run(Thread.java:662)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"WFProcess_2" id=41 idx=0xcc tid=24440 prio=5 alive, sleeping, native_waiting, daemon
at java/lang/Thread.sleep(J)V(Native Method)
at com/demantra/workflow/process/WFProcess.checkStepEndConditions(WFProcess.java:1405)
at com/demantra/workflow/process/WFProcess.mainLoop(WFProcess.java:1284)
at com/demantra/workflow/process/WFProcess.run(WFProcess.java:654)
at java/lang/Thread.run(Thread.java:662)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"WFProcess_3" id=42 idx=0xd0 tid=24441 prio=5 alive, sleeping, native_waiting, daemon
at java/lang/Thread.sleep(J)V(Native Method)
at com/demantra/workflow/process/WFProcess.checkStepEndConditions(WFProcess.java:1405)
at com/demantra/workflow/process/WFProcess.mainLoop(WFProcess.java:1284)
at com/demantra/workflow/process/WFProcess.run(WFProcess.java:654)
at java/lang/Thread.run(Thread.java:662)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"WFProcess_4" id=43 idx=0xd4 tid=24442 prio=5 alive, sleeping, native_waiting, daemon
at java/lang/Thread.sleep(J)V(Native Method)
at com/demantra/workflow/process/WFProcess.checkStepEndConditions(WFProcess.java:1405)
at com/demantra/workflow/process/WFProcess.mainLoop(WFProcess.java:1284)
at com/demantra/workflow/process/WFProcess.run(WFProcess.java:654)
at java/lang/Thread.run(Thread.java:662)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"WFProcess_5" id=44 idx=0xd8 tid=24443 prio=5 alive, sleeping, native_waiting, daemon
at java/lang/Thread.sleep(J)V(Native Method)
at com/demantra/workflow/process/WFProcess.checkStepEndConditions(WFProcess.java:1405)
at com/demantra/workflow/process/WFProcess.mainLoop(WFProcess.java:1284)
at com/demantra/workflow/process/WFProcess.run(WFProcess.java:654)
at java/lang/Thread.run(Thread.java:662)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"WFProcess_6" id=45 idx=0xdc tid=24444 prio=5 alive, sleeping, native_waiting, daemon
at java/lang/Thread.sleep(J)V(Native Method)
at com/demantra/workflow/process/WFProcess.checkStepEndConditions(WFProcess.java:1405)
at com/demantra/workflow/process/WFProcess.mainLoop(WFProcess.java:1284)
at com/demantra/workflow/process/WFProcess.run(WFProcess.java:654)
at java/lang/Thread.run(Thread.java:662)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"WFProcess_7" id=46 idx=0xe0 tid=24445 prio=5 alive, sleeping, native_waiting, daemon
at java/lang/Thread.sleep(J)V(Native Method)
at com/demantra/workflow/process/WFProcess.checkStepEndConditions(WFProcess.java:1405)
at com/demantra/workflow/process/WFProcess.mainLoop(WFProcess.java:1284)
at com/demantra/workflow/process/WFProcess.run(WFProcess.java:654)
at java/lang/Thread.run(Thread.java:662)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"Query_Execution_86" id=256 idx=0xe4 tid=28986 prio=5 alive, parked, native_blocked, daemon
-- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0xe2f645d8
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2230)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at java/util/concurrent/locks/LockSupport.park(LockSupport.java:156)
at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java/lang/Thread.run(Thread.java:662)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"Query_Execution_80" id=250 idx=0xe8 tid=28980 prio=5 alive, parked, native_blocked, daemon
-- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0xe2f645d8
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2230)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at java/util/concurrent/locks/LockSupport.park(LockSupport.java:156)
at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java/lang/Thread.run(Thread.java:662)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"Query_Execution_85" id=255 idx=0xec tid=28985 prio=5 alive, parked, native_blocked, daemon
-- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0xe2f645d8
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2230)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at java/util/concurrent/locks/LockSupport.park(LockSupport.java:156)
at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java/lang/Thread.run(Thread.java:662)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"Query_Execution_84" id=254 idx=0xf0 tid=28984 prio=5 alive, parked, native_blocked, daemon
-- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0xe2f645d8
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2230)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at java/util/concurrent/locks/LockSupport.park(LockSupport.java:156)
at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java/lang/Thread.run(Thread.java:662)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"Query_Execution_87" id=257 idx=0xf4 tid=28987 prio=5 alive, parked, native_blocked, daemon
-- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0xe2f645d8
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2230)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at java/util/concurrent/locks/LockSupport.park(LockSupport.java:156)
at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java/lang/Thread.run(Thread.java:662)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"Query_Execution_81" id=251 idx=0xf8 tid=28981 prio=5 alive, parked, native_blocked, daemon
-- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0xe2f645d8
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2230)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at java/util/concurrent/locks/LockSupport.park(LockSupport.java:156)
at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java/lang/Thread.run(Thread.java:662)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"Query_Execution_83" id=253 idx=0xfc tid=28983 prio=5 alive, parked, native_blocked, daemon
-- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0xe2f645d8
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2230)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at java/util/concurrent/locks/LockSupport.park(LockSupport.java:156)
at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java/lang/Thread.run(Thread.java:662)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"WFProcess_8" id=54 idx=0x100 tid=24453 prio=5 alive, sleeping, native_waiting, daemon
at java/lang/Thread.sleep(J)V(Native Method)
at com/demantra/workflow/process/WFProcess.checkStepEndConditions(WFProcess.java:1405)
at com/demantra/workflow/process/WFProcess.mainLoop(WFProcess.java:1284)
at com/demantra/workflow/process/WFProcess.run(WFProcess.java:654)
at java/lang/Thread.run(Thread.java:662)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"Query_Execution_82" id=252 idx=0x104 tid=28982 prio=5 alive, parked, native_blocked, daemon
-- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0xe2f645d8
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2230)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at java/util/concurrent/locks/LockSupport.park(LockSupport.java:156)
at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java/lang/Thread.run(Thread.java:662)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"Query_Execution_79" id=249 idx=0x108 tid=28979 prio=5 alive, parked, native_blocked, daemon
-- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0xe2f645d8
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2230)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at java/util/concurrent/locks/LockSupport.park(LockSupport.java:156)
at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java/lang/Thread.run(Thread.java:662)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"Query_Execution_92" id=262 idx=0x10c tid=28992 prio=5 alive, parked, native_blocked, daemon
-- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0xe2f645d8
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2230)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at java/util/concurrent/locks/LockSupport.park(LockSupport.java:156)
at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java/lang/Thread.run(Thread.java:662)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"Thread-17" id=58 idx=0x110 tid=24457 prio=5 alive, parked, native_blocked
-- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0xe4a33f00
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2230)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at java/util/concurrent/locks/LockSupport.park(LockSupport.java:156)
at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at weblogic/utils/concurrent/JDK15ConcurrentBlockingQueue.take(JDK15ConcurrentBlockingQueue.java:89)
at weblogic/store/internal/PersistentStoreImpl.getOutstandingWork(PersistentStoreImpl.java:672)
at weblogic/store/internal/PersistentStoreImpl.run(PersistentStoreImpl.java:721)
at java/lang/Thread.run(Thread.java:662)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"DynamicListenThread[Default]" id=60 idx=0x114 tid=24458 prio=9 alive, in native, daemon
at java/net/PlainSocketImpl.socketAccept(Ljava/net/SocketImpl;)V(Native Method)
at java/net/PlainSocketImpl.accept(PlainSocketImpl.java:408)
^-- Holding lock: java/net/SocksSocketImpl@0xe4aec048[biased lock]
at java/net/ServerSocket.implAccept(ServerSocket.java:462)
at java/net/ServerSocket.accept(ServerSocket.java:430)
at weblogic/socket/WeblogicServerSocket.accept(WeblogicServerSocket.java:38)
at weblogic/server/channels/DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:535)
at weblogic/server/channels/DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:417)
at weblogic/server/channels/DynamicListenThread.run(DynamicListenThread.java:173)
at java/lang/Thread.run(Thread.java:662)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"[STANDBY] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'" id=61 idx=0x118 tid=24459 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/work/ExecuteThread@0xe4a52f68[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:162)
^-- Lock released while waiting: weblogic/work/ExecuteThread@0xe4a52f68[fat lock]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:183)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" id=62 idx=0x11c tid=24460 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/work/ExecuteThread@0xe4afa538[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:162)
^-- Lock released while waiting: weblogic/work/ExecuteThread@0xe4afa538[fat lock]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:183)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"[ACTIVE] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'" id=63 idx=0x120 tid=24461 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/work/ExecuteThread@0xe4a626a8[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:162)
^-- Lock released while waiting: weblogic/work/ExecuteThread@0xe4a626a8[fat lock]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:183)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"Query_Execution_90" id=260 idx=0x124 tid=28990 prio=5 alive, parked, native_blocked, daemon
-- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0xe2f645d8
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2230)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at java/util/concurrent/locks/LockSupport.park(LockSupport.java:156)
at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java/lang/Thread.run(Thread.java:662)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"Query_Execution_88" id=258 idx=0x128 tid=28988 prio=5 alive, parked, native_blocked, daemon
-- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0xe2f645d8
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2230)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at java/util/concurrent/locks/LockSupport.park(LockSupport.java:156)
at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java/lang/Thread.run(Thread.java:662)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"Query_Execution_89" id=259 idx=0x12c tid=28989 prio=5 alive, parked, native_blocked, daemon
-- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0xe2f645d8
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2230)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at java/util/concurrent/locks/LockSupport.park(LockSupport.java:156)
at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java/lang/Thread.run(Thread.java:662)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"Query_Execution_91" id=261 idx=0x130 tid=28991 prio=5 alive, parked, native_blocked, daemon
-- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0xe2f645d8
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2230)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at java/util/concurrent/locks/LockSupport.park(LockSupport.java:156)
at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java/lang/Thread.run(Thread.java:662)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"Query_Execution_260" id=585 idx=0x134 tid=29439 prio=5 alive, parked, native_blocked, daemon
-- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0xe2f645d8
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2230)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at java/util/concurrent/locks/LockSupport.park(LockSupport.java:156)
at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java/lang/Thread.run(Thread.java:662)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"Timer-6" id=69 idx=0x138 tid=26756 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: java/util/TaskQueue@0xe17170f0[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/util/TimerThread.mainLoop(Timer.java:509)
^-- Lock released while waiting: java/util/TaskQueue@0xe17170f0[fat lock]
at java/util/TimerThread.run(Timer.java:462)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"Event_Manager_67" id=923 idx=0x13c tid=29870 prio=5 alive, parked, native_blocked, daemon
-- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0xf4b91b40
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2230)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at java/util/concurrent/locks/LockSupport.park(LockSupport.java:156)
at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java/lang/Thread.run(Thread.java:662)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"Task_Loader_Thread" id=71 idx=0x140 tid=27086 prio=5 alive, sleeping, native_waiting, daemon
at java/lang/Thread.sleep(J)V(Native Method)
at com/demantra/common/task/TaskLoader.run(TaskLoader.java:133)
at java/lang/Thread.run(Thread.java:662)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"Query_Execution_268" id=608 idx=0x144 tid=29473 prio=5 alive, parked, native_blocked, daemon
-- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0xe2f645d8
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2230)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at java/util/concurrent/locks/LockSupport.park(LockSupport.java:156)
at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java/lang/Thread.run(Thread.java:662)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'" id=73 idx=0x148 tid=28139 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/work/ExecuteThread@0xe2ffb420[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:162)
^-- Lock released while waiting: weblogic/work/ExecuteThread@0xe2ffb420[fat lock]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:183)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"[ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'" id=74 idx=0x14c tid=28140 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/work/ExecuteThread@0xe2feb970[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:162)
^-- Lock released while waiting: weblogic/work/ExecuteThread@0xe2feb970[fat lock]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:183)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"[ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'" id=75 idx=0x150 tid=28141 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/work/ExecuteThread@0xe300ab10[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:162)
^-- Lock released while waiting: weblogic/work/ExecuteThread@0xe300ab10[fat lock]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:183)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"[ACTIVE] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'" id=76 idx=0x154 tid=28142 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/work/ExecuteThread@0xe2f55dc0[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:162)
^-- Lock released while waiting: weblogic/work/ExecuteThread@0xe2f55dc0[fat lock]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:183)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"[ACTIVE] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)'" id=77 idx=0x158 tid=28143 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/work/ExecuteThread@0xe2f63970[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:162)
^-- Lock released while waiting: weblogic/work/ExecuteThread@0xe2f63970[fat lock]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:183)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"[ACTIVE] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'" id=78 idx=0x15c tid=28144 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/work/ExecuteThread@0xe2f73640[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:162)
^-- Lock released while waiting: weblogic/work/ExecuteThread@0xe2f73640[fat lock]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:183)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"[ACTIVE] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)'" id=79 idx=0x160 tid=28145 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/work/ExecuteThread@0xe2fec498[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:162)
^-- Lock released while waiting: weblogic/work/ExecuteThread@0xe2fec498[fat lock]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:183)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"[ACTIVE] ExecuteThread: '12' for queue: 'weblogic.kernel.Default (self-tuning)'" id=80 idx=0x164 tid=28146 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/work/ExecuteThread@0xe2f82d20[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:162)
^-- Lock released while waiting: weblogic/work/ExecuteThread@0xe2f82d20[fat lock]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:183)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"[ACTIVE] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)'" id=81 idx=0x168 tid=28147 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/work/ExecuteThread@0xe2fa0458[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:162)
^-- Lock released while waiting: weblogic/work/ExecuteThread@0xe2fa0458[fat lock]Check -Xmx and -Xms start parameters for java in opmn.xml for the container you are workingk, as well in the Console you can change this in the Properties for the container.
Greetings -
Hi All,
I am facing the below error . Can anyone help me in resolving this
Here are some more logs files related to stuck
Please let me know from which application is coming this error.
<Dec 17, 2011 4:12:55 AM CST> <Notice> <Stdout> <BEA-000000> <Inside flagshelf if condn.....>
<Dec 17, 2011 4:12:55 AM CST> <Notice> <Stdout> <BEA-000000> <<Dec 17, 2011 4:12:55 AM CST> <Error> <oracle.adf.controller.internal.binding.DCTaskFlowBinding> <BEA-000000> <oracle.adfinternal.view.faces.model.binding.FacesCtrlAttrsBinding>>
<Dec 17, 2011 4:12:55 AM CST> <Notice> <Stdout> <BEA-000000> <<Dec 17, 2011 4:12:55 AM CST> <Error> <oracle.adf.controller.internal.binding.DCTaskFlowBinding> <BEA-000000> <oracle.adfinternal.view.faces.model.binding.FacesCtrlAttrsBinding>>
<Dec 17, 2011 4:12:55 AM CST> <Notice> <Stdout> <BEA-000000> <Object Id :8d6c8f55-c841-4394-88a7-2b8a2118fe9c:DEVICE>
<Dec 17, 2011 4:12:55 AM CST> <Notice> <StdErr> <BEA-000000> <com.avocent.trellis.safetynet.TrellisException: Generic CDMR platform exception>
<Dec 17, 2011 4:12:55 AM CST> <Notice> <StdErr> <BEA-000000> <at com.avocent.trellis.apps.common.ejb.CapacityServiceEJBBean.getCapacityByTypeUOM(CapacityServiceEJBBean.java:490)>
<Dec 17, 2011 4:12:55 AM CST> <Notice> <StdErr> <BEA-000000> <at oracle.adf.model.binding.DCIteratorBinding.getCurrentRow(DCIteratorBinding.java:2201)>
<Dec 17, 2011 4:12:55 AM CST> <Notice> <StdErr> <BEA-000000> <at oracle.adf.model.binding.DCControlBinding.getCurrentRow(DCControlBinding.java:387)>
<Dec 17, 2011 4:13:03 AM CST> <Notice> <Stdout> <BEA-000000> <bye array image is :[B@1e802319>
<Dec 17, 2011 4:13:04 AM CST> <Warning> <Socket> <WL-000449> <Closing socket as no data read from it on 10.172.6.11:56,906 during the configured idle timeout of 5 secs>
<Dec 17, 2011 4:13:05 AM CST> <Notice> <Stdout> <BEA-000000> <Init method on shelf selected value from EJB -- null>
<Dec 17, 2011 4:13:05 AM CST> <Notice> <Stdout> <BEA-000000> <onShelfSelected false>
<Dec 17, 2011 4:13:05 AM CST> <Notice> <Stdout> <BEA-000000> <Inside flagshelf if condn.....>
<Dec 17, 2011 4:13:05 AM CST> <Notice> <Stdout> <BEA-000000> <Delete Device>
<Dec 17, 2011 4:13:05 AM CST> <Notice> <Stdout> <BEA-000000> <richPopup.getClientId(context)pt1:r3:0:pc1:p5>
<Dec 17, 2011 4:13:05 AM CST> <Notice> <Stdout> <BEA-000000> <deletion Completed in ManagedBean>
<Dec 17, 2011 4:13:06 AM CST> <Notice> <Stdout> <BEA-000000> <Node type:DEVICE>
<Dec 17, 2011 4:13:06 AM CST> <Notice> <Stdout> <BEA-000000> <Floor GridId:aab83e10-149f-4bc7-a578-075ac28cb255>
<Dec 17, 2011 4:13:06 AM CST> <Notice> <Stdout> <BEA-000000> <placement name is null>
<Dec 17, 2011 4:13:06 AM CST> <Notice> <Stdout> <BEA-000000> <Init method on shelf selected value from EJB -- null>
<Dec 17, 2011 4:13:06 AM CST> <Notice> <Stdout> <BEA-000000> <onShelfSelected false>
<Dec 17, 2011 4:13:06 AM CST> <Notice> <Stdout> <BEA-000000> <Inside flagshelf if condn.....>
<Dec 17, 2011 4:13:07 AM CST> <Notice> <Stdout> <BEA-000000> <Init method on shelf selected value from EJB -- null>
<Dec 17, 2011 4:13:07 AM CST> <Notice> <Stdout> <BEA-000000> <onShelfSelected false>
<Dec 17, 2011 4:13:07 AM CST> <Notice> <Stdout> <BEA-000000> <Inside flagshelf if condn.....>
<Dec 17, 2011 4:13:08 AM CST> <Error> <WebLogicServer> <WL-000337> <[STUCK] ExecuteThread: '12' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "709" seconds working on the request "weblogic.servlet.internal.ServletRequestImpl@1402a3c3[
POST /TrellisConsole-CpsConsole-context-root/faces/InventoryPortfolioTFT/InventoryPortfolio?_adf.ctrl-state=z1imd3ex8_12 HTTP/1.1
Connection: keep-alive
Content-Length: 12
Origin: http://10.207.120.120:7011
Adf-Page-Unregistered-Id: 2
Adf-Window-Unregistered: true
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.121 Safari/535.2
Content-Type: application/xml
Adf-Rich-Message: true
Adf-Window-Unregistered-Id: z1imd3ex8_1
Accept: */*
<Dec 17, 2011 4:12:08 AM CST> <Error> <WebLogicServer> <WL-000337> <[STUCK] ExecuteThread: '12' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "648" seconds working on the request "weblogic.servlet.internal.ServletRequestImpl@1402a3c3[
POST /TrellisConsole-CpsConsole-context-root/faces/InventoryPortfolioTFT/InventoryPortfolio?_adf.ctrl-state=z1imd3ex8_12 HTTP/1.1
Connection: keep-alive
Content-Length: 12
Origin: http://10.207.120.120:7011
Adf-Page-Unregistered-Id: 2
Adf-Window-Unregistered: true
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.121 Safari/535.2
Content-Type: application/xml
Adf-Rich-Message: true
Adf-Window-Unregistered-Id: z1imd3ex8_1
Accept: */*
Referer: http://10.207.120.120:7011/TrellisConsole-CpsConsole-context-root/faces/InventoryPortfolioTFT/InventoryPortfolio?_adf.ctrl-state=z1imd3ex8_12
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Cookie: JSESSIONID=1Jr3TsypG1p8HXsQvZwM8vQwB8Td2Pv2D1sZg1zhr0vHfMbnvfLr!1713687512
]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-110 "[STUCK] ExecuteThread: '12' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, suspended, waiting, priority=1, DAEMON> {
-- Waiting for notification on: oracle.adfinternal.controller.util.RequestLock@1db2322d[fat lock]
java.lang.Object.wait(Object.java:???)
java.lang.Object.wait(Object.java:485)
oracle.adfinternal.controller.util.RequestLock.lock(RequestLock.java:36)
oracle.adfinternal.controller.state.RootViewPortContextImpl.lockViewPortRequestLock(RootViewPortContextImpl.java:589)
oracle.adfinternal.controller.state.ControllerState.initializeRequest(ControllerState.java:766)
oracle.adfinternal.controller.state.ControllerState.initializeRequest(ControllerState.java:754)
oracle.adfinternal.controller.application.AdfcConfigurator.beginRequest(AdfcConfigurator.java:38)
org.apache.myfaces.trinidadinternal.config.GlobalConfiguratorImpl._startConfiguratorServiceRequest(GlobalConfiguratorImpl.java:552)
org.apache.myfaces.trinidadinternal.config.GlobalConfiguratorImpl.beginRequest(GlobalConfiguratorImpl.java:185)
org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl.doFilter(TrinidadFilterImpl.java:133)
org.apache.myfaces.trinidad.webapp.TrinidadFilter.doFilter(TrinidadFilter.java:91)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.adf.library.webapp.LibraryFilter.doFilter(LibraryFilter.java:163)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:111)
oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:299)
>
<Dec 17, 2011 4:12:08 AM CST> <Notice> <Diagnostics> <WL-320068> <Watch 'StuckThread' with severity 'Notice' on server 'adf_server1' has triggered at Dec 17, 2011 4:12:08 AM CST. Notification details:
WatchRuleType: Log
WatchRule: (SEVERITY = 'Error') AND ((MSGID = 'WL-000337') OR (MSGID = 'BEA-000337'))
WatchData: DATE = Dec 17, 2011 4:12:08 AM CST SERVER = adf_server1 MESSAGE = [STUCK] ExecuteThread: '12' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "648" seconds working on the request "weblogic.servlet.internal.ServletRequestImpl@1402a3c3[
POST /TrellisConsole-CpsConsole-context-root/faces/InventoryPortfolioTFT/InventoryPortfolio?_adf.ctrl-state=z1imd3ex8_12 HTTP/1.1
Connection: keep-alive
Content-Length: 12
Origin: http://10.207.120.120:7011
Adf-Page-Unregistered-Id: 2
Adf-Window-Unregistered: true
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.121 Safari/535.2
Content-Type: application/xml
Adf-Rich-Message: true
Adf-Window-Unregistered-Id: z1imd3ex8_1
Accept: */*
Referer: http://10.207.120.120:7011/TrellisConsole-CpsConsole-context-root/faces/InventoryPortfolioTFT/InventoryPortfolio?_adf.ctrl-state=z1imd3ex8_12
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Cookie: JSESSIONID=1Jr3TsypG1p8HXsQvZwM8vQwB8Td2Pv2D1sZg1zhr0vHfMbnvfLr!1713687512
]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-110 "[STUCK] ExecuteThread: '12' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, suspended, waiting, priority=1, DAEMON> {
-- Waiting for notification on: oracle.adfinternal.controller.util.RequestLock@1db2322d[fat lock]
java.lang.Object.wait(Object.java:???)Thanks Rene...although I have some stuck thread issues occasionally this is the first one I have seen with this particular error. So I am going to treat it as a one off for now and if it pops up again then I will review these suggestions. Unfortunately I have not seen this error in my 2 test environments (they aren't hit nearly as hard as my prod environments which is normal I am sure).
My complaint on the WL stuck threads is that they never seem to clear the console unless you restart the managed server. I don't know if I should assume that whatever caused them is still an issue or if WL just does a poor job on updating the cleanup of the error. I have had no user complaints though....
Thanks for the assistance and direction! -
Stuck thread in Oracle Weblogic server
Dear All,
We have a stuck thread in Oracle WLS 10.3.3. We have deployed 7 ear files in 5th managed server(psMS5). We have stuck thread in one of the application (psinsurance). We have allocated 1GB heap space for the psMS5 managed server. We are using this in Development environment. Please check the below message from psMS5 managed server log file. Kindly provide reason for stuck thread and solution.
WatchTime: Sep 23, 2011 10:28:29 AM IST
WatchDomainName: dev_domain
WatchServerName: psMS5
WatchSeverityLevel: Warning
WatchName: MS5-Watch
WatchRuleType: Log
WatchRule: ((MESSAGE LIKE '%has been busy for "%') AND (MSGID = 'BEA-000337') AND (SERVER = 'psMS5'))
WatchData: DATE = Sep 23, 2011 10:28:28 AM IST SERVER = psMS5 MESSAGE = [STUCK] ExecuteThread: '22' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "629" seconds working on the request "weblogic.servlet.internal.ServletRequestImpl@1cac6525[
POST */psinsurance/app/lending/insurance/PS00738D.jspx*?_adf.ctrl-state=24f0li74e_4 HTTP/1.1
Accept: image/jpeg, application/x-ms-application, image/gif, application/xaml+xml, image/pjpeg, application/x-ms-xbap, */*
Referer: http://sis-prosun11/psinsurance/faces/PS00003?jsessionid=r2BrT8TpqZBsnLhCw2Y8KDpTV7hMpgnLDFW54P87ZpHwn8JMy88s%21168932459%211316753022074&_afrWindowMode=0&_afrLoop=244724092027000&_adf.ctrl-state=24f0li74e_4
Accept-Language: en-US
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; Tablet PC 2.0)
Content-Type: application/x-www-form-urlencoded
Accept-Encoding: gzip, deflate
Content-Length: 554
Cache-Control: no-cache
Cookie: JSESSIONID=LkJcT8TfMT64SyQJT4Ls4sXhfDl0pTS7CXJVQ9KcryHGWJ1dhyfz!1373322659
Connection: Keep-Alive
Proxy-Client-IP: 192.168.9.179
X-Forwarded-For: 192.168.9.179
X-WebLogic-KeepAliveSecs: 30
X-WebLogic-Force-JVMID: 1373322659
]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-237 "[STUCK] ExecuteThread: '22' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, suspended, priority=1, DAEMON> {
java.io.UnixFileSystem.getBooleanAttributes0(UnixFileSystem.java:???)
java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:228)
java.io.File.exists(File.java:729)
weblogic.utils.classloaders.ClasspathClassFinder.getFileSource(ClasspathClassFinder.java:383)
weblogic.utils.classloaders.ClasspathClassFinder.getSourcesInternal(ClasspathClassFinder.java:333)
weblogic.utils.classloaders.ClasspathClassFinder.getSource(ClasspathClassFinder.java:316)
weblogic.utils.classloaders.JarClassFinder.getSource(JarClassFinder.java:58)
weblogic.utils.classloaders.AbstractClassFinder.getClassSource(AbstractClassFinder.java:22)
weblogic.utils.classloaders.MultiClassFinder.getClassSource(MultiClassFinder.java:56)
weblogic.utils.classloaders.CodeGenClassFinder.getClassSource(CodeGenClassFinder.java:26)
weblogic.utils.classloaders.GenericClassLoader.findLocalClass(GenericClassLoader.java:268)
*^-- Holding lock: weblogic.utils.classloaders.GenericClassLoader@1e3295a1[thin lock]*
weblogic.utils.classloaders.GenericClassLoader.findClass(GenericClassLoader.java:237)
java.lang.ClassLoader.loadClass(ClassLoader.java:292)
*^-- Holding lock: weblogic.utils.classloaders.GenericClassLoader@1e3295a1[thin lock]*
java.lang.ClassLoader.loadClass(ClassLoader.java:292)
java.lang.ClassLoader.loadClass(ClassLoader.java:248)
weblogic.utils.classloaders.GenericClassLoader.loadClass(GenericClassLoader.java:177)
weblogic.utils.classloaders.ChangeAwareClassLoader.loadClass(ChangeAwareClassLoader.java:37)
*^-- Holding lock: weblogic.utils.classloaders.ChangeAwareClassLoader@1e553474[thin lock]*
java.lang.Class.getDeclaredConstructors0(Class.java:???)
java.lang.Class.privateGetDeclaredConstructors(Class.java:2370)
java.lang.Class.getConstructor0(Class.java:2699)
java.lang.Class.newInstance0(Class.java:318)
java.lang.Class.newInstance(Class.java:305)
org.apache.myfaces.trinidadinternal.renderkit.ClassRendererInstantiator.instantiate(ClassRendererInstantiator.java:48)
org.apache.myfaces.trinidadinternal.renderkit.RenderKitBase.findRenderer(RenderKitBase.java:146)
org.apache.myfaces.trinidadinternal.renderkit.RenderKitDecorator.findRenderer(RenderKitDecorator.java:84)
oracle.adfinternal.view.faces.unified.renderkit.UnifiedRenderKit._getRenderer(UnifiedRenderKit.java:218)
oracle.adfinternal.view.faces.unified.renderkit.UnifiedRenderKit.findRenderer(UnifiedRenderKit.java:156)
org.apache.myfaces.trinidadinternal.renderkit.RenderKitBase.getRenderer(RenderKitBase.java:129)
org.apache.myfaces.trinidad.component.UIXComponentBase._getRendererImpl(UIXComponentBase.java:1143)
org.apache.myfaces.trinidad.component.UIXComponentBase.getRenderer(UIXComponentBase.java:1173)
org.apache.myfaces.trinidad.component.UIXComponentBase.getClientId(UIXComponentBase.java:316)
javax.faces.webapp.UIComponentClassicTagBase.doStartTag(UIComponentClassicTagBase.java:1095)
org.apache.myfaces.trinidad.webapp.UIXComponentELTag.doStartTag(UIXComponentELTag.java:70)
oracle.adfinternal.view.faces.unified.taglib.input.UnifiedSelectBooleanCheckboxTag.doStartTag(UnifiedSelectBooleanCheckboxTag.java:45)
oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
oracle.jsp.runtime.tree.OracleJspIterationTagNode.executeHandler(OracleJspIterationTagNode.java:41)
oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
oracle.jsp.runtime.tree.OracleJspIterationTagNode.executeHandler(OracleJspIterationTagNode.java:41)
oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
oracle.jsp.runtime.tree.OracleJspNode.execute(OracleJspNode.java:73)
oracle.jsp.runtimev2.ShortCutServlet._jspService(ShortCutServlet.java:57)
oracle.jsp.runtime.OracleJspBase.service(OracleJspBase.java:29)
oracle.jsp.runtimev2.JspPageTable.compileAndServe(JspPageTable.java:650)
oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.java:330)
oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.java:680)
oracle.jsp.runtimev2.JspServlet.service(JspServlet.java:653)
javax.servlet.http.HttpServlet.service(HttpServlet.java:815)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:206)
weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.adf.library.webapp.LibraryFilter.doFilter(LibraryFilter.java:147)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
weblogic.servlet.internal.RequestDispatcherImpl.invokeServlet(RequestDispatcherImpl.java:520)
weblogic.servlet.internal.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:112)
com.sun.faces.context.ExternalContextImpl.dispatch(ExternalContextImpl.java:402)
org.apache.myfaces.trinidad.context.ExternalContextDecorator.dispatch(ExternalContextDecorator.java:44)
org.apache.myfaces.trinidad.context.ExternalContextDecorator.dispatch(ExternalContextDecorator.java:44)
org.apache.myfaces.trinidad.context.ExternalContextDecorator.dispatch(ExternalContextDecorator.java:44)
org.apache.myfaces.trinidad.context.ExternalContextDecorator.dispatch(ExternalContextDecorator.java:44)
org.apache.myfaces.trinidadinternal.context.FacesContextFactoryImpl$OverrideDispatch.dispatch(FacesContextFactoryImpl.java:261)
com.sun.faces.application.ViewHandlerImpl.executePageToBuildView(ViewHandlerImpl.java:428)
com.sun.faces.application.ViewHandlerImpl.renderView(ViewHandlerImpl.java:132)
javax.faces.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:189)
org.apache.myfaces.trinidadinternal.application.ViewHandlerImpl.renderView(ViewHandlerImpl.java:155)
oracle.adfinternal.view.faces.lifecycle.LifecycleImpl._renderResponse(LifecycleImpl.java:706)
oracle.adfinternal.view.faces.lifecycle.LifecycleImpl._executePhase(LifecycleImpl.java:244)
oracle.adfinternal.view.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:199)
javax.faces.webapp.FacesServlet.service(FacesServlet.java:246)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:206)
weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.adf.model.servlet.ADFBindingFilter.doFilter(ADFBindingFilter.java:134)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.adfinternal.view.faces.webapp.rich.RegistrationFilter.doFilter(RegistrationFilter.java:69)
org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:411)
oracle.adfinternal.view.faces.activedata.AdsFilter.doFilter(AdsFilter.java:44)
org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:411)
org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl._doFilterImpl(TrinidadFilterImpl.java:210)
org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl.doFilter(TrinidadFilterImpl.java:126)
org.apache.myfaces.trinidad.webapp.TrinidadFilter.doFilter(TrinidadFilter.java:91)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:94)
oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:299)
oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:406)
oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:82)
oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:70)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.adf.library.webapp.LibraryFilter.doFilter(LibraryFilter.java:147)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.dms.wls.DMSServletFilter.doFilter(DMSServletFilter.java:114)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:25)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.doIt(WebAppServletContext.java:3656)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3650)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:308)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:117)
weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2204)
weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2126)
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1412)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:198)
weblogic.work.ExecuteThread.run(ExecuteThread.java:165)
SUBSYSTEM = WebLogicServer USERID = <WLS Kernel> SEVERITY = Error THREAD = [ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)' MSGID = BEA-000337 MACHINE = sis-prosun11.sis.ad TXID = CONTEXTID = 11d1def534ea1be0:3bbe257e:13286031cbf:-7ffd-0000000000002fe7 TIMESTAMP = 1316753908672
WatchAlarmType: AutomaticReset
WatchAlarmResetPeriod: 60000
SMTPNotificationName: MS5-Health-NotificationI found something interesting to the question I posted from the below link:
http://www.mail-archive.com/[email protected]/msg03217.html
Maybe you are looking for
-
Apple Mobile Device failed to start. What do I do?
When up grading iTunes I get the message "Apple Mobile Device failed to start." Verify you have sufficient privileges to start system services. I own this computer and am the only user. How do I install the new version of iTunes?
-
Add a Code to Change the ABAP Query
Hi All, I have a requirement in which i need to add a check box to a selection screen in ABAP query i inserted this check box in the infoset .The report is using joins from table EKKO,EKPO and EKES and display all the Vendor Confirmation details but
-
How can I make apps self-minimize upon Launch?
I created an AppleScript to Launch several programs at once. I would like several of the apps to self-minimize to the dock when they launch, leaving just one app with an open window. Can anyone help me learn how to do this? I am running Mavericks OS
-
Some of my album art images are appearing when i open up my images. can i delete them from images or will it delete them from album art too
-
We've encountered the following issues: Adobe Application Manager failed to install.
When I attmept to open / load Adobe Application Manager I get the following error message: We've encountered the following issues: Adobe Application Manager failed to install. OS: Windows XP Professional Any Help would be appreciated.