Stuck Thread attribute on WLS8.1 sp2

There is long process that requires a DB connection to be open for 2-3 hours. But currently we have the Stuck Thread Time Out set to '7200' seconds. Will Weblogic kill a thread if that thread is designated as Stuck??? (the backend is a Sybase server we had no issue with the DB at the point)
Given below is an excerpt from the server.log file
<Sep 22, 2004 5:43:27 PM EDT> <Warning> <WebLogicServer> <BEA-000337> <ExecuteTh
read: '22' for queue: 'weblogic.kernel.Default' has been busy for "8,290" second
s working on the request "connection15.session24", which is more than the config
ured time (StuckThreadMaxTime) of "7,200" seconds.>
<Sep 22, 2004 7:43:27 PM EDT> <Warning> <WebLogicServer> <BEA-000337> <ExecuteTh
read: '22' for queue: 'weblogic.kernel.Default' has been busy for "15,490" second
s working on the request "connection15.session24", which is more than the config
ured time (StuckThreadMaxTime) of "7,200" seconds.>
####<Sep 22, 2004 9:13:19 PM EDT> <Error> <JDBC> <HOSTNAME> <SERVERNAME
<ExecuteThread: '22' for queue: 'weblogic.kernel.Default'> <<anonymous>> <> <BEA-001112> <Test "SELECT COUNT(*) FROM sysobjects" set up for pool "CRS_TDS_CONN
_POOL" failed with exception: "java.sql.SQLException: JZ0C0: Connection is alrea
dy closed.".>

Arvind Varma wrote:
There is long process that requires a DB connection to be open for 2-3 hours.> But currently we have the Stuck Thread Time Out set to '7200' seconds. Will
> Weblogic kill a thread if that thread is designated as Stuck??? (the backend
> is a Sybase server we had no issue with the DB at the point)
No, it's just a warning, I believe. There should be no problem with
jdbc connections staying open indefinitely, unless there's a firewall
that kills sockets after a time, or the DBMS kills idle connections
after a time. From the log below, it sounds like the pool just tested
a connection before giving it to your app, and found it was dead, and
presumably replaced it and gave you a new one. Was there any appplication
error?
Joe
PS: show me your pool definition. Your test table should not be sysobjects.
That is a huge system table. That's like testing your phone by calling 411
and asking the operator to count all the listings in the phone book for you! ;)
Joe
Given below is an excerpt from the server.log file
<Sep 22, 2004 5:43:27 PM EDT> <Warning> <WebLogicServer> <BEA-000337> <ExecuteTh
read: '22' for queue: 'weblogic.kernel.Default' has been busy for "8,290" second
s working on the request "connection15.session24", which is more than the config
ured time (StuckThreadMaxTime) of "7,200" seconds.>
<Sep 22, 2004 7:43:27 PM EDT> <Warning> <WebLogicServer> <BEA-000337> <ExecuteTh
read: '22' for queue: 'weblogic.kernel.Default' has been busy for "15,490" second
s working on the request "connection15.session24", which is more than the config
ured time (StuckThreadMaxTime) of "7,200" seconds.>
####<Sep 22, 2004 9:13:19 PM EDT> <Error> <JDBC> <HOSTNAME> <SERVERNAME
<ExecuteThread: '22' for queue: 'weblogic.kernel.Default'> <<anonymous>> <> <BEA-001112> <Test "SELECT COUNT(*) FROM sysobjects" set up for pool "CRS_TDS_CONN
_POOL" failed with exception: "java.sql.SQLException: JZ0C0: Connection is alrea
dy closed.".>

Similar Messages

  • Weblogic 8.14: Stuck Threads during JMS - IO native write operations

    G'day!
    Hope somebody can advise a solution on the following...
    Recently message loads have started to increase on our weblogic 8.14 cluster (3 servers) and we started to see an almost daily occurrence of Stuck Thread issues. From thread dumps we could see these were caused by jrockit/net/SocketNativeIO.write processes. We concluded we had reached the max performance limit for the file based JMS stores and so we moved these to database based JMS stores. This certainly seemed to improve matters - but we have recently seen the issue return. Which is puzzling, as technically we thought all messages would be persisted to the database, and any file IO on the application server would be eliminated. Our database resides on a separate dedicated oracle server. We tracked a recent cause to a large message (200K) which caused the Struck Thread - resulting in a build up of pending messages. After bouncing the server the message processed ok. Any thoughts on how to overcome these issues - greatly appreciated. Here's some detail on the stuck thread from the thread dump: (+Why does weblogic use IO when the JMS store is on a database?+). All messages are persistant.
    "ExecuteThread: '6' for queue: 'weblogic.kernel.Default'" prio=5 id=0xc00 pid=27544 active, daemon
    at jrockit/net/SocketNativeIO.write(Native Method)@0xf3aeae10
    at jrockit/net/SocketNativeIO.write(Unknown Source)@0xf3aeae9c
    at jrockit/io/NativeIO.write(Unknown Source)@0xf3ae62c6
    at java/net/AbstractSocketImpl$2.write(Unknown Source)@0xf3aeada4
    at jrockit/io/NativeIOOutputStream.write(Unknown Source)@0xf3ae61ec
    at jrockit/io/NativeIOOutputStream.write(Unknown Source)@0xf3ae618d
    at java/io/DataOutputStream.write(Unknown Source)@0xf4e3ec25
    ^-- Holding lock: java/io/DataOutputStream@0x4309fc80[thin lock]
    at com/ibm/mq/MQInternalCommunications.send(MQInternalCommunications.java:1022)@0xed34364a
    ^-- Holding lock: com/ibm/mq/MQInternalCommunications@0x4309f7e0[thin lock]
    at com/ibm/mq/MQSESSIONClient.lowLevelComms(MQSESSIONClient.java:2832)@0xed342fbd
    ^-- Holding lock: java/lang/Integer@0x4309f8a0[thin lock]
    at com/ibm/mq/MQSESSIONClient.MQPUT(MQSESSIONClient.java:3844)@0xed34271b
    at com/ibm/mq/MQQueue.putMsg2(MQQueue.java:1486)@0xed342262
    ^-- Holding lock: com/ibm/mq/MQSPIQueue@0x4312ce80[thin lock]
    at com/ibm/mq/jms/MQMessageProducer.sendInternal(MQMessageProducer.java:1569)@0xed33d1ab
    at com/ibm/mq/jms/MQMessageProducer.send(MQMessageProducer.java:1012)@0xed860a49
    at com/ibm/mq/jms/MQMessageProducer.send(MQMessageProducer.java:1046)@0xed8605f4
    at weblogic/jms/adapter/JMSBaseConnection.sendInternal(JMSBaseConnection.java:667)@0xef7df0bf
    ^-- Holding lock: weblogic/jms/adapter/JMSBaseConnection@0x43093748[thin lock]
    at weblogic/jms/adapter/JMSBaseConnection.access$200(JMSBaseConnection.java:80)@0xef7dee28
    at weblogic/jms/adapter/JMSBaseConnection$6.run(JMSBaseConnection.java:647)@0xef7dee1d
    at weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)@0xf2fcbb88
    at weblogic/security/service/SecurityManager.runAs(SecurityManager.java:147)@0xf2fcbb0a
    at weblogic/jms/adapter/JMSBaseConnection.send(JMSBaseConnection.java:644)@0xef7decf3
    at weblogic/jms/adapter/JMSConnectionHandle.send(JMSConnectionHandle.java:144)@0xef7dec7c
    at jrockit/reflect/NativeMethodInvoker.invoke0(Native Method)@0xf4f10d10
    at jrockit/reflect/NativeMethodInvoker.invoke(Unknown Source)@0xf4f10e98
    at jrockit/reflect/VirtualNativeMethodInvoker.invoke(Unknown Source)@0xf4e86d70
    at java/lang/reflect/Method.invoke(Unknown Source)@0xf4f104c3
    at weblogic/connector/common/internal/ConnectionWrapper.invoke(ConnectionWrapper.java:149)@0xed96a498
    at $Proxy14.send(Unknown Source)@0xef7dec04
    at weblogic/jms/bridge/internal/MessagingBridge.onMessageInternal(MessagingBridge.java:1258)@0xed857b25
    at weblogic/jms/bridge/internal/MessagingBridge.onMessage(MessagingBridge.java:1181)@0xed857864
    at weblogic/jms/adapter/JMSBaseConnection$27.run(JMSBaseConnection.java:1943)@0xef7de623
    at weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)@0xf2fcbb88
    at weblogic/security/service/SecurityManager.runAs(SecurityManager.java:147)@0xf2fcbb0a
    at weblogic/jms/adapter/JMSBaseConnection.onMessage(JMSBaseConnection.java:1939)@0xef7de586
    at weblogic/jms/client/JMSSession.onMessage(JMSSession.java:2678)@0xeee7e30c
    at weblogic/jms/client/JMSSession.execute(JMSSession.java:2598)@0xeee7d25d
    at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:219)@0xf4e14070
    at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:178)@0xf4e1fddb
    at java/lang/Thread.startThreadFromVM(Unknown Source)@0xf4f3c9f3
    Thanks
    C.

    I'm not sure what operating system you're running, but I only have experience of WLS8.1 on Windows.
    We did extensive load testing of JMS before we went live, and found JDBC file stores to be rubbish in comparison to file store. When a message is persisted, it has to be written to the database, then when it is consumed, it must be deleted from the table. This simply killed our system when we load tested it.
    One thing we did do though, was use a second JMS server and second file store for one particularly heavy queue, where we did have IO issues. This was with a JMS distributed destination. We also use a synchronous write policy of cache-flush on the file store and we also have configured a paging store to page messages so the app server does not run out of memory.
    200K isn't by any means a huge message though, and shouldn't be any problem at all, we have 1mb messages quite regularly.
    I notice from the stack trace, that you have MQ series aswell. I'm not experienced with MQ, so can't give any information, but what I would try is the following.
    Using a JMS server with JMS File storage, develop a small java client to write as many messages to the queue and effectively stress the JMS subsystem just to establish if the filestore is the problem. This should be done without a connection to MQ and also use large messages. if you don't get any issues and WebLogic JMS with file store looks ok, then attach a consumer to the queue, which should also be a simple java client. Then you have production and consumption going on, which will result in writes and read's/deletes from the jms file store.
    Once you've stress tested JMS and you are satisfied that its ok or not, then add the MQseries connection and see how that affects things.
    You may well have already done all this, so please feel free to ignore me! I did a load of this kind of stuff before we went live to try and bash JMS to bits, but I didn't manage it! The only thing we did was add a second file store and second JMS server.
    I just looked back in my support cases, and found we'd had a similar issue - disk fragmentation seemed to be one of the factors here too. The JMS file store was heavily fragmented. That's windows for you though!!!
    Hope that helps.
    Pete

  • Stuck Threads on weblogic 10.3.2 Holding lock HttpURLConnection

    We are having issue with STUCK THREADS on weblogic 10.3.2. Basically my application makes a lot of web service calls over HTTPS and once in a while I see few STUCK THREADS and most of the time they clear but sometimes I have to bounce my server to clean.
    I have a thread dump, according to this its holding lock at weblogic.net.http.HttpURLConnection.getInputStream
    Any help to resolve this problem is highly appreciated.
    ThreadDump:
    <May 24, 2011 9:59:08 AM EDT> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "604" seconds working on the request "weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl@b607d75", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
    Thread-75 "[STUCK] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, priority=1, DAEMON> {
    jrockit.net.SocketNativeIO.readBytesPinned(SocketNativeIO.java:???)
    jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:31)
    java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)
    java.net.SocketInputStream.read(SocketInputStream.java:107)
    weblogic.utils.io.ChunkedInputStream.read(ChunkedInputStream.java:149)
    java.io.InputStream.read(InputStream.java:85)
    com.certicom.tls.record.ReadHandler.readFragment(Unknown Source)
    com.certicom.tls.record.ReadHandler.readRecord(Unknown Source)
    com.certicom.tls.record.ReadHandler.read(Unknown Source)
    ^-- Holding lock: com.certicom.tls.record.ReadHandler@bec563c[thin lock]
    com.certicom.io.InputSSLIOStreamWrapper.read(Unknown Source)
    java.io.BufferedInputStream.fill(BufferedInputStream.java:189)
    java.io.BufferedInputStream.read(BufferedInputStream.java:236)
    ^-- Holding lock: java.io.BufferedInputStream@bec5d9f[thin lock]
    weblogic.net.http.MessageHeader.isHTTP(MessageHeader.java:214)
    weblogic.net.http.MessageHeader.parseHeader(MessageHeader.java:141)
    weblogic.net.http.HttpClient.parseHTTP(HttpClient.java:452)
    ^-- Holding lock: weblogic.net.http.HttpsClient@bec5202[thin lock]
    weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:328)
    weblogic.net.http.SOAPHttpsURLConnection.getInputStream(SOAPHttpsURLConnection.java:37)
    ^-- Holding lock: weblogic.net.http.SOAPHttpsURLConnection@bd3b209[thin lock]
    weblogic.net.http.HttpURLConnection.getResponseCode(HttpURLConnection.java:939)
    com.sun.xml.ws.transport.http.client.HttpClientTransport.readResponseCodeAndMessage(HttpClientTransport.java:213)
    com.sun.xml.ws.transport.http.client.HttpTransportPipe.process(HttpTransportPipe.java:116)
    com.sun.xml.ws.transport.http.client.HttpTransportPipe.processRequest(HttpTransportPipe.java:101)
    com.sun.xml.ws.api.pipe.Fiber.__doRun(Fiber.java:580)
    com.sun.xml.ws.api.pipe.Fiber._doRun(Fiber.java:553)
    com.sun.xml.ws.api.pipe.Fiber.doRun(Fiber.java:535)
    com.sun.xml.ws.api.pipe.Fiber.runSync(Fiber.java:433)
    ^-- Holding lock: com.sun.xml.ws.api.pipe.Fiber@bd3afa8[thin lock]
    com.sun.xml.ws.client.Stub.process(Stub.java:238)
    com.sun.xml.ws.client.sei.SEIStub.doProcess(SEIStub.java:135)
    com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:102)
    com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:89)
    com.sun.xml.ws.client.sei.SEIStub.invoke(SEIStub.java:116)
    $Proxy107.jobSubmit(Unknown Source)
    com.somejob.invokeService(JobOperation.java:53)
    com.somejobws.operations.ticket.execute(ticket.java:30)
    com.somejobutil.Somehelper.process(Somehelper.java:174)
    com.somejobutil.Somehelper.processJob(Somehelper.java:147)
    com.somejobutil.Somehelper.processMDB(Somehelper.java:91)
    com.somejobejb.ProcessMDB.onMessage(ProcessMDB.java:28)
    sun.reflect.GeneratedMethodAccessor109.invoke(Unknown Source)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    java.lang.reflect.Method.invoke(Method.java:575)
    com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
    com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:102)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:86)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
    com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:102)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:151)
    $Proxy67.onMessage(Unknown Source)
    weblogic.ejb.container.internal.MDListener.execute(MDListener.java:403)
    weblogic.ejb.container.internal.MDListener.transactionalOnMessage(MDListener.java:352)
    weblogic.ejb.container.internal.MDListener.onMessage(MDListener.java:264)
    weblogic.jms.client.JMSSession.onMessage(JMSSession.java:4564)
    weblogic.jms.client.JMSSession.execute(JMSSession.java:4060)
    weblogic.jms.client.JMSSession.executeMessage(JMSSession.java:3746)
    weblogic.jms.client.JMSSession.access$000(JMSSession.java:114)
    weblogic.jms.client.JMSSession$UseForRunnable.run(JMSSession.java:5096)
    weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:516)
    weblogic.work.ExecuteThread.execute(ExecuteThread.java:198)
    weblogic.work.ExecuteThread.run(ExecuteThread.java:165)
    }

    Yes, we have the same problem, stuck thread on weblogic.net.http.HttpsURLConnection.getResponseCode() (actually implemented by weblogic.net.http.SOAPHttpsURLConnection which also appears to be deprecated ???)
    When we use the class javax.net.ssl.HttpsURLConnection the getResponseCode() method works perfectly well, so it looks like the weblogic implementation is to blame.
    This is becoming rather urgent now because we're looking to migrate from OC4J to Weblogic and this issue has therefore become a blocker.

  • Stuck thread in Oracle Weblogic server

    Dear All,
    We have a stuck thread in Oracle WLS 10.3.3. We have deployed 7 ear files in 5th managed server(psMS5). We have stuck thread in one of the application (psinsurance). We have allocated 1GB heap space for the psMS5 managed server. We are using this in Development environment. Please check the below message from psMS5 managed server log file. Kindly provide reason for stuck thread and solution.
    WatchTime: Sep 23, 2011 10:28:29 AM IST
    WatchDomainName: dev_domain
    WatchServerName: psMS5
    WatchSeverityLevel: Warning
    WatchName: MS5-Watch
    WatchRuleType: Log
    WatchRule: ((MESSAGE LIKE '%has been busy for "%') AND (MSGID = 'BEA-000337') AND (SERVER = 'psMS5'))
    WatchData: DATE = Sep 23, 2011 10:28:28 AM IST SERVER = psMS5 MESSAGE = [STUCK] ExecuteThread: '22' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "629" seconds working on the request "weblogic.servlet.internal.ServletRequestImpl@1cac6525[
    POST */psinsurance/app/lending/insurance/PS00738D.jspx*?_adf.ctrl-state=24f0li74e_4 HTTP/1.1
    Accept: image/jpeg, application/x-ms-application, image/gif, application/xaml+xml, image/pjpeg, application/x-ms-xbap, */*
    Referer: http://sis-prosun11/psinsurance/faces/PS00003?jsessionid=r2BrT8TpqZBsnLhCw2Y8KDpTV7hMpgnLDFW54P87ZpHwn8JMy88s%21168932459%211316753022074&_afrWindowMode=0&_afrLoop=244724092027000&_adf.ctrl-state=24f0li74e_4
    Accept-Language: en-US
    User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; Tablet PC 2.0)
    Content-Type: application/x-www-form-urlencoded
    Accept-Encoding: gzip, deflate
    Content-Length: 554
    Cache-Control: no-cache
    Cookie: JSESSIONID=LkJcT8TfMT64SyQJT4Ls4sXhfDl0pTS7CXJVQ9KcryHGWJ1dhyfz!1373322659
    Connection: Keep-Alive
    Proxy-Client-IP: 192.168.9.179
    X-Forwarded-For: 192.168.9.179
    X-WebLogic-KeepAliveSecs: 30
    X-WebLogic-Force-JVMID: 1373322659
    ]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
    Thread-237 "[STUCK] ExecuteThread: '22' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, suspended, priority=1, DAEMON> {
    java.io.UnixFileSystem.getBooleanAttributes0(UnixFileSystem.java:???)
    java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:228)
    java.io.File.exists(File.java:729)
    weblogic.utils.classloaders.ClasspathClassFinder.getFileSource(ClasspathClassFinder.java:383)
    weblogic.utils.classloaders.ClasspathClassFinder.getSourcesInternal(ClasspathClassFinder.java:333)
    weblogic.utils.classloaders.ClasspathClassFinder.getSource(ClasspathClassFinder.java:316)
    weblogic.utils.classloaders.JarClassFinder.getSource(JarClassFinder.java:58)
    weblogic.utils.classloaders.AbstractClassFinder.getClassSource(AbstractClassFinder.java:22)
    weblogic.utils.classloaders.MultiClassFinder.getClassSource(MultiClassFinder.java:56)
    weblogic.utils.classloaders.CodeGenClassFinder.getClassSource(CodeGenClassFinder.java:26)
    weblogic.utils.classloaders.GenericClassLoader.findLocalClass(GenericClassLoader.java:268)
    *^-- Holding lock: weblogic.utils.classloaders.GenericClassLoader@1e3295a1[thin lock]*
    weblogic.utils.classloaders.GenericClassLoader.findClass(GenericClassLoader.java:237)
    java.lang.ClassLoader.loadClass(ClassLoader.java:292)
    *^-- Holding lock: weblogic.utils.classloaders.GenericClassLoader@1e3295a1[thin lock]*
    java.lang.ClassLoader.loadClass(ClassLoader.java:292)
    java.lang.ClassLoader.loadClass(ClassLoader.java:248)
    weblogic.utils.classloaders.GenericClassLoader.loadClass(GenericClassLoader.java:177)
    weblogic.utils.classloaders.ChangeAwareClassLoader.loadClass(ChangeAwareClassLoader.java:37)
    *^-- Holding lock: weblogic.utils.classloaders.ChangeAwareClassLoader@1e553474[thin lock]*
    java.lang.Class.getDeclaredConstructors0(Class.java:???)
    java.lang.Class.privateGetDeclaredConstructors(Class.java:2370)
    java.lang.Class.getConstructor0(Class.java:2699)
    java.lang.Class.newInstance0(Class.java:318)
    java.lang.Class.newInstance(Class.java:305)
    org.apache.myfaces.trinidadinternal.renderkit.ClassRendererInstantiator.instantiate(ClassRendererInstantiator.java:48)
    org.apache.myfaces.trinidadinternal.renderkit.RenderKitBase.findRenderer(RenderKitBase.java:146)
    org.apache.myfaces.trinidadinternal.renderkit.RenderKitDecorator.findRenderer(RenderKitDecorator.java:84)
    oracle.adfinternal.view.faces.unified.renderkit.UnifiedRenderKit._getRenderer(UnifiedRenderKit.java:218)
    oracle.adfinternal.view.faces.unified.renderkit.UnifiedRenderKit.findRenderer(UnifiedRenderKit.java:156)
    org.apache.myfaces.trinidadinternal.renderkit.RenderKitBase.getRenderer(RenderKitBase.java:129)
    org.apache.myfaces.trinidad.component.UIXComponentBase._getRendererImpl(UIXComponentBase.java:1143)
    org.apache.myfaces.trinidad.component.UIXComponentBase.getRenderer(UIXComponentBase.java:1173)
    org.apache.myfaces.trinidad.component.UIXComponentBase.getClientId(UIXComponentBase.java:316)
    javax.faces.webapp.UIComponentClassicTagBase.doStartTag(UIComponentClassicTagBase.java:1095)
    org.apache.myfaces.trinidad.webapp.UIXComponentELTag.doStartTag(UIXComponentELTag.java:70)
    oracle.adfinternal.view.faces.unified.taglib.input.UnifiedSelectBooleanCheckboxTag.doStartTag(UnifiedSelectBooleanCheckboxTag.java:45)
    oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
    oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
    oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
    oracle.jsp.runtime.tree.OracleJspIterationTagNode.executeHandler(OracleJspIterationTagNode.java:41)
    oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
    oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
    oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
    oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
    oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
    oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
    oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
    oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
    oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
    oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
    oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
    oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
    oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
    oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
    oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
    oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
    oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
    oracle.jsp.runtime.tree.OracleJspIterationTagNode.executeHandler(OracleJspIterationTagNode.java:41)
    oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
    oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
    oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
    oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
    oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
    oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
    oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
    oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
    oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
    oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
    oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
    oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
    oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
    oracle.jsp.runtime.tree.OracleJspNode.execute(OracleJspNode.java:73)
    oracle.jsp.runtimev2.ShortCutServlet._jspService(ShortCutServlet.java:57)
    oracle.jsp.runtime.OracleJspBase.service(OracleJspBase.java:29)
    oracle.jsp.runtimev2.JspPageTable.compileAndServe(JspPageTable.java:650)
    oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.java:330)
    oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.java:680)
    oracle.jsp.runtimev2.JspServlet.service(JspServlet.java:653)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:815)
    weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
    weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
    weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:206)
    weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    oracle.adf.library.webapp.LibraryFilter.doFilter(LibraryFilter.java:147)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    weblogic.servlet.internal.RequestDispatcherImpl.invokeServlet(RequestDispatcherImpl.java:520)
    weblogic.servlet.internal.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:112)
    com.sun.faces.context.ExternalContextImpl.dispatch(ExternalContextImpl.java:402)
    org.apache.myfaces.trinidad.context.ExternalContextDecorator.dispatch(ExternalContextDecorator.java:44)
    org.apache.myfaces.trinidad.context.ExternalContextDecorator.dispatch(ExternalContextDecorator.java:44)
    org.apache.myfaces.trinidad.context.ExternalContextDecorator.dispatch(ExternalContextDecorator.java:44)
    org.apache.myfaces.trinidad.context.ExternalContextDecorator.dispatch(ExternalContextDecorator.java:44)
    org.apache.myfaces.trinidadinternal.context.FacesContextFactoryImpl$OverrideDispatch.dispatch(FacesContextFactoryImpl.java:261)
    com.sun.faces.application.ViewHandlerImpl.executePageToBuildView(ViewHandlerImpl.java:428)
    com.sun.faces.application.ViewHandlerImpl.renderView(ViewHandlerImpl.java:132)
    javax.faces.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:189)
    org.apache.myfaces.trinidadinternal.application.ViewHandlerImpl.renderView(ViewHandlerImpl.java:155)
    oracle.adfinternal.view.faces.lifecycle.LifecycleImpl._renderResponse(LifecycleImpl.java:706)
    oracle.adfinternal.view.faces.lifecycle.LifecycleImpl._executePhase(LifecycleImpl.java:244)
    oracle.adfinternal.view.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:199)
    javax.faces.webapp.FacesServlet.service(FacesServlet.java:246)
    weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
    weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
    weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:206)
    weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    oracle.adf.model.servlet.ADFBindingFilter.doFilter(ADFBindingFilter.java:134)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    oracle.adfinternal.view.faces.webapp.rich.RegistrationFilter.doFilter(RegistrationFilter.java:69)
    org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:411)
    oracle.adfinternal.view.faces.activedata.AdsFilter.doFilter(AdsFilter.java:44)
    org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:411)
    org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl._doFilterImpl(TrinidadFilterImpl.java:210)
    org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl.doFilter(TrinidadFilterImpl.java:126)
    org.apache.myfaces.trinidad.webapp.TrinidadFilter.doFilter(TrinidadFilter.java:91)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:94)
    oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:299)
    oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:406)
    oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:82)
    oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:70)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    oracle.adf.library.webapp.LibraryFilter.doFilter(LibraryFilter.java:147)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    oracle.dms.wls.DMSServletFilter.doFilter(DMSServletFilter.java:114)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:25)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.doIt(WebAppServletContext.java:3656)
    weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3650)
    weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:308)
    weblogic.security.service.SecurityManager.runAs(SecurityManager.java:117)
    weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2204)
    weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2126)
    weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1412)
    weblogic.work.ExecuteThread.execute(ExecuteThread.java:198)
    weblogic.work.ExecuteThread.run(ExecuteThread.java:165)
    SUBSYSTEM = WebLogicServer USERID = <WLS Kernel> SEVERITY = Error THREAD = [ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)' MSGID = BEA-000337 MACHINE = sis-prosun11.sis.ad TXID = CONTEXTID = 11d1def534ea1be0:3bbe257e:13286031cbf:-7ffd-0000000000002fe7 TIMESTAMP = 1316753908672
    WatchAlarmType: AutomaticReset
    WatchAlarmResetPeriod: 60000
    SMTPNotificationName: MS5-Health-Notification

    I found something interesting to the question I posted from the below link:
    http://www.mail-archive.com/[email protected]/msg03217.html

  • Need help with STUCK Threads.  What's causing them?

    In our production environment, we have a soa suite cluster with 3 nodes. All 3 of the servers keep turning up stuck and hogging threads, and eventually they all grind to a halt where I can't even view composite instances through em. There is a bpel process that is using a dbadapter to poll for records every 5 seconds and then performs some service call outs and other dbadapters. This process is generating thousands of instances a day and I think it is the culprit. If I view the logs, I see one of the dbadapters that it calls out to fails occasionally with a PK violation. I looked at the developer's code, and they have no error handling in the process at all. In the logs, there will the a hit for the PK violation, then about 10 minutes later the a STUCK thread gets logged(the configured timeout is 600 seconds, or 10 minutes). Would not handling an error in a BPEL process hog/STUCK a thread? I need to be able to show what code is causing the issue so I can have a resolution path.
    I also see a note in the trace about something holding a lock:
    Holding lock: oracle.jdbc.driver.T4CConnection@1fbc3486[thin lock]....would that be an un-rolled back transaction?
    Here is the dump of the STUCK thread. Any help is greatly appreciated. Thanks!
    <Aug 5, 2011 10:30:02 AM EDT> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '64' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "634" seconds working on the request "oracle.soa.management.internal.ejb.impl.FacadeFinderBean_4vacyo_FacadeFinderBeanImpl", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
    Thread-517 "[STUCK] ExecuteThread: '64' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, priority=1, DAEMON> {
    jrockit.net.SocketNativeIO.readBytesPinned(SocketNativeIO.java:???)
    jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:24)
    java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)
    java.net.SocketInputStream.read(SocketInputStream.java:107)
    oracle.net.nt.MetricsEnabledInputStream.read(TcpNTAdapter.java:553)
    oracle.net.ns.Packet.receive(Packet.java:249)
    oracle.net.ns.DataPacket.receive(DataPacket.java:94)
    oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:168)
    oracle.net.ns.NetInputStream.read(NetInputStream.java:113)
    oracle.net.ns.NetInputStream.read(NetInputStream.java:96)
    oracle.net.ns.NetInputStream.read(NetInputStream.java:81)
    oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:120)
    oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:75)
    oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1081)
    oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1062)
    oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:588)
    oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:70)
    oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:854)
    oracle.jdbc.driver.T4CPreparedStatement.executeMaybeDescribe(T4CPreparedStatement.java:918)
    oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1252)
    oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3541)
    oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3605)
    ^-- Holding lock: oracle.jdbc.driver.T4CConnection@1fbc3486[thin lock]
    oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1667)
    weblogic.jdbc.wrapper.PreparedStatement.executeQuery(PreparedStatement.java:123)
    org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeSelect(DatabaseAccessor.java:887)
    org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:541)
    org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:530)
    org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:521)
    org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:202)
    org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:192)
    org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelectCall(DatasourceCallQueryMechanism.java:254)
    org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelect(DatasourceCallQueryMechanism.java:245)
    org.eclipse.persistence.queries.DataReadQuery.executeNonCursor(DataReadQuery.java:188)
    org.eclipse.persistence.queries.DataReadQuery.executeDatabaseQuery(DataReadQuery.java:141)
    org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:603)
    org.eclipse.persistence.queries.DataReadQuery.execute(DataReadQuery.java:115)
    org.eclipse.persistence.queries.DatabaseQuery.executeInUnitOfWork(DatabaseQuery.java:589)
    org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2898)
    org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1218)
    org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1190)
    org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1176)
    org.eclipse.persistence.internal.jpa.EJBQueryImpl.executeReadQuery(EJBQueryImpl.java:378)
    org.eclipse.persistence.internal.jpa.EJBQueryImpl.getSingleResult(EJBQueryImpl.java:701)
    weblogic.deployment.TransactionalEntityManagerProxyImpl$QueryImpl.getSingleResult(TransactionalEntityManagerProxyImpl.java:232)
    com.collaxa.cube.persistence.dao.impl.jpa.JpaCubeInstanceDaoImpl.getNumberOfInstances(JpaCubeInstanceDaoImpl.java:439)
    oracle.fabric.CubeServiceEngine.getNumberOfComponentInstances(CubeServiceEngine.java:1962)
    oracle.soa.management.internal.ejb.impl.FacadeFinderBeanImpl.getNumberOfComponentInstances(FacadeFinderBeanImpl.java:1618)
    oracle.soa.management.internal.ejb.impl.FacadeFinderBeanImpl.getComponentSnapShot(FacadeFinderBeanImpl.java:1653)
    sun.reflect.GeneratedMethodAccessor5852.invoke(Unknown Source)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    java.lang.reflect.Method.invoke(Method.java:575)
    com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.jee.intercept.MethodInvocationInvocationContext.proceed(MethodInvocationInvocationContext.java:104)
    oracle.security.jps.ee.ejb.JpsAbsInterceptor$1.run(JpsAbsInterceptor.java:88)
    oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:299)
    oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:406)
    oracle.security.jps.wls.JpsWeblogicEjbInterceptor.runJaasMode(JpsWeblogicEjbInterceptor.java:61)
    oracle.security.jps.ee.ejb.JpsAbsInterceptor.intercept(JpsAbsInterceptor.java:81)
    oracle.security.jps.ee.ejb.JpsInterceptor.intercept(JpsInterceptor.java:101)
    sun.reflect.GeneratedMethodAccessor863.invoke(Unknown Source)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    java.lang.reflect.Method.invoke(Method.java:575)
    com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
    com.bea.core.repackaged.springframework.jee.intercept.JeeInterceptorInterceptor.invoke(JeeInterceptorInterceptor.java:66)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
    com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:102)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.jee.spi.MethodInvocationVisitorImpl.visit(MethodInvocationVisitorImpl.java:37)
    weblogic.ejb.container.injection.EnvironmentInterceptorCallbackImpl.callback(EnvironmentInterceptorCallbackImpl.java:22)
    com.bea.core.repackaged.springframework.jee.spi.EnvironmentInterceptor.invoke(EnvironmentInterceptor.java:49)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:86)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
    com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:102)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:151)
    $Proxy309.getComponentSnapShot(Unknown Source)
    oracle.soa.management.internal.ejb.impl.FacadeFinderBean_4vacyo_FacadeFinderBeanImpl.getComponentSnapShot(FacadeFinderBean_4vacyo_FacadeFinderBeanImpl.java:1604)
    oracle.soa.management.internal.ejb.impl.FacadeFinderBean_4vacyo_FacadeFinderBeanImpl_WLSkel.invoke(Unknown Source)
    weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:567)
    weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:477)
    weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:350)
    weblogic.security.service.SecurityManager.runAs(SecurityManager.java:143)
    weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:425)
    weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:117)
    weblogic.work.ExecuteThread.execute(ExecuteThread.java:198)
    weblogic.work.ExecuteThread.run(ExecuteThread.java:165)
    >
    <Aug 5, 2011 10:30:02 AM EDT> <Notice> <Diagnostics> <BEA-320068> <Watch 'StuckThread' with severity 'Notice' on server 'soa_server1' has triggered at Aug 5, 2011 10:30:02 AM EDT. Notification details:
    WatchRuleType: Log
    WatchRule: (SEVERITY = 'Error') AND (MSGID = 'BEA-000337')
    WatchData: DATE = Aug 5, 2011 10:30:02 AM EDT SERVER = soa_server1 MESSAGE = [STUCK] ExecuteThread: '64' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "634" seconds working on the request "oracle.soa.management.internal.ejb.impl.FacadeFinderBean_4vacyo_FacadeFinderBeanImpl", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
    Thread-517 "[STUCK] ExecuteThread: '64' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, priority=1, DAEMON> {
    jrockit.net.SocketNativeIO.readBytesPinned(SocketNativeIO.java:???)
    jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:24)
    java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)
    java.net.SocketInputStream.read(SocketInputStream.java:107)
    oracle.net.nt.MetricsEnabledInputStream.read(TcpNTAdapter.java:553)
    oracle.net.ns.Packet.receive(Packet.java:249)
    oracle.net.ns.DataPacket.receive(DataPacket.java:94)
    oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:168)
    oracle.net.ns.NetInputStream.read(NetInputStream.java:113)
    oracle.net.ns.NetInputStream.read(NetInputStream.java:96)
    oracle.net.ns.NetInputStream.read(NetInputStream.java:81)
    oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:120)
    oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:75)
    oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1081)
    oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1062)
    oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:588)
    oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:70)
    oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:854)
    oracle.jdbc.driver.T4CPreparedStatement.executeMaybeDescribe(T4CPreparedStatement.java:918)
    oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1252)
    oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3541)
    oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3605)
    ^-- Holding lock: oracle.jdbc.driver.T4CConnection@1fbc3486[thin lock]
    oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1667)
    weblogic.jdbc.wrapper.PreparedStatement.executeQuery(PreparedStatement.java:123)
    org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeSelect(DatabaseAccessor.java:887)
    org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:541)
    org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:530)
    org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:521)
    org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:202)
    org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:192)
    org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelectCall(DatasourceCallQueryMechanism.java:254)
    org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelect(DatasourceCallQueryMechanism.java:245)
    org.eclipse.persistence.queries.DataReadQuery.executeNonCursor(DataReadQuery.java:188)
    org.eclipse.persistence.queries.DataReadQuery.executeDatabaseQuery(DataReadQuery.java:141)
    org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:603)
    org.eclipse.persistence.queries.DataReadQuery.execute(DataReadQuery.java:115)
    org.eclipse.persistence.queries.DatabaseQuery.executeInUnitOfWork(DatabaseQuery.java:589)
    org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2898)
    org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1218)
    org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1190)
    org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1176)
    org.eclipse.persistence.internal.jpa.EJBQueryImpl.executeReadQuery(EJBQueryImpl.java:378)
    org.eclipse.persistence.internal.jpa.EJBQueryImpl.getSingleResult(EJBQueryImpl.java:701)
    weblogic.deployment.TransactionalEntityManagerProxyImpl$QueryImpl.getSingleResult(TransactionalEntityManagerProxyImpl.java:232)
    com.collaxa.cube.persistence.dao.impl.jpa.JpaCubeInstanceDaoImpl.getNumberOfInstances(JpaCubeInstanceDaoImpl.java:439)
    oracle.fabric.CubeServiceEngine.getNumberOfComponentInstances(CubeServiceEngine.java:1962)
    oracle.soa.management.internal.ejb.impl.FacadeFinderBeanImpl.getNumberOfComponentInstances(FacadeFinderBeanImpl.java:1618)
    oracle.soa.management.internal.ejb.impl.FacadeFinderBeanImpl.getComponentSnapShot(FacadeFinderBeanImpl.java:1653)
    sun.reflect.GeneratedMethodAccessor5852.invoke(Unknown Source)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    java.lang.reflect.Method.invoke(Method.java:575)
    com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.jee.intercept.MethodInvocationInvocationContext.proceed(MethodInvocationInvocationContext.java:104)
    oracle.security.jps.ee.ejb.JpsAbsInterceptor$1.run(JpsAbsInterceptor.java:88)
    oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:299)
    oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:406)
    oracle.security.jps.wls.JpsWeblogicEjbInterceptor.runJaasMode(JpsWeblogicEjbInterceptor.java:61)
    oracle.security.jps.ee.ejb.JpsAbsInterceptor.intercept(JpsAbsInterceptor.java:81)
    oracle.security.jps.ee.ejb.JpsInterceptor.intercept(JpsInterceptor.java:101)
    sun.reflect.GeneratedMethodAccessor863.invoke(Unknown Source)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    java.lang.reflect.Method.invoke(Method.java:575)
    com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
    com.bea.core.repackaged.springframework.jee.intercept.JeeInterceptorInterceptor.invoke(JeeInterceptorInterceptor.java:66)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
    com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:102)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.jee.spi.MethodInvocationVisitorImpl.visit(MethodInvocationVisitorImpl.java:37)
    weblogic.ejb.container.injection.EnvironmentInterceptorCallbackImpl.callback(EnvironmentInterceptorCallbackImpl.java:22)
    com.bea.core.repackaged.springframework.jee.spi.EnvironmentInterceptor.invoke(EnvironmentInterceptor.java:49)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:86)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
    com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:102)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:151)
    $Proxy309.getComponentSnapShot(Unknown Source)
    oracle.soa.management.internal.ejb.impl.FacadeFinderBean_4vacyo_FacadeFinderBeanImpl.getComponentSnapShot(FacadeFinderBean_4vacyo_FacadeFinderBeanImpl.java:1604)
    oracle.soa.management.internal.ejb.impl.FacadeFinderBean_4vacyo_FacadeFinderBeanImpl_WLSkel.invoke(Unknown Source)
    weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:567)
    weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:477)
    weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:350)
    weblogic.security.service.SecurityManager.runAs(SecurityManager.java:143)
    weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:425)
    weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:117)
    weblogic.work.ExecuteThread.execute(ExecuteThread.java:198)
    weblogic.work.ExecuteThread.run(ExecuteThread.java:165)
    SUBSYSTEM = WebLogicServer USERID = <WLS Kernel> SEVERITY = Error THREAD = [ACTIVE] ExecuteThread: '96' for queue: 'weblogic.kernel.Default (self-tuning)' MSGID = BEA-000337 MACHINE = soaprod TXID = CONTEXTID = f5d0b85a929630a2:6eada0f7:1304e0a2bdc:-7ffb-00000000017e1605 TIMESTAMP = 1312554602698
    WatchAlarmType: AutomaticReset
    WatchAlarmResetPeriod: 600000
    >
    Edited by: Brutus35 on Aug 5, 2011 8:04 AM

    my impression is that you have trouble on the DB,
    monitor the sessions to see if there is some table locked, and increase the number of processes on the DB
    you can use SQLDeveloper for that
    anyway your TX should rollback after some time, check that you have a timeout in place, it's better to rollback a TX than having the entire server crash

  • Stuck Threads Leading to server hang

    Hi ,
    We have been facing a problem with Stuck Threads in our production environment. We are currently running version 8.1. Though the stuck Threads do point to JDBC issues, we suspect that these problems may be related to some Weblogic/OS configuration issue as well. These probelms eventually lead to a server hang.
    The general pattern of the problem is that we see messages like :
    Dec 24, 2006 8:56:09 AM EST> <Debug> <Common> <BEA-000610> < RP(rtqdbConnectionPool):timeoutInactiveRes (20)>
    ####<Dec 25, 2006 8:54:51 AM EST> <Error> <WebLogicServer> <hygeia> <hygeia-rtq-vm7> <weblogic.health.CoreHealthMonitor> <<WLS Kernel>> <> <BEA-000337> <UNKNOWN_SESSION has been busy for "896" seconds working on the request "Http Request: /quote.do", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.> ==================== This email/fax message is for the sole use of the intended
    Sometimes a deadlock Exception would be thrown from the Database,
    nested exception is java.sql.SQLException: ORA-00060: deadlock detected while waiting for resource
    But this is not as often as we get the Stuck Thread messages.
    This is a typical Thread dump from one of our servers when it complained of a stuck thread.
    ///*************Thread Dump Start**********************///
    Full thread dump Java HotSpot(TM) Server VM (1.4.2_08-b03 mixed mode):
    "idle thread" daemon prio=5 tid=0x00d2c150 nid=0x7f in Object.wait() [94e7f000..94e7fc28]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xb2010000> (a org.xbill.Task.WorkerThread)
         at org.xbill.Task.WorkerThread.run(WorkerThread.java:130)
         - locked <0xb2010000> (a org.xbill.Task.WorkerThread)
    "Thread-28" daemon prio=5 tid=0x00c6a7c0 nid=0x7d in Object.wait() [92c7f000..92c7fc28]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xbce667f8> (a com.geico.util.LockManager)
         at com.geico.util.LockManager.waitForWork(LockManager.java:85)
         - locked <0xbce667f8> (a com.geico.util.LockManager)
         at com.geico.mail.Mail.run(Mail.java:259)
         at java.lang.Thread.run(Thread.java:534)
    "CacheManager-idle" daemon prio=5 tid=0x00c7d950 nid=0x7c in Object.wait() [92d7f000..92d7fc28]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xbce66878> (a com.geico.util.LockManager)
         at com.geico.util.LockManager.waitForWork(LockManager.java:85)
         - locked <0xbce66878> (a com.geico.util.LockManager)
         at com.geico.cache.CacheManager.run(CacheManager.java:353)
         at java.lang.Thread.run(Thread.java:534)
    "ExecuteThread: '2' for queue: 'weblogic.kernel.Non-Blocking'" daemon prio=5 tid=0x006a6c90 nid=0x7a in Object.wait() [92e7f000..92e7fc28]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xbc2a5db8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '1' for queue: 'weblogic.kernel.Non-Blocking'" daemon prio=5 tid=0x0120c568 nid=0x79 in Object.wait() [92f7f000..92f7fc28]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xbc2a5e38> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '0' for queue: 'weblogic.kernel.Non-Blocking'" daemon prio=5 tid=0x006a6af0 nid=0x78 in Object.wait() [9307f000..9307fc28]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xbc2a5eb8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '0' for queue: 'JMSStore<AptSoftFileStore-morpheus-rtq-vm7>.ioThreadPool'" daemon prio=5 tid=0x0052a198 nid=0x77 in Object.wait() [9317f000..9317fc28]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xbc2a5f38> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "Thread-26" daemon prio=5 tid=0x00635d58 nid=0x75 in Object.wait() [9327f000..9327fc28]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xbb97ad20> (a com.geico.util.LockManager)
         at java.lang.Object.wait(Object.java:429)
         at com.geico.util.LockManager.waitForWork(LockManager.java:63)
         - locked <0xbb97ad20> (a com.geico.util.LockManager)
         at com.geico.util.providers.StorageProvider.run(StorageProvider.java:158)
         at java.lang.Thread.run(Thread.java:534)
    "Thread-25" daemon prio=5 tid=0x00fac430 nid=0x72 waiting on condition [9337f000..9337fc28]
         at java.lang.Thread.sleep(Native Method)
         at org.apache.commons.pool.impl.GenericObjectPool$Evictor.run(GenericObjectPool.java:1080)
         at java.lang.Thread.run(Thread.java:534)
    "ListenThread.Default" prio=5 tid=0x014ec6e8 nid=0x71 runnable [9347f000..9347fc28]
         at java.net.PlainSocketImpl.socketAccept(Native Method)
         at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
         - locked <0xbb97ae60> (a java.net.PlainSocketImpl)
         at java.net.ServerSocket.implAccept(ServerSocket.java:448)
         at java.net.ServerSocket.accept(ServerSocket.java:419)
         at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:26)
         at weblogic.t3.srvr.ListenThread.accept(ListenThread.java:735)
         at weblogic.t3.srvr.ListenThread.run(ListenThread.java:301)
    "Thread-24" daemon prio=5 tid=0x011e9668 nid=0x70 waiting on condition [9357f000..9357fc28]
         at java.lang.Thread.sleep(Native Method)
         at com.geico.log.HeartbeatLogger$1.run(HeartbeatLogger.java:133)
    "Thread-22" daemon prio=5 tid=0x011e8610 nid=0x6f waiting on condition [9367f000..9367fc28]
         at java.lang.Thread.sleep(Native Method)
         at com.geico.log.StreamLogger.pause(StreamLogger.java:263)
         at com.geico.log.StreamLogger.access$300(StreamLogger.java:15)
         at com.geico.log.StreamLogger$WorkerThread.run(StreamLogger.java:217)
         at java.lang.Thread.run(Thread.java:534)
    "filewatcher" daemon prio=5 tid=0x0111ca50 nid=0x6e waiting on condition [9377f000..9377fc28]
         at java.lang.Thread.sleep(Native Method)
         at com.geico.watcher.FileWatcher.sleep(FileWatcher.java:135)
         at com.geico.watcher.FileWatcher.run(FileWatcher.java:118)
         at java.lang.Thread.run(Thread.java:534)
    "Thread-18" prio=5 tid=0x00de2388 nid=0x6d in Object.wait() [9437e000..9437fc28]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xa20f1a60> (a weblogic.jms.backend.BEConsumerReceiveRequest)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.jms.dispatcher.Request.sleepTillNotified(Request.java:240)
         at weblogic.jms.dispatcher.Request.wrappedFiniteStateMachine(Request.java:685)
         - locked <0xa20f1a60> (a weblogic.jms.backend.BEConsumerReceiveRequest)
         at weblogic.jms.dispatcher.DispatcherImpl.dispatchAsyncInternal(DispatcherImpl.java:132)
         at weblogic.jms.dispatcher.DispatcherImpl.dispatchAsync(DispatcherImpl.java:123)
         at weblogic.jms.dispatcher.Request.dispatchAsync(Request.java:878)
         at weblogic.jms.frontend.FEConsumer.receive(FEConsumer.java:452)
         at weblogic.jms.frontend.FEConsumer.invoke(FEConsumer.java:643)
         at weblogic.jms.dispatcher.Request.wrappedFiniteStateMachine(Request.java:643)
         at weblogic.jms.dispatcher.DispatcherImpl.dispatchSyncTran(DispatcherImpl.java:192)
         at weblogic.jms.client.JMSSession.receiveMessage(JMSSession.java:456)
         at weblogic.jms.client.JMSConsumer.receive(JMSConsumer.java:421)
         at org.springframework.jms.core.JmsTemplate.doReceive(JmsTemplate.java:800)
         at org.springframework.jms.core.JmsTemplate.doReceive(JmsTemplate.java:783)
         at org.springframework.jms.core.JmsTemplate$8.doInJms(JmsTemplate.java:741)
         at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:582)
         at org.springframework.jms.core.JmsTemplate.receive(JmsTemplate.java:739)
         at org.springframework.jms.core.JmsTemplate.receive(JmsTemplate.java:731)
         at com.geico.uierrorlog.UiErrorUtil.getMessage(UiErrorUtil.java:89)
         at com.geico.uierrorlog.UiErrorDaemon.getMessage(UiErrorDaemon.java:108)
         at com.geico.uierrorlog.UiErrorDaemon.run(UiErrorDaemon.java:166)
         at java.lang.Thread.run(Thread.java:534)
    "StatisticsThread" daemon prio=5 tid=0x00d0c798 nid=0x6c in Object.wait() [9447f000..9447fc28]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xb8ee7b18> (a com.geico.tangosol.stats.StatsDaemon)
         at com.geico.tangosol.stats.StatsDaemon.run(StatsDaemon.java:144)
         - locked <0xb8ee7b18> (a com.geico.tangosol.stats.StatsDaemon)
         at com.tangosol.util.Daemon$1.run(Daemon.java:63)
    "StatisticsThread" daemon prio=5 tid=0x01202c18 nid=0x6b in Object.wait() [9457f000..9457fc28]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xb8ee7bd8> (a com.geico.tangosol.stats.StatsDaemon)
         at com.geico.tangosol.stats.StatsDaemon.run(StatsDaemon.java:144)
         - locked <0xb8ee7bd8> (a com.geico.tangosol.stats.StatsDaemon)
         at com.tangosol.util.Daemon$1.run(Daemon.java:63)
    "StatisticsThread" daemon prio=5 tid=0x00b33f18 nid=0x6a in Object.wait() [9467f000..9467fc28]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xb8ee7c90> (a com.geico.tangosol.stats.StatsDaemon)
         at com.geico.tangosol.stats.StatsDaemon.run(StatsDaemon.java:144)
         - locked <0xb8ee7c90> (a com.geico.tangosol.stats.StatsDaemon)
         at com.tangosol.util.Daemon$1.run(Daemon.java:63)
    "DistributedCache:DistributedRTQOverflow" daemon prio=5 tid=0x009895e8 nid=0x69 in Object.wait() [9477f000..9477fc28]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xb8ee7d40> (a com.tangosol.coherence.component.util.daemon.QueueProcessor$Queue)
         at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
         - locked <0xb8ee7d40> (a com.tangosol.coherence.component.util.daemon.QueueProcessor$Queue)
         at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
         at java.lang.Thread.run(Thread.java:534)
    "DistributedCache:PoolStatsDistributedCache:EventDispatcher" daemon prio=5 tid=0x00dff060 nid=0x68 in Object.wait() [9487f000..9487fc28]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xb8ee7f18> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
         at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
         - locked <0xb8ee7f18> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
         at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
         at java.lang.Thread.run(Thread.java:534)
    "DistributedCache:PoolStatsDistributedCache" daemon prio=5 tid=0x006f3e98 nid=0x67 in Object.wait() [9497f000..9497fc28]
         at java.lang.Object.wait(Native Method)
         at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
         - locked <0xb8ee7fe8> (a com.tangosol.coherence.component.util.daemon.QueueProcessor$Queue)
         at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
         at java.lang.Thread.run(Thread.java:534)
    "StatisticsThread" daemon prio=5 tid=0x0090d8d0 nid=0x66 in Object.wait() [94a7f000..94a7fc28]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xb8ee81c8> (a com.geico.tangosol.stats.StatsDaemon)
         at com.geico.tangosol.stats.StatsDaemon.run(StatsDaemon.java:144)
         - locked <0xb8ee81c8> (a com.geico.tangosol.stats.StatsDaemon)
         at com.tangosol.util.Daemon$1.run(Daemon.java:63)
    "DistributedCache:dist-RTQSession:EventDispatcher" daemon prio=5 tid=0x00a05110 nid=0x65 in Object.wait() [94b7f000..94b7fc28]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xb8ee8278> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
         at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
         - locked <0xb8ee8278> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
         at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
         at java.lang.Thread.run(Thread.java:534)
    "DistributedCache:dist-RTQSession" daemon prio=5 tid=0x0036e600 nid=0x64 in Object.wait() [94c7f000..94c7fc28]
         at java.lang.Object.wait(Native Method)
         at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
         - locked <0xb8ee8348> (a com.tangosol.coherence.component.util.daemon.QueueProcessor$Queue)
         at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
         at java.lang.Thread.run(Thread.java:534)
    "ReplicatedCache" daemon prio=5 tid=0x00e9f9f8 nid=0x63 in Object.wait() [94d7f000..94d7fc28]
         at java.lang.Object.wait(Native Method)
         at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
         - locked <0xb8ee8520> (a com.tangosol.coherence.component.util.daemon.QueueProcessor$Queue)
         at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
         at java.lang.Thread.run(Thread.java:534)
    "Scheduler_Worker-9" prio=5 tid=0x00339820 nid=0x61 in Object.wait() [94f7f000..94f7fc28]
         at java.lang.Object.wait(Native Method)
         at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:423)
         - locked <0xb8ee8788> (a java.lang.Object)
         at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:53)
         at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:514)
    "Scheduler_Worker-8" prio=5 tid=0x0033af58 nid=0x60 in Object.wait() [9507f000..9507fc28]
         at java.lang.Object.wait(Native Method)
         at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:423)
         - locked <0xb8ee8788> (a java.lang.Object)
         at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:53)
         at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:514)
    "Scheduler_Worker-7" prio=5 tid=0x0033a878 nid=0x5f in Object.wait() [9517f000..9517fc28]
         at java.lang.Object.wait(Native Method)
         at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:423)
         - locked <0xb8ee8788> (a java.lang.Object)
         at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:53)
         at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:514)
    "Scheduler_Worker-6" prio=5 tid=0x00935488 nid=0x5e in Object.wait() [9527f000..9527fc28]
         at java.lang.Object.wait(Native Method)
         at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:423)
         - locked <0xb8ee8788> (a java.lang.Object)
         at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:53)
         at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:514)
    "Scheduler_Worker-5" prio=5 tid=0x00bb2190 nid=0x5d in Object.wait() [9537f000..9537fc28]
         at java.lang.Object.wait(Native Method)
         at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:423)
         - locked <0xb8ee8788> (a java.lang.Object)
         at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:53)
         at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:514)
    "Scheduler_Worker-4" prio=5 tid=0x010b9038 nid=0x5c in Object.wait() [9547f000..9547fc28]
         at java.lang.Object.wait(Native Method)
         at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:423)
         - locked <0xb8ee8788> (a java.lang.Object)
         at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:53)
         at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:514)
    "Scheduler_Worker-3" prio=5 tid=0x00a34678 nid=0x5b in Object.wait() [9557f000..9557fc28]
         at java.lang.Object.wait(Native Method)
         at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:423)
         - locked <0xb8ee8788> (a java.lang.Object)
         at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:53)
         at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:514)
    "Scheduler_Worker-2" prio=5 tid=0x014f9860 nid=0x5a in Object.wait() [9567f000..9567fc28]
         at java.lang.Object.wait(Native Method)
         at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:423)
         - locked <0xb8ee8788> (a java.lang.Object)
         at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:53)
         at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:514)
    "Scheduler_Worker-1" prio=5 tid=0x0044e530 nid=0x59 in Object.wait() [9577f000..9577fc28]
         at java.lang.Object.wait(Native Method)
         at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:423)
         - locked <0xb8ee8788> (a java.lang.Object)
         at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:53)
         at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:514)
    "Scheduler_Worker-0" prio=5 tid=0x0110b8e8 nid=0x58 in Object.wait() [9587f000..9587fc28]
         at java.lang.Object.wait(Native Method)
         at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:423)
         - locked <0xb8ee8788> (a java.lang.Object)
         at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:53)
         at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:514)
    "StatisticsThread" daemon prio=5 tid=0x011352b8 nid=0x57 in Object.wait() [9597f000..9597fc28]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xb8ee8b20> (a com.geico.tangosol.stats.StatsDaemon)
         at com.geico.tangosol.stats.StatsDaemon.run(StatsDaemon.java:144)
         - locked <0xb8ee8b20> (a com.geico.tangosol.stats.StatsDaemon)
         at com.tangosol.util.Daemon$1.run(Daemon.java:63)
    "DistributedCache:dist-rtq-sessionheader:EventDispatcher" daemon prio=5 tid=0x004222a0 nid=0x56 in Object.wait() [95a7f000..95a7fc28]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xb8ee8bd8> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
         at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
         - locked <0xb8ee8bd8> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
         at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
         at java.lang.Thread.run(Thread.java:534)
    "DistributedCache:dist-rtq-sessionheader" daemon prio=5 tid=0x01278fa0 nid=0x55 in Object.wait() [95b7f000..95b7fc28]
         at java.lang.Object.wait(Native Method)
         at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
         - locked <0xb8ee8ca8> (a com.tangosol.coherence.component.util.daemon.QueueProcessor$Queue)
         at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
         at java.lang.Thread.run(Thread.java:534)
    "DistributedCache:dist-hibernate-default:EventDispatcher" daemon prio=5 tid=0x015273f0 nid=0x54 in Object.wait() [95c7f000..95c7fc28]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xb8ee8e80> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
         at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
         - locked <0xb8ee8e80> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
         at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
         at java.lang.Thread.run(Thread.java:534)
    "DistributedCache:dist-hibernate-default" daemon prio=5 tid=0x00ea1328 nid=0x53 in Object.wait() [9607f000..9607fc28]
         at java.lang.Object.wait(Native Method)
         at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
         - locked <0xb8ee8f50> (a com.tangosol.coherence.component.util.daemon.QueueProcessor$Queue)
         at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
         at java.lang.Thread.run(Thread.java:534)
    "Invocation:Management" daemon prio=5 tid=0x003d8968 nid=0x52 in Object.wait() [95d7f000..95d7fc28]
         at java.lang.Object.wait(Native Method)
         at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
         - locked <0xb8448698> (a com.tangosol.coherence.component.util.daemon.QueueProcessor$Queue)
         at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
         at java.lang.Thread.run(Thread.java:534)
    "Invocation:Management:EventDispatcher" daemon prio=5 tid=0x007e6040 nid=0x51 in Object.wait() [95e7f000..95e7fc28]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xb84487f0> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
         at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
         - locked <0xb84487f0> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
         at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
         at java.lang.Thread.run(Thread.java:534)
    "TcpRingListener" daemon prio=6 tid=0x007e6f40 nid=0x50 runnable [95f7f000..95f7fc28]
         at java.net.PlainSocketImpl.socketAccept(Native Method)
         at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
         - locked <0xb84488d0> (a java.net.PlainSocketImpl)
         at java.net.ServerSocket.implAccept(ServerSocket.java:448)
         at java.net.ServerSocket.accept(ServerSocket.java:419)
         at com.tangosol.coherence.component.net.socket.TcpSocketAccepter.accept(TcpSocketAccepter.CDB:17)
         at com.tangosol.coherence.component.util.daemon.TcpRingListener.acceptConnection(TcpRingListener.CDB:9)
         at com.tangosol.coherence.component.util.daemon.TcpRingListener.onNotify(TcpRingListener.CDB:1)
         at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:34)
         at java.lang.Thread.run(Thread.java:534)
    "Cluster" daemon prio=5 tid=0x00c94738 nid=0x4e in Object.wait() [9617f000..9617fc28]
         at java.lang.Object.wait(Native Method)
         at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
         - locked <0xb8448ac0> (a com.tangosol.coherence.component.net.Cluster$ClusterService$Queue)
         at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
         at java.lang.Thread.run(Thread.java:534)
    "PacketPublisher" daemon prio=6 tid=0x00d83418 nid=0x4d runnable [9627f000..9627fc28]
         at java.lang.Object.wait(Native Method)
         at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
         - locked <0xb8448c78> (a com.tangosol.coherence.component.net.Cluster$PacketPublisher$Queue)
         at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
         at java.lang.Thread.run(Thread.java:534)
    "PacketReceiver" daemon prio=7 tid=0x00b3cd70 nid=0x4c in Object.wait() [9637f000..9637fc28]
         at java.lang.Object.wait(Native Method)
         at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
         - locked <0xb8448dc0> (a com.tangosol.coherence.component.net.Cluster$PacketReceiver$Queue)
         at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
         at java.lang.Thread.run(Thread.java:534)
    "PacketListenerN" daemon prio=8 tid=0x003d7cd0 nid=0x4b runnable [9647f000..9647fc28]
         at java.net.PlainDatagramSocketImpl.receive(Native Method)
         at java.net.DatagramSocket.receive(DatagramSocket.java:711)
         - locked <0xb84494c8> (a java.net.DatagramPacket)
         - locked <0xb84494e8> (a java.net.MulticastSocket)
         at com.tangosol.coherence.component.net.socket.UdpSocket.receive(UdpSocket.CDB:18)
         at com.tangosol.coherence.component.net.UdpPacket.receive(UdpPacket.CDB:17)
         at com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onWait(PacketListener.CDB:5)
         at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
         at java.lang.Thread.run(Thread.java:534)
    "PacketListener1" daemon prio=8 tid=0x00df5538 nid=0x4a runnable [9657f000..9657fc28]
         at java.net.PlainDatagramSocketImpl.receive(Native Method)
         at java.net.DatagramSocket.receive(DatagramSocket.java:711)
         - locked <0xb8449cb8> (a java.net.DatagramPacket)
         - locked <0xb8449cd8> (a java.net.DatagramSocket)
         at com.tangosol.coherence.component.net.socket.UdpSocket.receive(UdpSocket.CDB:18)
         at com.tangosol.coherence.component.net.UdpPacket.receive(UdpPacket.CDB:17)
         at com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onWait(PacketListener.CDB:5)
         at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
         at java.lang.Thread.run(Thread.java:534)
    "Logger@13785534 3.1.2/343" daemon prio=3 tid=0x006b7358 nid=0x49 in Object.wait() [9667f000..9667fc28]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xb8449e90> (a com.tangosol.coherence.component.application.console.Coherence$Logger$Queue)
         at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
         - locked <0xb8449e90> (a com.tangosol.coherence.component.application.console.Coherence$Logger$Queue)
         at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
         at java.lang.Thread.run(Thread.java:534)
    "ExecuteThread: '5' for queue: 'JMS.TimerClientPool'" daemon prio=5 tid=0x011384b0 nid=0x48 in Object.wait() [9677f000..9677fc28]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xb82c91f8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '4' for queue: 'JMS.TimerClientPool'" daemon prio=5 tid=0x01137bc8 nid=0x47 in Object.wait() [9687f000..9687fc28]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xb82c9278> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '3' for queue: 'JMS.TimerClientPool'" daemon prio=5 tid=0x0111d4e0 nid=0x46 in Object.wait() [9697f000..9697fc28]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xb82c92f8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '2' for queue: 'JMS.TimerClientPool'" daemon prio=5 tid=0x010dd218 nid=0x45 in Object.wait() [96a7f000..96a7fc28]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xb82c9378> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '1' for queue: 'JMS.TimerClientPool'" daemon prio=5 tid=0x00bb1c98 nid=0x44 in Object.wait() [96b7f000..96b7fc28]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xb82c93f8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '0' for queue: 'JMS.TimerClientPool'" daemon prio=5 tid=0x00979ac8 nid=0x43 in Object.wait() [96c7f000..96c7fc28]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xb82c9478> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "Thread-12" daemon prio=5 tid=0x00200610 nid=0x42 waiting on condition [96d7f000..96d7fc28]
         at java.lang.Thread.sleep(Native Method)
         at com.geico.log.HeartbeatLogger$1.run(HeartbeatLogger.java:133)
    "Thread-10" daemon prio=5 tid=0x00200470 nid=0x41 waiting on condition [96e7f000..96e7fc28]
         at java.lang.Thread.sleep(Native Method)
         at com.geico.log.StreamLogger.pause(StreamLogger.java:267)
         at com.geico.log.StreamLogger.access$300(StreamLogger.java:19)
         at com.geico.log.StreamLogger$WorkerThread.run(StreamLogger.java:221)
         at java.lang.Thread.run(Thread.java:534)
    "ClockDaemonThread-0" daemon prio=5 tid=0x00ab6a48 nid=0x40 in Object.wait() [96f7f000..96f7fc28]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xb8051b20> (a com.geico.util.concurrent.ClockDaemon)
         at com.geico.util.concurrent.ClockDaemon.nextTask(ClockDaemon.java:328)
         - locked <0xb8051b20> (a com.geico.util.concurrent.ClockDaemon)
         at com.geico.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:362)
         at java.lang.Thread.run(Thread.java:534)
    "Thread-9" prio=5 tid=0x008912e0 nid=0x3f in Object.wait() [9ad7f000..9ad7fc28]
         at java.lang.Object.wait(Native Method)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <0xb8051bc0> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "ExecuteThread: '0' for queue: 'JMS.TimerTreePool'" daemon prio=5 tid=0x001d7fa8 nid=0x3e in Object.wait() [9ae7f000..9ae7fc28]
         at java.lang.Object.wait(Native Method)
         at weblogic.jms.backend.BETimerTree.execute(BETimerTree.java:146)
         - locked <0xb8051c90> (a weblogic.jms.backend.BETimerTree)
         at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)
    "Thread-8" prio=5 tid=0x00247418 nid=0x3d in Object.wait() [9af7f000..9af7fc28]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xb8051d20> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <0xb8051d20> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "Thread-7" prio=5 tid=0x001d8a28 nid=0x3c in Object.wait() [9b07f000..9b07fc28]
         at java.lang.Object.wait(Native Method)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <0xb8051dc0> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "ExecuteThread: '0' for queue: 'weblogic.cluster.MulticastManager'" daemon prio=5 tid=0x00da0a60 nid=0x3b runnable [9b17f000..9b17fc28]
         at java.net.PlainDatagramSocketImpl.receive(Native Method)
         at java.net.DatagramSocket.receive(DatagramSocket.java:711)
         - locked <0xa22dc3e8> (a java.net.DatagramPacket)
         - locked <0xb7558130> (a java.net.MulticastSocket)
         at weblogic.cluster.FragmentSocket.receive(FragmentSocket.java:169)
         at weblogic.cluster.MulticastManager.execute(MulticastManager.java:395)
         at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)
    "SSLListenThread.Administrator" prio=5 tid=0x00c7db08 nid=0x3a runnable [9b27f000..9b27fc28]
         at java.net.PlainSocketImpl.socketAccept(Native Method)
         at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
         - locked <0xb75582a0> (a java.net.PlainSocketImpl)
         at java.net.ServerSocket.implAccept(ServerSocket.java:448)
         at javax.net.ssl.impl.SSLServerSocketImp

    Hi,
    I guess this issue is related to your connectivity and hyperion services. Can you take a restart and try to execute the reports. Also try to check all your Hyperion services are up and running before executing the FR Reports. You can also check the Reporting services by login to Weblogic server (if you have applied it in your environment). In weblogic it should be in Running mode and not in Admin mode. If it is not in Running mode then try to check the Data Source setting from Connection Pool in Weblogic Server.
    Soumya

  • Page stuck while load and leads to stuck thread

    Hi
    I am facing an issue from a long time.
    When i run the page of my application, when it opens for first time it get stuck and get loading and take very much time and after that it leads to stuck thread issue. All the component of my page opens but there is a table which gets value from OIM APIs also loads for the fiirst time. As in coding i have done it will pass null value to the table and wont fetch data for the first time. it will only fetch when value will be passed in a text box and submit was hit.
    So my page gets hanged or loading in the table section everytime when it loads. for that i have to login in weblogic server, run the workmanager, forcefully shut down the server instance and run the page again. But sometime that doesnt work too.
    Please help its a bug trouble for me and creating a huge problem.
    Thanks
    Sudeep

    I'm not sure I understand your issue - but if you want to delay the fetching of information into the table until you actually have a value to pass to it see:
    http://blogs.oracle.com/shay/2010/08/delay_method_execution_when_us.html

  • Missing listener-threads attribute in orion-ejb-jar.xml !!

    Hi All,
    I was trying to do some performance tuning as given in BPEL admin guide. For one of the BPEL property namely 'dspMaxThreads', guide says "Sum of dspMaxThreads of ALL domains should be <= the number of mdb listener threads on the workerbean". Now when i open <SOA_HOME>/j2ee/oc4j_soa/application-deployments/orabpel/orion-ejb-jar.xml and navigate to workerbean, i can not find any listener-threads attribute in message-driven-deployment tag.
    <message-driven-deployment name="WorkerBean" min-instances="100" resource-adapter="BPELjms">
    However there is a ReceiverThreads property set to 40.
    <message-driven-deployment name="WorkerBean" min-instances="100" resource-adapter="BPELjms">
    <config-property-name>ReceiverThreads</config-property-name>
    <config-property-value>40</config-property-value>
    </config-property>
    </message-driven-deployment>
    Now my question is that can i add listener-threads attribute to message-driven-deployment tag and then tune dspmaxthreads accordingly, Or
    i can make use of ReceiverThreads property to match with dspmaxthreads property. I am really confused b/w listener-threads and ReceiverThreads.
    Can anyone clarify the same ?
    I have ORACLE SOA Suite ver 10.1.3.1 (with 10.1.3.3 patch) installed.
    Thanks in advance.

    Jingzhi -- If you define the correct EJB-QL you should get the correct SQL generated for the finder. If that's not happening can you verify it against the OC4J v903 production release and see if it is still happening. I don't know if this helps with the JDeveloper problem specifically but if you don't need to create your own finders then that I hope should help.
    Thanks -- Jeff

  • Weblogic 10.3.4 stuck thread after attempting to passify ApplicationModule

    Using ADF 11.1.1.4
    Is there any reason that Weblogic should ever end up with a stuck thread when attempting to passify an application module. See the stack trace below.
    There is no indication of what business components if any were responsible. It goes right up to the panel stretch layout and jsp template.
    ]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
      oracle.jbo.common.Base64$Base64OutputStream.getDeflatedBytes(Base64.java:135)
      oracle.jbo.common.Base64$Base64OutputStream.toString(Base64.java:116)
      oracle.jbo.common.VariableValueManagerImpl.writeVarsToXML(VariableValueManagerImpl.java:819)
      oracle.jbo.server.ViewObjectImpl.doPassivateSettings(ViewObjectImpl.java:19100)
      oracle.jbo.server.ApplicationModuleImpl.passivateVOs(ApplicationModuleImpl.java:7712)
      oracle.jbo.server.ApplicationModuleImpl.doPassivateState(ApplicationModuleImpl.java:7522)
      oracle.jbo.server.ApplicationModuleImpl.doPassivateAMState(ApplicationModuleImpl.java:7509)
      oracle.jbo.server.Serializer.passivate(Serializer.java:213)
      oracle.jbo.server.DBSerializer.passivateRootAM(DBSerializer.java:290)
      oracle.jbo.server.DBSerializer.passivateRootAM(DBSerializer.java:267)
      oracle.jbo.server.ApplicationModuleImpl.passivateStateInternal(ApplicationModuleImpl.java:5913)
      oracle.jbo.server.ApplicationModuleImpl.passivateState(ApplicationModuleImpl.java:5783)
      oracle.jbo.server.ApplicationPoolMessageHandler.doPoolMessage(ApplicationPoolMessageHandler.java:381)
      oracle.jbo.server.ApplicationModuleImpl.doPoolMessage(ApplicationModuleImpl.java:8933)
      oracle.jbo.common.ampool.ApplicationPoolImpl.sendPoolMessage(ApplicationPoolImpl.java:4496)
      oracle.jbo.common.ampool.ApplicationPoolImpl.manageReferencingState(ApplicationPoolImpl.java:1445)
      oracle.jbo.common.ampool.ApplicationPoolImpl.recycleReferencedInstance(ApplicationPoolImpl.java:2544)
      oracle.jbo.common.ampool.ApplicationPoolImpl.prepareApplicationModule(ApplicationPoolImpl.java:2365)
      oracle.jbo.common.ampool.ApplicationPoolImpl.doCheckout(ApplicationPoolImpl.java:2270)
      oracle.jbo.common.ampool.ApplicationPoolImpl.useApplicationModule(ApplicationPoolImpl.java:3168)
      oracle.jbo.common.ampool.SessionCookieImpl.useApplicationModule(SessionCookieImpl.java:460)
      oracle.jbo.http.HttpSessionCookieImpl.useApplicationModule(HttpSessionCookieImpl.java:234)
      oracle.jbo.common.ampool.SessionCookieImpl.useApplicationModule(SessionCookieImpl.java:431)
      oracle.jbo.common.ampool.SessionCookieImpl.useApplicationModule(SessionCookieImpl.java:426)
      oracle.adf.model.bc4j.DCJboDataControl.initializeApplicationModule(DCJboDataControl.java:516)
      oracle.adf.model.bc4j.DCJboDataControl.getApplicationModule(DCJboDataControl.java:862)
      oracle.adf.model.binding.DCDataControl.setErrorHandler(DCDataControl.java:483)
      oracle.jbo.uicli.binding.JUApplication.setErrorHandler(JUApplication.java:261)
      oracle.adf.model.BindingContext.put(BindingContext.java:1300)
      oracle.adf.model.binding.DCDataControlReference.getDataControl(DCDataControlReference.java:174)
      oracle.adf.model.BindingContext.instantiateDataControl(BindingContext.java:1024)
      oracle.adf.model.dcframe.DataControlFrameImpl.doFindDataControl(DataControlFrameImpl.java:1282)
      oracle.adf.model.dcframe.DataControlFrameImpl.internalFindDataControl(DataControlFrameImpl.java:1186)
      oracle.adf.model.dcframe.DataControlFrameImpl.findDataControl(DataControlFrameImpl.java:1146)
      oracle.adf.model.BindingContext.internalFindDataControl(BindingContext.java:1149)
      oracle.adf.model.BindingContext.get(BindingContext.java:1102)
      oracle.adf.model.binding.DCParameter.evaluateValue(DCParameter.java:82)
      oracle.adf.model.binding.DCParameter.getValue(DCParameter.java:111)
      oracle.adf.model.binding.DCBindingContainer.getChildByName(DCBindingContainer.java:2708)
      oracle.adf.model.binding.DCBindingContainer.internalGet(DCBindingContainer.java:2756)
      oracle.adf.model.binding.DCExecutableBinding.get(DCExecutableBinding.java:115)
      oracle.adf.model.binding.DCUtil.findSpelObject(DCUtil.java:328)
      oracle.adf.model.binding.DCBindingContainer.evaluateParameterWithElCheck(DCBindingContainer.java:1460)
      oracle.adf.model.binding.DCBindingContainer.findDataControl(DCBindingContainer.java:1590)
      oracle.adf.model.binding.DCIteratorBinding.initDataControl(DCIteratorBinding.java:2470)
      oracle.adf.model.binding.DCIteratorBinding.getDataControl(DCIteratorBinding.java:2414)
      oracle.adf.model.binding.DCIteratorBinding.refresh(DCIteratorBinding.java:4313)
      oracle.adf.model.binding.DCExecutableBinding.refreshIfNeeded(DCExecutableBinding.java:341)
      oracle.adf.model.binding.DCIteratorBinding.getAllRowsInRange(DCIteratorBinding.java:2337)
      sun.reflect.GeneratedMethodAccessor437.invoke(Unknown Source)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      java.lang.reflect.Method.invoke(Method.java:597)
      oracle.jbo.common.JboBeanUtils.getProperty(JboBeanUtils.java:113)
      oracle.jbo.common.JboAbstractMap.internalGet(JboAbstractMap.java:97)
      oracle.adf.model.binding.DCExecutableBinding.internalGet(DCExecutableBinding.java:169)
      oracle.adf.model.binding.DCIteratorBinding.internalGet(DCIteratorBinding.java:4558)
      oracle.adf.model.binding.DCExecutableBinding.get(DCExecutableBinding.java:115)
      javax.el.MapELResolver.getValue(MapELResolver.java:164)
      com.sun.faces.el.DemuxCompositeELResolver._getValue(DemuxCompositeELResolver.java:173)
      com.sun.faces.el.DemuxCompositeELResolver.getValue(DemuxCompositeELResolver.java:200)
      com.sun.el.parser.AstValue.getValue(Unknown Source)
      com.sun.el.ValueExpressionImpl.getValue(Unknown Source)
      org.apache.myfaces.trinidad.bean.FacesBeanImpl.getProperty(FacesBeanImpl.java:68)
      org.apache.myfaces.trinidad.component.UIXComponentBase.getProperty(UIXComponentBase.java:1194)
      org.apache.myfaces.trinidad.component.UIXIterator.getValue(UIXIterator.java:706)
      org.apache.myfaces.trinidad.component.UIXCollection._flushCachedModel(UIXCollection.java:1386)
      org.apache.myfaces.trinidad.component.UIXCollection.__processFlattenedChildrenBegin(UIXCollection.java:1369)
      org.apache.myfaces.trinidad.component.UIXIterator.processFlattenedChildren(UIXIterator.java:156)
      org.apache.myfaces.trinidad.component.UIXComponent.processFlattenedChildren(UIXComponent.java:160)
      org.apache.myfaces.trinidad.component.UIXComponent.processFlattenedChildren(UIXComponent.java:290)
      org.apache.myfaces.trinidad.component.UIXComponent.processFlattenedChildren(UIXComponent.java:213)
      oracle.adfinternal.view.faces.renderkit.rich.PanelGroupLayoutRenderer.encodeAll(PanelGroupLayoutRenderer.java:315)
      oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1396)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:335)
      org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:767)
      javax.faces.component.UIComponent.encodeAll(UIComponent.java:937)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:399)
      oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2633)
      oracle.adfinternal.view.faces.renderkit.rich.ShowDetailItemRenderer.access$500(ShowDetailItemRenderer.java:31)
      oracle.adfinternal.view.faces.renderkit.rich.ShowDetailItemRenderer$ChildEncoderCallback.processComponent(ShowDetailItemRenderer.java:531)
      oracle.adfinternal.view.faces.renderkit.rich.ShowDetailItemRenderer$ChildEncoderCallback.processComponent(ShowDetailItemRenderer.java:504)
      org.apache.myfaces.trinidad.component.UIXComponent.processFlattenedChildren(UIXComponent.java:170)
      org.apache.myfaces.trinidad.component.UIXComponent.processFlattenedChildren(UIXComponent.java:290)
      org.apache.myfaces.trinidad.component.UIXComponent.encodeFlattenedChildren(UIXComponent.java:255)
      oracle.adfinternal.view.faces.renderkit.rich.ShowDetailItemRenderer._encodeChildren(ShowDetailItemRenderer.java:446)
      oracle.adfinternal.view.faces.renderkit.rich.ShowDetailItemRenderer.encodeAll(ShowDetailItemRenderer.java:115)
      oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1396)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:335)
      org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:767)
      javax.faces.component.UIComponent.encodeAll(UIComponent.java:937)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:399)
      oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2633)
      oracle.adf.view.rich.render.RichRenderer.encodeStretchedChild(RichRenderer.java:2004)
      oracle.adfinternal.view.faces.renderkit.rich.PanelAccordionRenderer._encodeNonScreenReaderDetailItem(PanelAccordionRenderer.java:1422)
      oracle.adfinternal.view.faces.renderkit.rich.PanelAccordionRenderer._encodeDetailItem(PanelAccordionRenderer.java:1483)
      oracle.adfinternal.view.faces.renderkit.rich.PanelAccordionRenderer.access$4800(PanelAccordionRenderer.java:52)
      oracle.adfinternal.view.faces.renderkit.rich.PanelAccordionRenderer$PanelEncoderCallback.processComponent(PanelAccordionRenderer.java:2503)
      oracle.adfinternal.view.faces.renderkit.rich.PanelAccordionRenderer$PanelEncoderCallback.processComponent(PanelAccordionRenderer.java:2447)
      org.apache.myfaces.trinidad.component.UIXComponent.processFlattenedChildren(UIXComponent.java:170)
      org.apache.myfaces.trinidad.component.UIXComponent.processFlattenedChildren(UIXComponent.java:290)
      org.apache.myfaces.trinidad.component.UIXComponent.encodeFlattenedChildren(UIXComponent.java:255)
      oracle.adfinternal.view.faces.renderkit.rich.PanelAccordionRenderer.encodeAll(PanelAccordionRenderer.java:456)
      oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1396)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:335)
      org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:767)
      javax.faces.component.UIComponent.encodeAll(UIComponent.java:937)
      oracle.adfinternal.view.faces.taglib.region.IncludeTag$FacetWrapper.encodeAll(IncludeTag.java:536)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:399)
      oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2633)
      oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer.encodeCenterFacet(PanelStretchLayoutRenderer.java:769)
      oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer._encodeCenterPane(PanelStretchLayoutRenderer.java:1140)
      oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer._encodeMiddlePanes(PanelStretchLayoutRenderer.java:348)
      oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer.encodeAll(PanelStretchLayoutRenderer.java:313)
      oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1396)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:335)
      org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:767)
      javax.faces.component.UIComponent.encodeAll(UIComponent.java:937)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:399)
      oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2633)
      oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer.encodeStartFacet(PanelStretchLayoutRenderer.java:805)
      oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer._encodeVerticalPane(PanelStretchLayoutRenderer.java:1369)
      oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer._encodeMiddlePanes(PanelStretchLayoutRenderer.java:347)
      oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer.encodeAll(PanelStretchLayoutRenderer.java:313)
      oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1396)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:335)
      org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:767)
      javax.faces.component.UIComponent.encodeAll(UIComponent.java:937)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:399)
      oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2633)
      oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer.encodeCenterFacet(PanelStretchLayoutRenderer.java:769)
      oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer._encodeCenterPane(PanelStretchLayoutRenderer.java:1140)
      oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer._encodeMiddlePanes(PanelStretchLayoutRenderer.java:348)
      oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer.encodeAll(PanelStretchLayoutRenderer.java:313)
      oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1396)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:335)
      org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:767)
      javax.faces.component.UIComponent.encodeAll(UIComponent.java:937)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:399)
      oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2633)
      oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer.encodeCenterFacet(PanelStretchLayoutRenderer.java:769)
      oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer._encodeCenterPane(PanelStretchLayoutRenderer.java:1140)
      oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer._encodeMiddlePanes(PanelStretchLayoutRenderer.java:348)
      oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer.encodeAll(PanelStretchLayoutRenderer.java:313)
      oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1396)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:335)
      org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:767)
      javax.faces.component.UIComponent.encodeAll(UIComponent.java:937)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:399)
      oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2633)
      oracle.adf.view.rich.render.RichRenderer.encodeStretchedChild(RichRenderer.java:2004)
      oracle.adfinternal.view.faces.renderkit.rich.PanelSplitterRenderer._renderPane(PanelSplitterRenderer.java:1353)
      oracle.adfinternal.view.faces.renderkit.rich.PanelSplitterRenderer.encodeAll(PanelSplitterRenderer.java:259)
      oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1396)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:335)
      org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:767)
      javax.faces.component.UIComponent.encodeAll(UIComponent.java:937)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:399)
      oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2633)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeAllChildren(CoreRenderer.java:415)
      oracle.adfinternal.view.faces.renderkit.rich.PageTemplateRenderer.encodeAll(PageTemplateRenderer.java:69)
      oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1396)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:335)
      org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:767)
      oracle.adf.view.rich.component.fragment.ContextSwitchingComponent.encodeEnd(ContextSwitchingComponent.java:155)
      javax.faces.component.UIComponent.encodeAll(UIComponent.java:937)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:399)
      oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2633)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeAllChildren(CoreRenderer.java:415)
      oracle.adfinternal.view.faces.renderkit.rich.FormRenderer.encodeAll(FormRenderer.java:220)
      oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1396)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:335)
      org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:767)
      javax.faces.component.UIComponent.encodeAll(UIComponent.java:937)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:399)
      oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2633)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeAllChildren(CoreRenderer.java:415)
      oracle.adfinternal.view.faces.renderkit.rich.DocumentRenderer.encodeAll(DocumentRenderer.java:1181)
      oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1396)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:335)
      org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:767)
      javax.faces.component.UIComponent.encodeAll(UIComponent.java:937)
      javax.faces.component.UIComponent.encodeAll(UIComponent.java:933)
      com.sun.faces.application.ViewHandlerImpl.doRenderView(ViewHandlerImpl.java:266)
      com.sun.faces.application.ViewHandlerImpl.renderView(ViewHandlerImpl.java:197)
      javax.faces.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:189)
      org.apache.myfaces.trinidadinternal.application.ViewHandlerImpl.renderView(ViewHandlerImpl.java:193)
      oracle.adfinternal.view.faces.lifecycle.LifecycleImpl._renderResponse(LifecycleImpl.java:777)
      oracle.adfinternal.view.faces.lifecycle.LifecycleImpl._executePhase(LifecycleImpl.java:293)
      oracle.adfinternal.view.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:213)
      javax.faces.webapp.FacesServlet.service(FacesServlet.java:266)
      weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
      weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
      weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:300)
      weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
      weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
      oracle.adf.model.servlet.ADFBindingFilter.doFilter(ADFBindingFilter.java:205)
      weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
      oracle.adfinternal.view.faces.webapp.rich.RegistrationFilter.doFilter(RegistrationFilter.java:106)
      org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:446)
      oracle.adfinternal.view.faces.activedata.AdsFilter.doFilter(AdsFilter.java:60)
      org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:446)
      org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl._doFilterImpl(TrinidadFilterImpl.java:271)
      org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl.doFilter(TrinidadFilterImpl.java:177)
      org.apache.myfaces.trinidad.webapp.TrinidadFilter.doFilter(TrinidadFilter.java:92)
      weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
      oracle.adf.library.webapp.LibraryFilter.doFilter(LibraryFilter.java:175)
      weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
      oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:111)
      java.security.AccessController.doPrivileged(Native Method)
      oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:313)
      oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:413)
      oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:94)
      oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:161)
      oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:71)
      weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
      oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:136)
      weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
      weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:27)
      weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
      weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3715)
      weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3681)
      weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
      weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
      weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2277)
      weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2183)
      weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1454)
      weblogic.work.ExecuteThread.execute(ExecuteThread.java:207)
      weblogic.work.ExecuteThread.run(ExecuteThread.java:176)
    SUBSYSTEM = WebLogicServer USERID = <WLS Kernel> SEVERITY = Error THREAD = [ACTIVE] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)' MSGID = BEA-000337 MACHINE = app-sahe.dmz TXID =  CONTEXTID = ed3140ecffa37389:3c925abd:13f6031c9fe:-8000-0000000000015dff TIMESTAMP = 1371715760129 
    WatchAlarmType: AutomaticReset
    WatchAlarmResetPeriod: 600000

    Was their any changes in any of configuration files or something else. If not, then try bouncing the system once again and see if its working.

  • Help on stuck thread

    Hi, From my server log I noticed that there are some stuck threads. After looked at the thread dump thread 18 is blocking. MainPortalServlet.java is extends PortalServlet and line 96 is super.service(request,response); Please help me to analyse why it is blocking. Thanks.
    "ExecuteThread: '20' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x0090e6b0 nid=0x3b waiting for monitor entry [0x600fd000..0x600ffc28]
         at com.bea.wlw.netui.script.el.util.ParseUtils.parse(ParseUtils.java:100)
         - waiting to lock <0x8d3a3f18> (a com.bea.wlw.netui.script.el.util.ParseUtils$Cache)
         at com.bea.wlw.netui.script.el.ExpressionEvaluatorImpl.containsExpression(ExpressionEvaluatorImpl.java:239)
         at com.bea.wlw.netui.pageflow.ProcessPopulate.populate(ProcessPopulate.java:295)
         at com.bea.wlw.netui.pageflow.PageFlowRequestProcessor.processPopulate(PageFlowRequestProcessor.java:402)
         at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:254)
         at com.bea.wlw.netui.pageflow.PageFlowRequestProcessor.process(PageFlowRequestProcessor.java:691)
         at com.bea.wlw.netui.pageflow.AutoRegisterActionServlet.process(AutoRegisterActionServlet.java:527)
         at com.bea.wlw.netui.pageflow.PageFlowActionServlet.process(PageFlowActionServlet.java:152)
         at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:507)
         at com.bea.wlw.netui.pageflow.PageFlowUtils.strutsLookup(PageFlowUtils.java:1786)
         at com.bea.portlet.adapter.scopedcontent.ScopedContentCommonSupport.executeAction(ScopedContentCommonSupport.java:561)
         at com.bea.portlet.adapter.scopedcontent.ScopedContentCommonSupport.processActionInternal(ScopedContentCommonSupport.java:121)
         at com.bea.portlet.adapter.scopedcontent.PageFlowStubImpl.processAction(PageFlowStubImpl.java:98)
         at com.bea.netuix.servlets.controls.content.NetuiContent.raiseScopedAction(NetuiContent.java:154)
         at com.bea.netuix.servlets.controls.content.NetuiContent.raiseScopedAction(NetuiContent.java:113)
         at com.bea.netuix.servlets.controls.content.NetuiContent.handlePostbackData(NetuiContent.java:228)
         at com.bea.netuix.nf.ControlLifecycle$3.visit(ControlLifecycle.java:171)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:356)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:126)
         at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:105)
         at com.bea.netuix.nf.Lifecycle.runInbound(Lifecycle.java:173)
         at com.bea.netuix.nf.Lifecycle.run(Lifecycle.java:137)
         at com.bea.netuix.servlets.manager.UIServlet.runLifecycle(UIServlet.java:324)
         at com.bea.netuix.servlets.manager.UIServlet.processControlTree(UIServlet.java:220)
         at com.bea.netuix.servlets.manager.PortalServlet.doPost(PortalServlet.java:820)
         at com.bea.netuix.servlets.manager.UIServlet.service(UIServlet.java:150)
         at com.cd.servlets.portal.MainPortalServlet.service(MainPortalServlet.java:96)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
         at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
         at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:28)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
         at com.bea.p13n.servlets.PortalServletFilter.doFilter(PortalServletFilter.java:293)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
         at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:7053)
         at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
         at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
         at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3902)
         at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2773)
         at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)
    "ExecuteThread: '19' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x0090dab8 nid=0x3a runnable [0x601fb000..0x601ffc28]
         at java.util.HashMap.getEntry(HashMap.java:361)
         at java.util.LinkedHashMap.get(LinkedHashMap.java:266)
         at com.bea.wlw.netui.script.el.util.ParseUtils.parse(ParseUtils.java:84)
         at com.bea.wlw.netui.script.el.util.ParseUtils.evaluate(ParseUtils.java:123)
         at com.bea.wlw.netui.script.el.ExpressionEvaluatorImpl.evaluateStrict(ExpressionEvaluatorImpl.java:87)
         at com.bea.wlw.netui.tags.AbstractBaseTag.evaluateExpressionInternal(AbstractBaseTag.java:573)
         at com.bea.wlw.netui.tags.AbstractBaseTag.evaluateExpression(AbstractBaseTag.java:362)
         at com.bea.wlw.netui.tags.databinding.script.GetData.doEndTag(GetData.java:249)
         at jsp_servlet._portlets.___portlet1._jspService(_portlet1.jsp:27)
         at weblogic.servlet.jsp.JspBase.service(JspBase.java:33)
         at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
         at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:28)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
         at com.bea.p13n.servlets.PortalServletFilter.doFilter(PortalServletFilter.java:293)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
         at com.bea.wlw.netui.pageflow.PageFlowJspFilter.doFilter(PageFlowJspFilter.java:250)
         - locked <0x78ccf6c8> (a portlets.Portlet1Controller)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
         at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:652)
         at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:431)
         at com.bea.wlw.netui.pageflow.scoping.internal.ScopedRequestDispatcher.include(ScopedRequestDispatcher.java:120)
         at com.bea.netuix.servlets.controls.content.JspContent.beginRender(JspContent.java:534)
         at com.bea.netuix.servlets.controls.content.NetuiContent.beginRender(NetuiContent.java:425)
         at com.bea.netuix.nf.ControlLifecycle$1.visit(ControlLifecycle.java:495)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:543)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:247)
         at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:105)
         at com.bea.netuix.nf.Lifecycle.run(Lifecycle.java:356)
         at com.bea.netuix.nf.UIControl.render(UIControl.java:536)
         at com.bea.netuix.servlets.controls.PresentationContext.render(PresentationContext.java:405)
         at com.bea.netuix.servlets.util.RenderToolkit.renderChild(RenderToolkit.java:123)
         at com.bea.netuix.servlets.jsp.taglib.RenderChild.doStartTag(RenderChild.java:58)
         at jsp_servlet._framework._skeletons._cd.__flowlayout._jspService(flowlayout.jsp:30)
         at weblogic.servlet.jsp.JspBase.service(JspBase.java:33)
         at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:348)
         at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:646)
         at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:431)
         at com.bea.netuix.servlets.controls.JspRenderer.renderAlt(JspRenderer.java:194)
         at com.bea.netuix.servlets.controls.JspRenderer.beginRender(JspRenderer.java:96)
         at com.bea.netuix.nf.ControlLifecycle$1.visit(ControlLifecycle.java:491)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:543)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:247)
         at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:105)
         at com.bea.netuix.nf.Lifecycle.run(Lifecycle.java:356)
         at com.bea.netuix.nf.UIControl.render(UIControl.java:536)
         at com.bea.netuix.servlets.controls.PresentationContext.render(PresentationContext.java:405)
         at com.bea.netuix.servlets.util.RenderToolkit.renderChild(RenderToolkit.java:123)
         at com.bea.netuix.servlets.jsp.taglib.RenderChild.doStartTag(RenderChild.java:58)
         at jsp_servlet._framework._skeletons._cd.__flowlayout._jspService(flowlayout.jsp:30)
         at weblogic.servlet.jsp.JspBase.service(JspBase.java:33)
         at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:348)
         at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:646)
         at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:431)
         at com.bea.netuix.servlets.controls.JspRenderer.renderAlt(JspRenderer.java:194)
         at com.bea.netuix.servlets.controls.JspRenderer.beginRender(JspRenderer.java:96)
         at com.bea.netuix.nf.ControlLifecycle$1.visit(ControlLifecycle.java:491)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:543)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:247)
         at com.bea.netuix.nf.Lifecycle.runOutbound(Lifecycle.java:204)
         at com.bea.netuix.nf.Lifecycle.run(Lifecycle.java:153)
         at com.bea.netuix.servlets.manager.UIServlet.runLifecycle(UIServlet.java:324)
         at com.bea.netuix.servlets.manager.UIServlet.processControlTree(UIServlet.java:220)
         at com.bea.netuix.servlets.manager.PortalServlet.doPost(PortalServlet.java:820)
         at com.bea.netuix.servlets.manager.PortalServlet.doGet(PortalServlet.java:671)
         at com.bea.netuix.servlets.manager.UIServlet.service(UIServlet.java:147)
         at com.cd.servlets.portal.MainPortalServlet.service(MainPortalServlet.java:116)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
         at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
         at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:28)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
         at com.bea.p13n.servlets.PortalServletFilter.doFilter(PortalServletFilter.java:293)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
         at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:7053)
         at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
         at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
         at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3902)
         at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2773)
         at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)
    "ExecuteThread: '18' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x0090cec0 nid=0x39 runnable [0x602fd000..0x602ffc28]
         at java.util.LinkedHashMap.transfer(LinkedHashMap.java:224)
         at java.util.HashMap.resize(HashMap.java:452)
         at java.util.LinkedHashMap.addEntry(LinkedHashMap.java:399)
         at java.util.HashMap.put(HashMap.java:392)
         at com.bea.wlw.netui.script.el.util.ParseUtils.parse(ParseUtils.java:100)
         - locked <0x8d3a3f18> (a com.bea.wlw.netui.script.el.util.ParseUtils$Cache)
         at com.bea.wlw.netui.script.el.ExpressionEvaluatorImpl.containsExpression(ExpressionEvaluatorImpl.java:239)
         at com.bea.wlw.netui.pageflow.ProcessPopulate.populate(ProcessPopulate.java:295)
         at com.bea.wlw.netui.pageflow.PageFlowRequestProcessor.processPopulate(PageFlowRequestProcessor.java:402)
         at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:254)
         at com.bea.wlw.netui.pageflow.PageFlowRequestProcessor.process(PageFlowRequestProcessor.java:691)
         at com.bea.wlw.netui.pageflow.AutoRegisterActionServlet.process(AutoRegisterActionServlet.java:527)
         at com.bea.wlw.netui.pageflow.PageFlowActionServlet.process(PageFlowActionServlet.java:152)
         at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:507)
         at com.bea.wlw.netui.pageflow.PageFlowUtils.strutsLookup(PageFlowUtils.java:1786)
         at com.bea.portlet.adapter.scopedcontent.ScopedContentCommonSupport.executeAction(ScopedContentCommonSupport.java:561)
         at com.bea.portlet.adapter.scopedcontent.ScopedContentCommonSupport.processActionInternal(ScopedContentCommonSupport.java:121)
         at com.bea.portlet.adapter.scopedcontent.PageFlowStubImpl.processAction(PageFlowStubImpl.java:98)
         at com.bea.netuix.servlets.controls.content.NetuiContent.raiseScopedAction(NetuiContent.java:154)
         at com.bea.netuix.servlets.controls.content.NetuiContent.raiseScopedAction(NetuiContent.java:113)
         at com.bea.netuix.servlets.controls.content.NetuiContent.handlePostbackData(NetuiContent.java:228)
         at com.bea.netuix.nf.ControlLifecycle$3.visit(ControlLifecycle.java:171)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:356)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:126)
         at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:105)
         at com.bea.netuix.nf.Lifecycle.runInbound(Lifecycle.java:173)
         at com.bea.netuix.nf.Lifecycle.run(Lifecycle.java:137)
         at com.bea.netuix.servlets.manager.UIServlet.runLifecycle(UIServlet.java:324)
         at com.bea.netuix.servlets.manager.UIServlet.processControlTree(UIServlet.java:220)
         at com.bea.netuix.servlets.manager.PortalServlet.doPost(PortalServlet.java:820)
         at com.bea.netuix.servlets.manager.UIServlet.service(UIServlet.java:150)
         at com.cd.servlets.portal.MainPortalServlet.service(MainPortalServlet.java:96)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
         at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
         at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:28)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
         at com.bea.p13n.servlets.PortalServletFilter.doFilter(PortalServletFilter.java:293)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
         at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:7053)
         at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
         at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
         at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3902)
         at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2773)
         at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)

    Hi user9503787,
    Can you please confirm if the Weblogic instance / java process you took the Thread Dump from was showing high CPU utilization?
    Your Thread Dump data is quite interesting and a perfect example of a non Thread safe HashMap combined with a read/write race condition:
    Thread #23 (hanging for 600 seconds+) is showing a potential infinite looping condition in at java.util.HashMap.getEntry(HashMap.java:361)
    Thread #22 (hanging for 600 seconds+) is showing a potential infinite looping condition while performing a resize() following a put() operation
    Weblogic Portal has some known issues which regards to usage of non Thread safe HashHap data structures; which under load can cause infinite looping and sharp surge of CPU (due to corruption of the internal HashMap index preventing exiting the internal loop); ultimately hanging the whole JVM.
    We had the same issue for one of our Weblogic Portal 10.0 production environment and Oracle provided a patch (via Collections.synchronizedMap() ).
    Please open a SR and Oracle, share these 2 Threads and discuss with the engineer about a patch / know issues regarding non Thread safe HashMap and infinite looping.
    Regards,
    P-H
    http://javaeesupportpatterns.blogspot.com/

  • Stuck threads reading blob column from db table

    WLS 10.3.5, JDK 1.6u29, Oracle 11g RAC, ojdbc6 latest driver
    We're having problems with stuck threads trying to read a blob column from a DB table. The query to extract the blob is a simple select, without any locking such as "for update" clauses or whatever. The blob's size is <= 100k.
    The thread dump shows the following stack trace:
    +"[STUCK] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" RUNNABLE native+
    +     java.net.SocketInputStream.socketRead0(Native Method)+
    +     java.net.SocketInputStream.read(SocketInputStream.java:129)+
    +     oracle.net.ns.Packet.receive(Packet.java:300)+
    +     oracle.net.ns.DataPacket.receive(DataPacket.java:106)+
    +     oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:315)+
    +     oracle.net.ns.NetInputStream.read(NetInputStream.java:260)+
    +     oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:105)+
    +     oracle.jdbc.driver.T4CMAREngine.getNBytes(T4CMAREngine.java:1517)+
    +     oracle.jdbc.driver.T4C8TTILobd.unmarshalLobData(T4C8TTILobd.java:476)+
    +     oracle.jdbc.driver.T4C8TTILob.readLOBD(T4C8TTILob.java:770)+
    +     oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:361)+
    +     oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)+
    +     oracle.jdbc.driver.T4C8TTILob.read(T4C8TTILob.java:146)+
    +     oracle.jdbc.driver.T4CConnection.getBytes(T4CConnection.java:2392)+
    +     oracle.sql.BLOB.getBytes(BLOB.java:348)+
    +     oracle.sql.BLOB.getBytes(BLOB.java:222)+
    +     weblogic.jdbc.wrapper.Blob_oracle_sql_BLOB.getBytes(Unknown Source)+
    +     com.ibatis.sqlmap.engine.type.BlobTypeHandlerCallback.getResult(BlobTypeHandlerCallback.java:33)+
    +     com.ibatis.sqlmap.engine.type.CustomTypeHandler.getResult(CustomTypeHandler.java:52)+
    +     com.ibatis.sqlmap.engine.mapping.result.ResultMap.getPrimitiveResultMappingValue(ResultMap.java:619)+
    +     com.ibatis.sqlmap.engine.mapping.result.ResultMap.getResults(ResultMap.java:345)+
    +     com.ibatis.sqlmap.engine.execution.SqlExecutor.handleResults(SqlExecutor.java:384)+
    +     com.ibatis.sqlmap.engine.execution.SqlExecutor.handleMultipleResults(SqlExecutor.java:300)+
    +     com.ibatis.sqlmap.engine.execution.SqlExecutor.executeQuery(SqlExecutor.java:189)+
    +     com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.sqlExecuteQuery(MappedStatement.java:221)+
    +     com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.executeQueryWithCallback(MappedStatement.java:189)+
    +     com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.executeQueryForList(MappedStatement.java:139)+
    +     com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryForList(SqlMapExecutorDelegate.java:567)+
    +     com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryForList(SqlMapExecutorDelegate.java:541)+
    +     com.ibatis.sqlmap.engine.impl.SqlMapSessionImpl.queryForList(SqlMapSessionImpl.java:118)+
    +     com.ibatis.sqlmap.engine.impl.SqlMapClientImpl.queryForList(SqlMapClientImpl.java:94)+
    +     com.ibatis.dao.client.template.SqlMapDaoTemplate.queryForList(SqlMapDaoTemplate.java:282)"[STUCK] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" RUNNABLE native+
    +     java.net.SocketInputStream.socketRead0(Native Method)+
    +     java.net.SocketInputStream.read(SocketInputStream.java:129)+
    +     oracle.net.ns.Packet.receive(Packet.java:300)+
    +     oracle.net.ns.DataPacket.receive(DataPacket.java:106)+
    +     oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:315)+
    +     oracle.net.ns.NetInputStream.read(NetInputStream.java:260)+
    +     oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:105)+
    +     oracle.jdbc.driver.T4CMAREngine.getNBytes(T4CMAREngine.java:1517)+
    +     oracle.jdbc.driver.T4C8TTILobd.unmarshalLobData(T4C8TTILobd.java:476)+
    +     oracle.jdbc.driver.T4C8TTILob.readLOBD(T4C8TTILob.java:770)+
    +     oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:361)+
    +     oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)+
    +     oracle.jdbc.driver.T4C8TTILob.read(T4C8TTILob.java:146)+
    +     oracle.jdbc.driver.T4CConnection.getBytes(T4CConnection.java:2392)+
    +     oracle.sql.BLOB.getBytes(BLOB.java:348)+
    +     oracle.sql.BLOB.getBytes(BLOB.java:222)+
    +     weblogic.jdbc.wrapper.Blob_oracle_sql_BLOB.getBytes(Unknown Source)+
    +     com.ibatis.sqlmap.engine.type.BlobTypeHandlerCallback.getResult(BlobTypeHandlerCallback.java:33)+
    +     com.ibatis.sqlmap.engine.type.CustomTypeHandler.getResult(CustomTypeHandler.java:52)+
    +     com.ibatis.sqlmap.engine.mapping.result.ResultMap.getPrimitiveResultMappingValue(ResultMap.java:619)+
    +     com.ibatis.sqlmap.engine.mapping.result.ResultMap.getResults(ResultMap.java:345)+
    +     com.ibatis.sqlmap.engine.execution.SqlExecutor.handleResults(SqlExecutor.java:384)+
    +     com.ibatis.sqlmap.engine.execution.SqlExecutor.handleMultipleResults(SqlExecutor.java:300)+
    +     com.ibatis.sqlmap.engine.execution.SqlExecutor.executeQuery(SqlExecutor.java:189)+
    +     com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.sqlExecuteQuery(MappedStatement.java:221)+
         com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.executeQueryWithCallback(MappedStatement.java:189)
    +     com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.executeQueryForList(MappedStatement.java:139)+
    +     com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryForList(SqlMapExecutorDelegate.java:567)+
    +     com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryForList(SqlMapExecutorDelegate.java:541)+
    +     com.ibatis.sqlmap.engine.impl.SqlMapSessionImpl.queryForList(SqlMapSessionImpl.java:118)+
    +     com.ibatis.sqlmap.engine.impl.SqlMapClientImpl.queryForList(SqlMapClientImpl.java:94)+
    +     com.ibatis.dao.client.template.SqlMapDaoTemplate.queryForList(SqlMapDaoTemplate.java:282)+
    Some threads eventually end (after 1-2 hours), most of them remain there for days.
    Any hint would be quite useful, thanks.

    Threads are executing the actual allocated request from the Weblogic Kernel. Most of the problems happen when the Thread execution is reaching the application or business layer.
    At this point your application Java code module is sending or receiving data from external sources such as a an Oracle database for example. Any problem with such external system will cause the Thread to hang and wait for data to come back.
    Other situations can occur such as internal deadlock, infinite looping, heavy IO contention on your server etc.
    Doesn't loo like a driver issue.
    http://docs.oracle.com/cd/E21764_01/doc.1111/e14770/weblogic_server_issues.htm#autoId2
    Check at the Database end
    Cheers ...

  • Login Failing Everytime - Stuck Threads

    Whenever I call Authentication.login(username, password, request) - I am getting
    a stuck thread. any ideas?
    Mar 31, 2004 8:26:50 AM PST> <Warning> <WebLogicServer> <BEA-000337> <ExecuteTh
    ead: '14' for queue: 'default' has been busy for "660" seconds working on the
    r
    quest "Http Request: /mySpt/sptdotcom.portal", which is more than the configure
    time (StuckThreadMaxTime) of "600" seconds.>
    Thanks
    Kunal

    Kunal,
    Download a copy of jprobe or something similar, that should point you at the problem...
    Steve
    "Kunal Mittal" <[email protected]> wrote:
    >
    Whenever I call Authentication.login(username, password, request) - I
    am getting
    a stuck thread. any ideas?
    Mar 31, 2004 8:26:50 AM PST> <Warning> <WebLogicServer> <BEA-000337>
    <ExecuteTh
    ead: '14' for queue: 'default' has been busy for "660" seconds working
    on the
    r
    quest "Http Request: /mySpt/sptdotcom.portal", which is more than the
    configure
    time (StuckThreadMaxTime) of "600" seconds.>
    Thanks
    Kunal

  • Detecting and Pinning a stuck thread

    While using WLS 7.0 there exists a problem where some execute threads get stuck (i.e don't respond) and do not release the associated resources such as connections, files etc. How does one detect such a stuck thread ? Also, how can we go about pinning stuck threads?

    "Haziq Pathan" <[email protected]> wrote in message news:3ffbfad4$[email protected]..
    While using WLS 7.0 there exists a problem where some execute threads
    get stuck (i.e don't respond) and do not release the associated resources
    such as connections, files etc. How does one detect such a stuck thread ?Basically there is a server health monitor thread that periodically checks
    threads and reports those who started execution and didn't finish it within
    configured time.
    Also, how can we go about pinning stuck threads?The best thing to do is to get a thread dump and see where the thread stuck,
    and deliver a fix for the problem.
    Hope this helps.
    Regards,
    Slava Imeshev

  • Problem Deploying Composites to Oracle SOA 11.1.1.5 (STUCK Thread Issue)

    Hi All,
    I got stuck thread issues when deploying composites to Oracle SOA 11.1.1.5 running on Windows Server 2008 R2. Even the SimpleApproval composite which ships with the standard installation is having trouble deploying.
    The composites are deploying fine in Oracle SOA 11.1.1.5 running on Windows XP.
    I'm not so good with thread dumps, so I post it below for rescue.
    Please advice,
    Rommel
    <Jul 18, 2011 10:15:25 AM AST> <Error> <WebLogicServer> <BEA-000337> <[STUCK] Ex
    ecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)' has been bus
    y for "605" seconds working on the request "weblogic.servlet.internal.ServletReq
    uestImpl@32d047d4[
    POST /soa-infra/deployer HTTP/1.1
    Connection: TE
    TE: trailers, deflate, gzip, compress
    User-Agent: Oracle HTTPClient Version 10h
    Accept-Encoding: gzip, x-gzip, compress, x-compress
    ECID-Context:
    Authorization: Basic bmFnaGl3ZWJsb2dpY3NvYTp3ZWJsb2dpY3NvYTEwIQ==
    Content-type: application/octet-stream
    Content-Length: 25876
    ]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds
    . Stack trace:
    Thread-55 "[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-
    tuning)'" <alive, in native, suspended, priority=1, DAEMON> {
    jrockit.net.SocketNativeIO.readBytesPinned(SocketNativeIO.java:???)
    jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:24)
    java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)
    java.net.SocketInputStream.read(SocketInputStream.java:107)
    oracle.net.nt.MetricsEnabledInputStream.read(TcpNTAdapter.java:707)
    oracle.net.ns.Packet.receive(Packet.java:243)
    oracle.net.ns.DataPacket.receive(DataPacket.java:106)
    oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:309)
    oracle.net.ns.NetInputStream.read(NetInputStream.java:254)
    oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrap
    per.java:93)
    oracle.jdbc.driver.T4CMAREngine.getNBytes(T4CMAREngine.java:1573)
    oracle.jdbc.driver.T4C8TTILobd.unmarshalLobData(T4C8TTILobd.java:360)
    oracle.jdbc.driver.T4C8TTILob.readLOBD(T4C8TTILob.java:787)
    oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:303)
    oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:181)
    oracle.jdbc.driver.T4C8TTILob.read(T4C8TTILob.java:139)
    oracle.jdbc.driver.T4CConnection.getBytes(T4CConnection.java:2690)
    ^-- Holding lock: oracle.jdbc.driver.T4CConnection@32386377[thin lock]
    oracle.sql.BLOB.getBytes(BLOB.java:557)
    oracle.jdbc.driver.OracleBlobInputStream.needBytes(OracleBlobInputStream.jav
    a:188)
    oracle.jdbc.driver.OracleBufferedStream.readInternal(OracleBufferedStream.ja
    va:171)
    oracle.jdbc.driver.OracleBufferedStream.read(OracleBufferedStream.java:151)
    ^-- Holding lock: oracle.jdbc.driver.OracleBlobInputStream@341d0b87[thin loc
    k]
    java.io.BufferedInputStream.fill(BufferedInputStream.java:189)
    java.io.BufferedInputStream.read(BufferedInputStream.java:236)
    ^-- Holding lock: oracle.mds.internal.persistence.db.DBInputStream@341d0bb2[
    thin lock]
    oracle.xml.parser.v2.XMLReader.pushXMLReader(XMLReader.java:363)
    oracle.xml.parser.v2.XMLReader.pushXMLReader(XMLReader.java:230)
    oracle.xml.parser.v2.XMLParser.parse(XMLParser.java:217)
    oracle.fabric.common.wsdl.SchemaBuilder.parseXMLSrc(SchemaBuilder.java:1016)
    oracle.fabric.common.wsdl.SchemaBuilder.parseXSDSrc(SchemaBuilder.java:984)
    oracle.fabric.common.wsdl.SchemaBuilder.processSchemaImportsAndInclude(Schem
    aBuilder.java:755)
    oracle.fabric.common.wsdl.SchemaBuilder.loadEmbeddedSchemas(SchemaBuilder.ja
    va:474)
    oracle.fabric.common.wsdl.SchemaBuilder.loadSchemasFromWSDL(SchemaBuilder.ja
    va:423)
    oracle.fabric.common.wsdl.SchemaBuilder.loadAllSchemas(SchemaBuilder.java:21
    4)
    oracle.fabric.common.wsdl.SchemaManager.loadAllXSD(SchemaManager.java:209)
    ^-- Holding lock: oracle.fabric.common.wsdl.SchemaManager@36279195[thin lock
    oracle.fabric.common.wsdl.SchemaManager.getXSDEntityResolver(SchemaManager.j
    ava:424)
    oracle.fabric.composite.model.CompositeModel.loadImports(CompositeModel.java
    :361)
    oracle.fabric.composite.model.CompositeModel.getWSDLManager(CompositeModel.j
    ava:196)
    oracle.integration.platform.blocks.adapter.AbstractAdapterBindingComponent.g
    etWSDLDefinition(AbstractAdapterBindingComponent.java:168)
    oracle.integration.platform.blocks.adapter.AdapterReference.loadDefinition(A
    dapterReference.java:543)
    oracle.integration.platform.blocks.adapter.AdapterReference.load(AdapterRefe
    rence.java:377)
    oracle.integration.platform.blocks.adapter.AdapterReference.load(AdapterRefe
    rence.java:73)
    oracle.integration.platform.blocks.deploy.CompositeDeploymentConnection.depl
    oyReferences(CompositeDeploymentConnection.java:177)
    oracle.integration.platform.blocks.deploy.CompositeDeploymentConnection.depl
    oy(CompositeDeploymentConnection.java:81)
    oracle.integration.platform.blocks.deploy.CompositeDeploymentManagerImpl.ini
    tDeployment(CompositeDeploymentManagerImpl.java:143)
    oracle.integration.platform.blocks.deploy.CompositeDeploymentManagerImpl.loa
    d(CompositeDeploymentManagerImpl.java:61)
    sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:?
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:27
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
    .java:25)
    java.lang.reflect.Method.invoke(Method.java:575)
    org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopU
    tils.java:306)
    org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint
    (ReflectiveMethodInvocation.java:182)
    org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
    iveMethodInvocation.java:148)
    oracle.integration.platform.blocks.deploy.DeploymentEventPublisher.invoke(De
    ploymentEventPublisher.java:57)
    org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
    iveMethodInvocation.java:148)
    org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopPro
    xy.java:151)
    $Proxy309.load(Unknown Source)
    oracle.integration.platform.blocks.deploy.StandaloneCompositeDeploymentCoord
    inatorImpl.coordinateCompositeDeployment(StandaloneCompositeDeploymentCoordinato
    rImpl.java:54)
    oracle.integration.platform.blocks.deploy.servlet.BaseDeployProcessor.deploy
    NewComposite(BaseDeployProcessor.java:365)
    oracle.integration.platform.blocks.deploy.servlet.BaseDeployProcessor.deploy
    SARs(BaseDeployProcessor.java:123)
    ^-- Holding lock: java.lang.Object@3621c97c[thin lock]
    oracle.integration.platform.blocks.deploy.servlet.DeployProcessor.doDeployWo
    rk(DeployProcessor.java:122)
    oracle.integration.platform.blocks.deploy.servlet.DeployProcessor.doDeployWo
    rk(DeployProcessor.java:107)
    oracle.integration.platform.blocks.deploy.servlet.DeployProcessor.doDeploy(D
    eployProcessor.java:96)
    oracle.integration.platform.blocks.deploy.servlet.DeployProcessor.process(De
    ployProcessor.java:68)
    oracle.integration.platform.blocks.deploy.servlet.CompositeDeployerServlet.d
    oPostInsideLoggingSession(CompositeDeployerServlet.java:141)
    oracle.integration.platform.blocks.deploy.servlet.CompositeDeployerServlet.d
    oPost(CompositeDeployerServlet.java:119)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:700)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:815)
    weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSe
    curityHelper.java:224)
    weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelpe
    r.java:108)
    weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:206)
    weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:111)
    oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:299)
    oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java
    :405)
    oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:94)
    oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:98)
    oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:70)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:111)
    oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:299)
    oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java
    :405)
    oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:94)
    oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:98)
    oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:70)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:86)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapR
    un(WebAppServletContext.java:3687)
    weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(W
    ebAppServletContext.java:3681)
    weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubjec
    t.java:308)
    weblogic.security.service.SecurityManager.runAs(SecurityManager.java:116)
    weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletC
    ontext.java:2213)
    weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.
    java:2135)
    weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:142
    0)
    weblogic.work.ExecuteThread.execute(ExecuteThread.java:203)
    weblogic.work.ExecuteThread.run(ExecuteThread.java:170)
    >

    I just noted that this is a database connectivity issue because when I point my soa related data sources (mds, soa_infra, etc...)to a local xe database, the deployment is doing fine.

  • STUCK thread equals elevated CPU

    Agile PLM 9.3.0.1
    Weblogic 10.0
    Solaris 10
    Recently installed Agile 9301 in production. Three times in the week since go-live CPU has elevated up to 100% (twice on one managed host, once on the other), forcing me to kill Agile on that managed host and restart. We sure did not see anything like this on the development environment.
    * I'd like to know why this is happening.
    * Obviously I'd like to make it stop.
    * If I can't do that I can take steps to make the problem less deadly to the system.
    Installed VisualVM to look inside the JVM. I've got a STUCK thread that seems to be elevating the CPU past 25% (baseline appears to be around 10%). Now I have more data to confuse myself with. Here are the snipped bits relating to this thread from the thread dump.
    * Does this mean anything to anyone?
    * Any advice on how to track down a problem like this?
    Thread Dump snip with the STUCK threads
    "[STUCK] ExecuteThread: '14' for queue: 'weblogic.kernel.Default (self-tuning)'" - Thread t@101
    java.lang.Thread.State: RUNNABLE
    at java.util.HashMap.get(HashMap.java:303)
    at com.agile.ui.web.action.ClientSession.getAttribute(ClientSession.java:154)
    at com.agile.ui.web.action.ActionContext.getAttribute(ActionContext.java:385)
    at com.agile.ui.pcm.common.model.AbstractDataModel.saveContext(AbstractDataModel.java:1701)
    at com.agile.ui.pcm.common.CMBaseModuleHandler.removeAll(CMBaseModuleHandler.java:153)
    at com.agile.ui.pcm.common.CMBaseModuleHandler.tabContextChanged(CMBaseModuleHandler.java:136)
    at com.agile.ui.web.action.ActionServlet.raiseRequestEvents(ActionServlet.java:1240)
    at com.agile.ui.web.action.ActionServlet.handleRequest(ActionServlet.java:658)
    at com.agile.ui.web.action.ActionServlet.doPost(ActionServlet.java:309)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
    at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
    at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
    at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:292)
    at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
    at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
    at com.agile.ui.pcm.common.filter.RemoteFSRequestFilter.doFilter(RemoteFSRequestFilter.java:148)
    at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
    at com.agile.ui.web.filter.LoggingFilter.doFilter(LoggingFilter.java:108)
    at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
    at com.agile.ui.pcm.common.filter.WebClientLog.doFilter(WebClientLog.java:78)
    at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
    at com.jspbook.GZIPFilter.doFilter(GZIPFilter.java:21)
    at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
    at com.agile.ui.pcm.common.filter.SSOTicketFilter.doFilter(SSOTicketFilter.java:84)
    at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
    at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3496)
    at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
    at weblogic.security.service.SecurityManager.runAs(Unknown Source)
    at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2180)
    at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2086)
    at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1406)
    at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
    at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
    Locked ownable synchronizers:
    - None
    "[STUCK] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" - Thread t@14
    java.lang.Thread.State: RUNNABLE
    at java.util.HashMap.get(HashMap.java:303)
    at com.agile.ui.web.action.ClientSession.getAttribute(ClientSession.java:154)
    at com.agile.ui.web.action.ActionContext.getAttribute(ActionContext.java:385)
    at com.agile.ui.pcm.common.ObjectViewHandler.setGridPage(ObjectViewHandler.java:20617)
    at sun.reflect.GeneratedMethodAccessor231.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.agile.ui.web.action.ActionServlet.invokeMethod(ActionServlet.java:1062)
    at com.agile.ui.web.action.ActionServlet.handleRequest(ActionServlet.java:667)
    at com.agile.ui.web.action.ActionServlet.doPost(ActionServlet.java:309)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
    at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
    at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
    at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:292)
    at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
    at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
    at com.agile.ui.pcm.common.filter.RemoteFSRequestFilter.doFilter(RemoteFSRequestFilter.java:148)
    at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
    at com.agile.ui.web.filter.LoggingFilter.doFilter(LoggingFilter.java:108)
    at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
    at com.agile.ui.pcm.common.filter.WebClientLog.doFilter(WebClientLog.java:78)
    at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
    at com.jspbook.GZIPFilter.doFilter(GZIPFilter.java:21)
    at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
    at com.agile.ui.pcm.common.filter.SSOTicketFilter.doFilter(SSOTicketFilter.java:84)
    at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
    at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3496)
    at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
    at weblogic.security.service.SecurityManager.runAs(Unknown Source)
    at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2180)
    at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2086)
    at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1406)
    at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
    at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
    Locked ownable synchronizers:
    - None

    Did you take multiple thread dumps to see if the thread is always in the same place? It seems that a "Runnable" thread is making progress and not blocked on any locks.
    There is additional troubleshooting steps at support.oracle.com with a document called Oracle WebLogic Server Core Server / JVM / Memory Issues Support Patterns - Unexpected High CPU Usage with WLS
    Document ID 874292.1 in the Knowledge Base
    The article is a bit old, but the approach should still be sound. I'm not sure if VisualVM gives you CPU by thread or not, but I know it has a profiler.

Maybe you are looking for

  • Safe Mode Over 6 Hours?

    I just bought a Macbook Pro 2006 online, not from apple though. Had the computer about six months now, at four months it started to freeze up. I would reboot it, and same problem. So I got told to put it into safe mode. I let it run for 3 hours, noth

  • RSS Subscribe Button point to old domain name?

    Hey, A .mac site I am help maintain is doing something strange with the RSS feeds. http://web.mac.com/calvarymissions/CCMA/home.html There are currently 2 blog style pages on there and both of the RSS feeds point to an old domain name address rather

  • Adding functionality to stand. sap application

    hi,   can any body help how we can add process on value request or search help for a field in standared sap application.

  • CM replication issue

    hi all We have a replication problem between PUB ans SUB call manager servers where some IP Phone are registered on the SUB and other IP Phone are registered on the PUB when reviewing database reporting on the SUB we get the attached pic up to date 2

  • HP G 42 UNABLE TO RECOVER OLD DATA AND FOLDERS AFTER I RAN RECOVERY TO FACTORY SETTINGS

    HP G 42 UNABLE TO RECOVER OLD DATA AND FOLDERS AFTER I RAN RECOVERY TO FACTORY SETTINGS