Stuck Threads on weblogic 10.3.2 Holding lock HttpURLConnection

We are having issue with STUCK THREADS on weblogic 10.3.2. Basically my application makes a lot of web service calls over HTTPS and once in a while I see few STUCK THREADS and most of the time they clear but sometimes I have to bounce my server to clean.
I have a thread dump, according to this its holding lock at weblogic.net.http.HttpURLConnection.getInputStream
Any help to resolve this problem is highly appreciated.
ThreadDump:
<May 24, 2011 9:59:08 AM EDT> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "604" seconds working on the request "weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl@b607d75", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-75 "[STUCK] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, priority=1, DAEMON> {
jrockit.net.SocketNativeIO.readBytesPinned(SocketNativeIO.java:???)
jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:31)
java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)
java.net.SocketInputStream.read(SocketInputStream.java:107)
weblogic.utils.io.ChunkedInputStream.read(ChunkedInputStream.java:149)
java.io.InputStream.read(InputStream.java:85)
com.certicom.tls.record.ReadHandler.readFragment(Unknown Source)
com.certicom.tls.record.ReadHandler.readRecord(Unknown Source)
com.certicom.tls.record.ReadHandler.read(Unknown Source)
^-- Holding lock: com.certicom.tls.record.ReadHandler@bec563c[thin lock]
com.certicom.io.InputSSLIOStreamWrapper.read(Unknown Source)
java.io.BufferedInputStream.fill(BufferedInputStream.java:189)
java.io.BufferedInputStream.read(BufferedInputStream.java:236)
^-- Holding lock: java.io.BufferedInputStream@bec5d9f[thin lock]
weblogic.net.http.MessageHeader.isHTTP(MessageHeader.java:214)
weblogic.net.http.MessageHeader.parseHeader(MessageHeader.java:141)
weblogic.net.http.HttpClient.parseHTTP(HttpClient.java:452)
^-- Holding lock: weblogic.net.http.HttpsClient@bec5202[thin lock]
weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:328)
weblogic.net.http.SOAPHttpsURLConnection.getInputStream(SOAPHttpsURLConnection.java:37)
^-- Holding lock: weblogic.net.http.SOAPHttpsURLConnection@bd3b209[thin lock]
weblogic.net.http.HttpURLConnection.getResponseCode(HttpURLConnection.java:939)
com.sun.xml.ws.transport.http.client.HttpClientTransport.readResponseCodeAndMessage(HttpClientTransport.java:213)
com.sun.xml.ws.transport.http.client.HttpTransportPipe.process(HttpTransportPipe.java:116)
com.sun.xml.ws.transport.http.client.HttpTransportPipe.processRequest(HttpTransportPipe.java:101)
com.sun.xml.ws.api.pipe.Fiber.__doRun(Fiber.java:580)
com.sun.xml.ws.api.pipe.Fiber._doRun(Fiber.java:553)
com.sun.xml.ws.api.pipe.Fiber.doRun(Fiber.java:535)
com.sun.xml.ws.api.pipe.Fiber.runSync(Fiber.java:433)
^-- Holding lock: com.sun.xml.ws.api.pipe.Fiber@bd3afa8[thin lock]
com.sun.xml.ws.client.Stub.process(Stub.java:238)
com.sun.xml.ws.client.sei.SEIStub.doProcess(SEIStub.java:135)
com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:102)
com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:89)
com.sun.xml.ws.client.sei.SEIStub.invoke(SEIStub.java:116)
$Proxy107.jobSubmit(Unknown Source)
com.somejob.invokeService(JobOperation.java:53)
com.somejobws.operations.ticket.execute(ticket.java:30)
com.somejobutil.Somehelper.process(Somehelper.java:174)
com.somejobutil.Somehelper.processJob(Somehelper.java:147)
com.somejobutil.Somehelper.processMDB(Somehelper.java:91)
com.somejobejb.ProcessMDB.onMessage(ProcessMDB.java:28)
sun.reflect.GeneratedMethodAccessor109.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:102)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:86)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:102)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:151)
$Proxy67.onMessage(Unknown Source)
weblogic.ejb.container.internal.MDListener.execute(MDListener.java:403)
weblogic.ejb.container.internal.MDListener.transactionalOnMessage(MDListener.java:352)
weblogic.ejb.container.internal.MDListener.onMessage(MDListener.java:264)
weblogic.jms.client.JMSSession.onMessage(JMSSession.java:4564)
weblogic.jms.client.JMSSession.execute(JMSSession.java:4060)
weblogic.jms.client.JMSSession.executeMessage(JMSSession.java:3746)
weblogic.jms.client.JMSSession.access$000(JMSSession.java:114)
weblogic.jms.client.JMSSession$UseForRunnable.run(JMSSession.java:5096)
weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:516)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:198)
weblogic.work.ExecuteThread.run(ExecuteThread.java:165)
}

Yes, we have the same problem, stuck thread on weblogic.net.http.HttpsURLConnection.getResponseCode() (actually implemented by weblogic.net.http.SOAPHttpsURLConnection which also appears to be deprecated ???)
When we use the class javax.net.ssl.HttpsURLConnection the getResponseCode() method works perfectly well, so it looks like the weblogic implementation is to blame.
This is becoming rather urgent now because we're looking to migrate from OC4J to Weblogic and this issue has therefore become a blocker.

Similar Messages

  • Stuck thread in weblogic

    Hello All,
    We are getting stuck thread in weblogic under high load.
    following is the thread dump from weblogic 10
    "[STANDBY] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)'" id=83 idx=0x134 tid=5376 prio=5 alive, in native, waiting, daemon
    -- Waiting for notification on: weblogic/work/ExecuteThread@0x06DEC880[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at java/lang/Object.wait(Object.java:474)
    at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)
    ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x06DEC880[fat lock]
    at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
    "[STANDBY] ExecuteThread: '14' for queue: 'weblogic.kernel.Default (self-tuning)'" id=84 idx=0x138 tid=1688 prio=5 alive, in native, waiting, daemon
    -- Waiting for notification on: weblogic/work/ExecuteThread@0x06E50070[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at java/lang/Object.wait(Object.java:474)
    at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)
    ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x06E50070[fat lock]
    at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
    "[STUCK] ExecuteThread: '15' for queue: 'weblogic.kernel.Default (self-tuning)'" id=85 idx=0x13c tid=3984 prio=1 alive, daemon
    at jrockit/vm/Allocator.allocArray(Allocator.java:392)
    at java/lang/AbstractStringBuilder.<init>(AbstractStringBuilder.java:44)
    at java/lang/StringBuffer.<init>(StringBuffer.java:80)
    at com/vignette/exolab/castor/jdo/oql/ParseTreeWalker.addWhereClause(ParseTreeWalker.java:994)
    at com/vignette/exolab/castor/jdo/oql/ParseTreeWalker.createQueryExpression(ParseTreeWalker.java:816)
    at com/vignette/exolab/castor/jdo/oql/ParseTreeWalker.<init>(ParseTreeWalker.java:139)
    at com/vignette/exolab/castor/jdo/engine/OQLQueryImpl.create(OQLQueryImpl.java:274)
    at com/vignette/exolab/castor/jdo/engine/DatabaseImpl.getOQLQuery(DatabaseImpl.java:502)
    at com/vignette/as/server/persist/dao/JDODAOPersistor.findByQuery(JDODAOPersistor.java:395)
    at com/vignette/as/server/persist/dao/JDODAOPersistor.findByIds(JDODAOPersistor.java:360)
    at com/vignette/as/server/persist/ExtensibleApplicationObjectPersistor.findByIdsHelper(ExtensibleApplicationObjectPersistor.java:159)
    at com/vignette/as/server/persist/ExtensibleApplicationObjectPersistor.findByIds(ExtensibleApplicationObjectPersistor.java:90)
    at com/vignette/as/server/logic/channel/ChannelOpsLogic.findById(ChannelOpsLogic.java:194)
    at com/vignette/as/server/logic/channel/ChannelOpsLogicGen.findById(ChannelOpsLogicGen.java:1553)
    at com/vignette/as/client/javabean/Channel.findById(Channel.java:247)
    at com/vignette/as/server/logic/channel/ChannelOpsLogic.getSubchannels(ChannelOpsLogic.java:563)
    at com/vignette/as/server/logic/channel/ChannelOpsLogicGen.getSubchannels(ChannelOpsLogicGen.java:3237)
    at com/vignette/as/client/javabean/ChannelBase.getSubchannels(ChannelBase.java:480)
    at com/vignette/as/client/javabean/ChannelBase.getAllSubchannels(ChannelBase.java:375)
    at com/astrazeneca/portal/dpm/ChannelWrapper.getSubchannels(ChannelWrapper.java:203)
    at jsp_servlet/_jsp/_grid/__xmlgeneration.subChannelRec(__xmlgeneration.java:97)
    at jsp_servlet/_jsp/_grid/__xmlgeneration.subChannelRec(__xmlgeneration.java:146)
    at jsp_servlet/_jsp/_grid/__xmlgeneration._jspService(__xmlgeneration.java:430)
    at weblogic/servlet/jsp/JspBase.service(JspBase.java:34)
    at weblogic/servlet/internal/StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:226)
    at weblogic/servlet/internal/StubSecurityHelper.invokeServlet(StubSecurityHelper.java:124)
    at weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:283)
    at weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:175)
    at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3395)
    at weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
    at weblogic/security/service/SecurityManager.runAs(Lweblogic/security/acl/internal/AuthenticatedSubject;Lweblogic/security/acl/internal/AuthenticatedSubject;Ljava/security/PrivilegedAction;)Ljava/lang/Object;(Unknown Source)
    at weblogic/servlet/internal/WebAppServletContext.securedExecute(WebAppServletContext.java:2140)
    at weblogic/servlet/internal/WebAppServletContext.execute(WebAppServletContext.java:2046)
    at weblogic/servlet/internal/ServletRequestImpl.run(ServletRequestImpl.java:1366)
    at weblogic/work/ExecuteThread.execute(ExecuteThread.java:200)
    at weblogic/work/ExecuteThread.run(ExecuteThread.java:172)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
    "[STANDBY] ExecuteThread: '16' for queue: 'weblogic.kernel.Default (self-tuning)'" id=86 idx=0x140 tid=1192 prio=5 alive, in native, waiting, daemon
    -- Waiting for notification on: weblogic/work/ExecuteThread@0x06E50AF0[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at java/lang/Object.wait(Object.java:474)
    at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)
    ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x06E50AF0[fat lock]
    at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
    "VAP Topic Consumer Thread - 0" id=87 idx=0x144 tid=6040 prio=5 alive, in native, parked, daemon
    at jrockit/vm/Locks.park0(J)V(Native Method)
    at jrockit/vm/Locks.park(Locks.java:2416)
    at sun/misc/Unsafe.park(ZJ)V(Native Method)
    at java/util/concurrent/locks/LockSupport.park(LockSupport.java:118)
    at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1767)
    at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
    at com/epicentric/topic/internal/MessageQueue.remove(MessageQueue.java:67)
    at com/epicentric/topic/internal/Consumer.run(Consumer.java:72)
    at java/lang/Thread.run(Thread.java:595)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
    "VAP Topic Consumer Thread - 1" id=88 idx=0x148 tid=4752 prio=5 alive, in native, parked, daemon
    at jrockit/vm/Locks.park0(J)V(Native Method)
    at jrockit/vm/Locks.park(Locks.java:2416)
    at sun/misc/Unsafe.park(ZJ)V(Native Method)
    at java/util/concurrent/locks/LockSupport.park(LockSupport.java:118)
    at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1767)
    at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
    at com/epicentric/topic/internal/MessageQueue.remove(MessageQueue.java:67)
    at com/epicentric/topic/internal/Consumer.run(Consumer.java:72)
    at java/lang/Thread.run(Thread.java:595)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
    "VAP Topic Consumer Thread - 2" id=89 idx=0x14c tid=4504 prio=5 alive, in native, parked, daemon
    at jrockit/vm/Locks.park0(J)V(Native Method)
    at jrockit/vm/Locks.park(Locks.java:2416)
    at sun/misc/Unsafe.park(ZJ)V(Native Method)
    at java/util/concurrent/locks/LockSupport.park(LockSupport.java:118)
    at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1767)
    at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
    at com/epicentric/topic/internal/MessageQueue.remove(MessageQueue.java:67)
    at com/epicentric/topic/internal/Consumer.run(Consumer.java:72)
    at java/lang/Thread.run(Thread.java:595)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
    "VAP Topic Consumer Thread - 3" id=90 idx=0x150 tid=5572 prio=5 alive, in native, parked, daemon
    at jrockit/vm/Locks.park0(J)V(Native Method)
    at jrockit/vm/Locks.park(Locks.java:2416)
    at sun/misc/Unsafe.park(ZJ)V(Native Method)
    at java/util/concurrent/locks/LockSupport.park(LockSupport.java:118)
    at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1767)
    at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
    at com/epicentric/topic/internal/MessageQueue.remove(MessageQueue.java:67)
    at com/epicentric/topic/internal/Consumer.run(Consumer.java:72)
    at java/lang/Thread.run(Thread.java:595)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
    "VAP Topic Consumer Thread - 4" id=91 idx=0x154 tid=4340 prio=5 alive, in native, parked, daemon
    at jrockit/vm/Locks.park0(J)V(Native Method)
    at jrockit/vm/Locks.park(Locks.java:2416)
    at sun/misc/Unsafe.park(ZJ)V(Native Method)
    at java/util/concurrent/locks/LockSupport.park(LockSupport.java:118)
    at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1767)
    at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
    at com/epicentric/topic/internal/MessageQueue.remove(MessageQueue.java:67)
    at com/epicentric/topic/internal/Consumer.run(Consumer.java:72)
    at java/lang/Thread.run(Thread.java:595)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
    "CS_Local_Monitor_0" id=92 idx=0x158 tid=3864 prio=5 alive, in native, sleeping, native_waiting, daemon
    at java/lang/Thread.sleep(J)V(Native Method)
    at com/vignette/config/client/cachefile/ConfigSpaceLocal$Monitor.run(ConfigSpaceLocal.java:1364)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
    "Thread-41" id=93 idx=0x15c tid=4884 prio=1 alive, in native, waiting, daemon
    -- Waiting for notification on: java/util/ArrayList@0x0A899918[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at java/lang/Object.wait(Object.java:474)
    at com/vignette/logging/LocaleAppender$Dispatcher.run(LocaleAppender.java:393)
    ^-- Lock released while waiting: java/util/ArrayList@0x0A899918[fat lock]
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
    "Thread-42" id=94 idx=0x160 tid=4968 prio=1 alive, in native, waiting, daemon
    -- Waiting for notification on: java/util/ArrayList@0x06BE2238[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at java/lang/Object.wait(Object.java:474)
    at com/vignette/logging/LocaleAppender$Dispatcher.run(LocaleAppender.java:393)
    ^-- Lock released while waiting: java/util/ArrayList@0x06BE2238[fat lock]
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
    "TaskScheduler thread" id=98 idx=0x164 tid=1580 prio=5 alive, in native, sleeping, native_waiting, daemon
    at java/lang/Thread.sleep(J)V(Native Method)
    at com/epicentric/task/TaskScheduler.sleepUntilRuntime(TaskScheduler.java:226)
    at com/epicentric/task/TaskScheduler.run(TaskScheduler.java:83)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
    "TaskQueue Lease Manager" id=100 idx=0x168 tid=3908 prio=5 alive, in native, sleeping, native_waiting, daemon
    at java/lang/Thread.sleep(J)V(Native Method)
    at com/epicentric/task/internal/LeaseManager.waitForNextExecutionTime(LeaseManager.java:69)
    at com/epicentric/task/internal/LeaseManager.run(LeaseManager.java:52)
    at java/lang/Thread.run(Thread.java:595)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
    "Thread System Expiration and Ostracizing Thread" id=101 idx=0x16c tid=4200 prio=5 alive, in native, sleeping, native_waiting, daemon
    at java/lang/Thread.sleep(J)V(Native Method)
    at com/epicentric/threadpool/standard/StandardThreadPool$ExpirationThread.run(StandardThreadPool.java:238)
    at java/lang/Thread.run(Thread.java:595)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
    "Task Executor" id=102 idx=0x170 tid=3592 prio=1 alive, in native, sleeping, native_waiting, daemon
    at java/lang/Thread.sleep(J)V(Native Method)
    at com/epicentric/task/internal/TaskExecutor.waitForNextExecutionTime(TaskExecutor.java:205)
    at com/epicentric/task/internal/TaskExecutor.run(TaskExecutor.java:186)
    at java/lang/Thread.run(Thread.java:595)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
    "Task Metric Gatherer" id=103 idx=0x174 tid=5988 prio=1 alive, in native, sleeping, native_waiting, daemon
    at java/lang/Thread.sleep(J)V(Native Method)
    at com/epicentric/task/internal/TaskExecutor$1.run(TaskExecutor.java:142)
    at java/lang/Thread.run(Thread.java:595)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
    "CS_Local_Monitor_0" id=142 idx=0x178 tid=5196 prio=5 alive, in native, sleeping, native_waiting, daemon
    at java/lang/Thread.sleep(J)V(Native Method)
    at com/vignette/config/client/cachefile/ConfigSpaceLocal$Monitor.run(ConfigSpaceLocal.java:1364)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
    "Thread-54" id=130 idx=0x17c tid=648 prio=1 alive, in native, waiting, daemon
    -- Waiting for notification on: java/util/ArrayList@0x0AD3C798[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at java/lang/Object.wait(Object.java:474)
    at com/vignette/logging/LocaleAppender$Dispatcher.run(LocaleAppender.java:393)
    ^-- Lock released while waiting: java/util/ArrayList@0x0AD3C798[fat lock]
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
    "Timer-10" id=113 idx=0x180 tid=1092 prio=5 alive, in native, waiting, daemon
    -- Waiting for notification on: java/util/TaskQueue@0x00B22F40[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at java/util/TimerThread.mainLoop(Timer.java:509)
    ^-- Lock released while waiting: java/util/TaskQueue@0x00B22F40[fat lock]
    at java/util/TimerThread.run(Timer.java:462)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
    "JNDI Monitor" id=116 idx=0x184 tid=1856 prio=5 alive, in native, waiting, daemon
    -- Waiting for notification on: com/epicentric/jndi/Monitor@0x00B12D60[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at com/epicentric/jndi/Monitor.run(Monitor.java:100)
    ^-- Lock released while waiting: com/epicentric/jndi/Monitor@0x00B12D60[fat lock]
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
    "Thread-47" id=117 idx=0x188 tid=816 prio=5 alive, in native, daemon
    at jrockit/net/SocketNativeIO.readBytesPinned(Ljava/io/FileDescriptor;[BIII)I(Native Method)
                    at jrockit/net/SocketNativeIO.socketRead(SocketNativeIO.java:31)
                    at java/net/SocketInputStream.socketRead0(Ljava/io/FileDescriptor;[BIII)I(SocketInputStream.java)
                    at java/net/SocketInputStream.read(SocketInputStream.java:129)
                    at java/io/BufferedInputStream.fill(BufferedInputStream.java:218)
                    at java/io/BufferedInputStream.read1(BufferedInputStream.java:256)
                    at java/io/BufferedInputStream.read(BufferedInputStream.java:313)
                    ^-- Holding lock: java/io/BufferedInputStream@0x00B3EA48[thin lock]
    at com/sun/jndi/ldap/Connection.run(Connection.java:784)
    at java/lang/Thread.run(Thread.java:595)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
    "TaskScheduler thread" id=119 idx=0x18c tid=4520 prio=5 alive, in native, sleeping, native_waiting, daemon
    at java/lang/Thread.sleep(J)V(Native Method)
    at com/epicentric/task/TaskScheduler.run(TaskScheduler.java:94)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
    "OracleTimeoutPollingThread" id=124 idx=0x190 tid=2628 prio=10 alive, in native, sleeping, native_waiting, daemon
    at java/lang/Thread.sleep(J)V(Native Method)
    at oracle/jdbc/driver/OracleTimeoutPollingThread.run(OracleTimeoutPollingThread.java:158)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
    "Thread-55" id=131 idx=0x194 tid=5700 prio=1 alive, in native, waiting, daemon
    -- Waiting for notification on: java/util/ArrayList@0x0973F378[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at java/lang/Object.wait(Object.java:474)
    at com/vignette/logging/LocaleAppender$Dispatcher.run(LocaleAppender.java:393)
    ^-- Lock released while waiting: java/util/ArrayList@0x0973F378[fat lock]
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
    "Thread-56" id=132 idx=0x198 tid=5596 prio=5 alive, in native, sleeping, native_waiting, daemon
    at java/lang/Thread.sleep(J)V(Native Method)
    at org/apache/commons/pool/impl/GenericObjectPool$Evictor.run(GenericObjectPool.java:1122)
    at java/lang/Thread.run(Thread.java:595)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
    "CacheSyncThread" id=133 idx=0x19c tid=708 prio=5 alive, in native, sleeping, native_waiting, daemon
    at java/lang/Thread.sleep(J)V(Native Method)
    at com/vignette/as/server/caching/CacheSynchronization.run(CacheSynchronization.java:135)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
    "Thread-58" id=135 idx=0x1a0 tid=5652 prio=5 alive, in native, sleeping, native_waiting, daemon
    at java/lang/Thread.sleep(J)V(Native Method)
    at org/apache/commons/pool/impl/GenericObjectPool$Evictor.run(GenericObjectPool.java:1122)
    at java/lang/Thread.run(Thread.java:595)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
    "Thread-61" id=143 idx=0x1a4 tid=4196 prio=1 alive, in native, waiting, daemon
    -- Waiting for notification on: java/util/ArrayList@0x0BEF1160[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at java/lang/Object.wait(Object.java:474)
    at com/vignette/logging/LocaleAppender$Dispatcher.run(LocaleAppender.java:393)
    ^-- Lock released while waiting: java/util/ArrayList@0x0BEF1160[fat lock]
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
    "Thread-62" id=144 idx=0x1a8 tid=5456 prio=1 alive, in native, waiting, daemon
    -- Waiting for notification on: java/util/ArrayList@0x0D650770[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at java/lang/Object.wait(Object.java:474)
    at com/vignette/logging/LocaleAppender$Dispatcher.run(LocaleAppender.java:393)
    ^-- Lock released while waiting: java/util/ArrayList@0x0D650770[fat lock]
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
    "Thread-63" id=145 idx=0x1ac tid=1056 prio=5 alive, in native, sleeping, native_waiting, daemon
    at java/lang/Thread.sleep(J)V(Native Method)
    at org/apache/commons/pool/impl/GenericObjectPool$Evictor.run(GenericObjectPool.java:1122)
    at java/lang/Thread.run(Thread.java:595)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
    "CacheSyncThread" id=146 idx=0x1b0 tid=3496 prio=5 alive, in native, sleeping, native_waiting, daemon
    at java/lang/Thread.sleep(J)V(Native Method)
    at com/vignette/as/server/caching/CacheSynchronization.run(CacheSynchronization.java:135)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
    "Thread-64" id=147 idx=0x1b4 tid=4132 prio=5 alive, in native, sleeping, native_waiting, daemon
    at java/lang/Thread.sleep(J)V(Native Method)
    at org/apache/commons/pool/impl/GenericObjectPool$Evictor.run(GenericObjectPool.java:1122)
    at java/lang/Thread.run(Thread.java:595)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
    "CS_Local_Monitor_0" id=221 idx=0x1b8 tid=6104 prio=5 alive, in native, sleeping, native_waiting, daemon
    at java/lang/Thread.sleep(J)V(Native Method)
    at com/vignette/config/client/cachefile/ConfigSpaceLocal$Monitor.run(ConfigSpaceLocal.java:1364)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
    "Thread-84" id=222 idx=0x1bc tid=3500 prio=1 alive, in native, waiting, daemon
    -- Waiting for notification on: java/util/ArrayList@0x1673EAD8[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at java/lang/Object.wait(Object.java:474)
    at com/vignette/logging/LocaleAppender$Dispatcher.run(LocaleAppender.java:393)
    ^-- Lock released while waiting: java/util/ArrayList@0x1673EAD8[fat lock]
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
    "Thread-85" id=223 idx=0x1c0 tid=5492 prio=1 alive, in native, waiting, daemon
    -- Waiting for notification on: java/util/ArrayList@0x1932F128[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at java/lang/Object.wait(Object.java:474)
    at com/vignette/logging/LocaleAppender$Dispatcher.run(LocaleAppender.java:393)
    ^-- Lock released while waiting: java/util/ArrayList@0x1932F128[fat lock]
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
    "[STANDBY] ExecuteThread: '17' for queue

    Hi,
    We are facing a similar stuck thread problem with the following stack trace...
    Any help on this is much appreciated.
    Thanks,
    Sen
    Thread-86 "[STUCK] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, waiting, priority=1, DAEMON> {
    -- Waiting for notification on: netscape.ldap.LDAPSearchListener@2accce6[fat lock]
    java.lang.Object.wait(Object.java:???)
    java.lang.Object.wait(Object.java:485)
    netscape.ldap.LDAPMessageQueue.waitForMessage(LDAPMessageQueue.java:198)
    netscape.ldap.LDAPMessageQueue.waitFirstMessage(LDAPMessageQueue.java:100)
    netscape.ldap.LDAPConnection.sendRequest(LDAPConnection.java:1782)
    ^-- Holding lock: netscape.ldap.LDAPConnection@138555c[thin lock]
    netscape.ldap.LDAPConnection.search(LDAPConnection.java:2500)
    weblogic.security.providers.authentication.LDAPAtnDelegate.getDNForUser(LDAPAtnDelegate.java:3317)
    weblogic.security.providers.authentication.LDAPAtnDelegate.listMemberGroups(LDAPAtnDelegate.java:2091)
    weblogic.security.providers.authentication.LDAPAtnLoginModuleImpl.addAllGroups(LDAPAtnLoginModuleImpl.java:419)
    weblogic.security.providers.authentication.LDAPAtnLoginModuleImpl.addGroups(LDAPAtnLoginModuleImpl.java:411)
    weblogic.security.providers.authentication.LDAPAtnLoginModuleImpl.login(LDAPAtnLoginModuleImpl.java:136)
    com.bea.common.security.internal.service.LoginModuleWrapper$1.run(LoginModuleWrapper.java:110)
    com.bea.common.security.internal.service.LoginModuleWrapper.login(LoginModuleWrapper.java:101)
    sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:???)
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:27)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    java.lang.reflect.Method.invoke(Method.java:575)
    javax.security.auth.login.LoginContext.invoke(LoginContext.java:720)
    javax.security.auth.login.LoginContext.access$000(LoginContext.java:186)
    javax.security.auth.login.LoginContext$4.run(LoginContext.java:683)
    javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
    javax.security.auth.login.LoginContext.login(LoginContext.java:566)
    com.bea.common.security.internal.service.JAASLoginServiceImpl.login(JAASLoginServiceImpl.java:77)
    sun.reflect.GeneratedMethodAccessor2726.invoke(Unknown Source)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    java.lang.reflect.Method.invoke(Method.java:575)
    com.bea.common.security.internal.utils.Delegator$ProxyInvocationHandler.invoke(Delegator.java:57)
    $Proxy17.login(Unknown Source)
    weblogic.security.service.internal.WLSJAASLoginServiceImpl$ServiceImpl.login(Unknown Source)
    com.bea.common.security.internal.service.JAASAuthenticationServiceImpl.authenticate(JAASAuthenticationServiceImpl.java:74)
    sun.reflect.GeneratedMethodAccessor2727.invoke(Unknown Source)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    java.lang.reflect.Method.invoke(Method.java:575)
    com.bea.common.security.internal.utils.Delegator$ProxyInvocationHandler.invoke(Delegator.java:57)
    $Proxy37.authenticate(Unknown Source)
    weblogic.security.service.WLSJAASAuthenticationServiceWrapper.authenticate(Unknown Source)
    weblogic.security.service.PrincipalAuthenticator.authenticate(Unknown Source)
    weblogic.servlet.security.internal.SecurityModule.checkAuthenticate(SecurityModule.java:213)
    weblogic.servlet.security.ServletAuthentication.login(ServletAuthentication.java:391)
    weblogic.servlet.security.ServletAuthentication.weak(ServletAuthentication.java:371)
    com.bea.p13n.security.Authentication.login(Authentication.java:118)
    com.ppluk.moneyout.portal.auth.LoginServlet.doGet(LoginServlet.java:40)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:700)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:815)
    weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
    weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
    weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:198)
    weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:41)
    com.bea.portal.tools.servlet.http.HttpContextFilter.doFilter(HttpContextFilter.java:50)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:41)
    com.bea.p13n.servlets.PortalServletFilter.doFilter(PortalServletFilter.java:241)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:41)
    com.bea.jsptools.servlet.PagedResultServiceFilter.doFilter(PagedResultServiceFilter.java:61)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:41)
    weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:25)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:41)
    weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3468)
    weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:308)
    weblogic.security.service.SecurityManager.runAs(Unknown Source)
    weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2116)
    weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2038)
    weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1372)
    weblogic.work.ExecuteThread.execute(ExecuteThread.java:198)
    weblogic.work.ExecuteThread.run(ExecuteThread.java:165)
    }

  • How to resolve stuck threads in weblogic

    JDeveloper: 11.3
    Weblogic: 10.3
    Hi,
    System Admin is complaining about the stuck threads in our ADF Application. Can some one help me in resolving those in weblogic server or at code level.
    Basically, I am looking for an approach to resolve them.
    Thanks.

    If you get stuck threads you should also get a dump of all running threads, indicating which threads are stuck. This is the starting point for your investigation. Try to find out which threads are stuck and what they are supposed to do.
    This should be you started.
    Timo

  • Stuck thread in Oracle Weblogic server

    Dear All,
    We have a stuck thread in Oracle WLS 10.3.3. We have deployed 7 ear files in 5th managed server(psMS5). We have stuck thread in one of the application (psinsurance). We have allocated 1GB heap space for the psMS5 managed server. We are using this in Development environment. Please check the below message from psMS5 managed server log file. Kindly provide reason for stuck thread and solution.
    WatchTime: Sep 23, 2011 10:28:29 AM IST
    WatchDomainName: dev_domain
    WatchServerName: psMS5
    WatchSeverityLevel: Warning
    WatchName: MS5-Watch
    WatchRuleType: Log
    WatchRule: ((MESSAGE LIKE '%has been busy for "%') AND (MSGID = 'BEA-000337') AND (SERVER = 'psMS5'))
    WatchData: DATE = Sep 23, 2011 10:28:28 AM IST SERVER = psMS5 MESSAGE = [STUCK] ExecuteThread: '22' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "629" seconds working on the request "weblogic.servlet.internal.ServletRequestImpl@1cac6525[
    POST */psinsurance/app/lending/insurance/PS00738D.jspx*?_adf.ctrl-state=24f0li74e_4 HTTP/1.1
    Accept: image/jpeg, application/x-ms-application, image/gif, application/xaml+xml, image/pjpeg, application/x-ms-xbap, */*
    Referer: http://sis-prosun11/psinsurance/faces/PS00003?jsessionid=r2BrT8TpqZBsnLhCw2Y8KDpTV7hMpgnLDFW54P87ZpHwn8JMy88s%21168932459%211316753022074&_afrWindowMode=0&_afrLoop=244724092027000&_adf.ctrl-state=24f0li74e_4
    Accept-Language: en-US
    User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; Tablet PC 2.0)
    Content-Type: application/x-www-form-urlencoded
    Accept-Encoding: gzip, deflate
    Content-Length: 554
    Cache-Control: no-cache
    Cookie: JSESSIONID=LkJcT8TfMT64SyQJT4Ls4sXhfDl0pTS7CXJVQ9KcryHGWJ1dhyfz!1373322659
    Connection: Keep-Alive
    Proxy-Client-IP: 192.168.9.179
    X-Forwarded-For: 192.168.9.179
    X-WebLogic-KeepAliveSecs: 30
    X-WebLogic-Force-JVMID: 1373322659
    ]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
    Thread-237 "[STUCK] ExecuteThread: '22' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, suspended, priority=1, DAEMON> {
    java.io.UnixFileSystem.getBooleanAttributes0(UnixFileSystem.java:???)
    java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:228)
    java.io.File.exists(File.java:729)
    weblogic.utils.classloaders.ClasspathClassFinder.getFileSource(ClasspathClassFinder.java:383)
    weblogic.utils.classloaders.ClasspathClassFinder.getSourcesInternal(ClasspathClassFinder.java:333)
    weblogic.utils.classloaders.ClasspathClassFinder.getSource(ClasspathClassFinder.java:316)
    weblogic.utils.classloaders.JarClassFinder.getSource(JarClassFinder.java:58)
    weblogic.utils.classloaders.AbstractClassFinder.getClassSource(AbstractClassFinder.java:22)
    weblogic.utils.classloaders.MultiClassFinder.getClassSource(MultiClassFinder.java:56)
    weblogic.utils.classloaders.CodeGenClassFinder.getClassSource(CodeGenClassFinder.java:26)
    weblogic.utils.classloaders.GenericClassLoader.findLocalClass(GenericClassLoader.java:268)
    *^-- Holding lock: weblogic.utils.classloaders.GenericClassLoader@1e3295a1[thin lock]*
    weblogic.utils.classloaders.GenericClassLoader.findClass(GenericClassLoader.java:237)
    java.lang.ClassLoader.loadClass(ClassLoader.java:292)
    *^-- Holding lock: weblogic.utils.classloaders.GenericClassLoader@1e3295a1[thin lock]*
    java.lang.ClassLoader.loadClass(ClassLoader.java:292)
    java.lang.ClassLoader.loadClass(ClassLoader.java:248)
    weblogic.utils.classloaders.GenericClassLoader.loadClass(GenericClassLoader.java:177)
    weblogic.utils.classloaders.ChangeAwareClassLoader.loadClass(ChangeAwareClassLoader.java:37)
    *^-- Holding lock: weblogic.utils.classloaders.ChangeAwareClassLoader@1e553474[thin lock]*
    java.lang.Class.getDeclaredConstructors0(Class.java:???)
    java.lang.Class.privateGetDeclaredConstructors(Class.java:2370)
    java.lang.Class.getConstructor0(Class.java:2699)
    java.lang.Class.newInstance0(Class.java:318)
    java.lang.Class.newInstance(Class.java:305)
    org.apache.myfaces.trinidadinternal.renderkit.ClassRendererInstantiator.instantiate(ClassRendererInstantiator.java:48)
    org.apache.myfaces.trinidadinternal.renderkit.RenderKitBase.findRenderer(RenderKitBase.java:146)
    org.apache.myfaces.trinidadinternal.renderkit.RenderKitDecorator.findRenderer(RenderKitDecorator.java:84)
    oracle.adfinternal.view.faces.unified.renderkit.UnifiedRenderKit._getRenderer(UnifiedRenderKit.java:218)
    oracle.adfinternal.view.faces.unified.renderkit.UnifiedRenderKit.findRenderer(UnifiedRenderKit.java:156)
    org.apache.myfaces.trinidadinternal.renderkit.RenderKitBase.getRenderer(RenderKitBase.java:129)
    org.apache.myfaces.trinidad.component.UIXComponentBase._getRendererImpl(UIXComponentBase.java:1143)
    org.apache.myfaces.trinidad.component.UIXComponentBase.getRenderer(UIXComponentBase.java:1173)
    org.apache.myfaces.trinidad.component.UIXComponentBase.getClientId(UIXComponentBase.java:316)
    javax.faces.webapp.UIComponentClassicTagBase.doStartTag(UIComponentClassicTagBase.java:1095)
    org.apache.myfaces.trinidad.webapp.UIXComponentELTag.doStartTag(UIXComponentELTag.java:70)
    oracle.adfinternal.view.faces.unified.taglib.input.UnifiedSelectBooleanCheckboxTag.doStartTag(UnifiedSelectBooleanCheckboxTag.java:45)
    oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
    oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
    oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
    oracle.jsp.runtime.tree.OracleJspIterationTagNode.executeHandler(OracleJspIterationTagNode.java:41)
    oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
    oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
    oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
    oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
    oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
    oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
    oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
    oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
    oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
    oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
    oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
    oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
    oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
    oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
    oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
    oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
    oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
    oracle.jsp.runtime.tree.OracleJspIterationTagNode.executeHandler(OracleJspIterationTagNode.java:41)
    oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
    oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
    oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
    oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
    oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
    oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
    oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
    oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
    oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
    oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
    oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
    oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
    oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
    oracle.jsp.runtime.tree.OracleJspNode.execute(OracleJspNode.java:73)
    oracle.jsp.runtimev2.ShortCutServlet._jspService(ShortCutServlet.java:57)
    oracle.jsp.runtime.OracleJspBase.service(OracleJspBase.java:29)
    oracle.jsp.runtimev2.JspPageTable.compileAndServe(JspPageTable.java:650)
    oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.java:330)
    oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.java:680)
    oracle.jsp.runtimev2.JspServlet.service(JspServlet.java:653)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:815)
    weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
    weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
    weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:206)
    weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    oracle.adf.library.webapp.LibraryFilter.doFilter(LibraryFilter.java:147)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    weblogic.servlet.internal.RequestDispatcherImpl.invokeServlet(RequestDispatcherImpl.java:520)
    weblogic.servlet.internal.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:112)
    com.sun.faces.context.ExternalContextImpl.dispatch(ExternalContextImpl.java:402)
    org.apache.myfaces.trinidad.context.ExternalContextDecorator.dispatch(ExternalContextDecorator.java:44)
    org.apache.myfaces.trinidad.context.ExternalContextDecorator.dispatch(ExternalContextDecorator.java:44)
    org.apache.myfaces.trinidad.context.ExternalContextDecorator.dispatch(ExternalContextDecorator.java:44)
    org.apache.myfaces.trinidad.context.ExternalContextDecorator.dispatch(ExternalContextDecorator.java:44)
    org.apache.myfaces.trinidadinternal.context.FacesContextFactoryImpl$OverrideDispatch.dispatch(FacesContextFactoryImpl.java:261)
    com.sun.faces.application.ViewHandlerImpl.executePageToBuildView(ViewHandlerImpl.java:428)
    com.sun.faces.application.ViewHandlerImpl.renderView(ViewHandlerImpl.java:132)
    javax.faces.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:189)
    org.apache.myfaces.trinidadinternal.application.ViewHandlerImpl.renderView(ViewHandlerImpl.java:155)
    oracle.adfinternal.view.faces.lifecycle.LifecycleImpl._renderResponse(LifecycleImpl.java:706)
    oracle.adfinternal.view.faces.lifecycle.LifecycleImpl._executePhase(LifecycleImpl.java:244)
    oracle.adfinternal.view.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:199)
    javax.faces.webapp.FacesServlet.service(FacesServlet.java:246)
    weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
    weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
    weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:206)
    weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    oracle.adf.model.servlet.ADFBindingFilter.doFilter(ADFBindingFilter.java:134)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    oracle.adfinternal.view.faces.webapp.rich.RegistrationFilter.doFilter(RegistrationFilter.java:69)
    org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:411)
    oracle.adfinternal.view.faces.activedata.AdsFilter.doFilter(AdsFilter.java:44)
    org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:411)
    org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl._doFilterImpl(TrinidadFilterImpl.java:210)
    org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl.doFilter(TrinidadFilterImpl.java:126)
    org.apache.myfaces.trinidad.webapp.TrinidadFilter.doFilter(TrinidadFilter.java:91)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:94)
    oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:299)
    oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:406)
    oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:82)
    oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:70)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    oracle.adf.library.webapp.LibraryFilter.doFilter(LibraryFilter.java:147)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    oracle.dms.wls.DMSServletFilter.doFilter(DMSServletFilter.java:114)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:25)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.doIt(WebAppServletContext.java:3656)
    weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3650)
    weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:308)
    weblogic.security.service.SecurityManager.runAs(SecurityManager.java:117)
    weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2204)
    weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2126)
    weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1412)
    weblogic.work.ExecuteThread.execute(ExecuteThread.java:198)
    weblogic.work.ExecuteThread.run(ExecuteThread.java:165)
    SUBSYSTEM = WebLogicServer USERID = <WLS Kernel> SEVERITY = Error THREAD = [ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)' MSGID = BEA-000337 MACHINE = sis-prosun11.sis.ad TXID = CONTEXTID = 11d1def534ea1be0:3bbe257e:13286031cbf:-7ffd-0000000000002fe7 TIMESTAMP = 1316753908672
    WatchAlarmType: AutomaticReset
    WatchAlarmResetPeriod: 60000
    SMTPNotificationName: MS5-Health-Notification

    I found something interesting to the question I posted from the below link:
    http://www.mail-archive.com/[email protected]/msg03217.html

  • How does WebLogic clean up stuck threads?

    Hi,
    Using WebLogic 9.2.2 on Linux OS. In our logs, we are finding some stuck threads and it seems to be due to something WebLogic is doing. While we don't understand the cause (maybe the stack trace helps), does WebLogic do anything to remove stuck threads from its pool, or is there anything we can do to configure it to?
    <Jul 21, 2009 2:50:21 PM UTC> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "605" seconds working on the request "Http Request: /RoutingEngine/pcFlow.do", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
    Thread-69 "[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, priority=1, DAEMON> {
    jrockit.net.SocketNativeIO.readBytesPinned(Unknown Source)
    jrockit.net.SocketNativeIO.socketRead(Unknown Source)
    java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)
    java.net.SocketInputStream.read(SocketInputStream.java:107)
    java.io.BufferedInputStream.fill(BufferedInputStream.java:189)
    java.io.BufferedInputStream.read(BufferedInputStream.java:234)
    ^-- Holding lock: java.io.BufferedInputStream@3537712[thin lock]
    weblogic.net.http.MessageHeader.isHTTP(MessageHeader.java:214)
    weblogic.net.http.MessageHeader.parseHeader(MessageHeader.java:141)
    weblogic.net.http.HttpClient.parseHTTP(HttpClient.java:473)
    ^-- Holding lock: java.io.BufferedInputStream@3537712[thin lock]
    weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:332)
    ^-- Holding lock: java.io.BufferedInputStream@3537712[thin lock]
    weblogic.net.http.SOAPHttpURLConnection.getInputStream(SOAPHttpURLConnection.java:36)
    weblogic.wsee.util.dom.DOMParser.getWebLogicDocumentImpl(DOMParser.java:92)
    weblogic.wsee.util.dom.DOMParser.getDocument(DOMParser.java:56)
    weblogic.wsee.wsdl.WsdlReader.getDocument(WsdlReader.java:311)
    weblogic.wsee.wsdl.WsdlReader.getDocument(WsdlReader.java:305)
    weblogic.wsee.wsdl.WsdlSchema.parse(WsdlSchema.java:120)
    weblogic.wsee.wsdl.WsdlSchemaImport.parse(WsdlSchemaImport.java:55)
    weblogic.wsee.wsdl.WsdlSchema.parse(WsdlSchema.java:82)
    weblogic.wsee.wsdl.WsdlSchema.parse(WsdlSchema.java:62)
    weblogic.wsee.wsdl.WsdlTypes.parse(WsdlTypes.java:145)
    weblogic.wsee.wsdl.WsdlDefinitions.parseChild(WsdlDefinitions.java:452)
    weblogic.wsee.wsdl.WsdlExtensible.parse(WsdlExtensible.java:84)
    weblogic.wsee.wsdl.WsdlDefinitions.parse(WsdlDefinitions.java:378)
    weblogic.wsee.wsdl.WsdlDefinitions.parse(WsdlDefinitions.java:358)
    weblogic.wsee.wsdl.WsdlFactory.parse(WsdlFactory.java:50)
    weblogic.wsee.jaxrpc.ServiceImpl.loadWsdlDefinition(ServiceImpl.java:437)
    weblogic.wsee.jaxrpc.ServiceImpl.<init>(ServiceImpl.java:95)
    com.myco.regui.provisioning.client.ProvisioningAccountService_Impl.<init>(ProvisioningAccountService_Impl.java:21)
    com.myco.regui.provisioning.client.ProvisioningAccountService_Impl.<init>(ProvisioningAccountService_Impl.java:17)
    com.myco.regui.struts.accounts.AccountsForm.contactWPS(AccountsForm.java:208)
    com.myco.regui.struts.accounts.AccountsForm.validateAccountid(AccountsForm.java:161)
    com.myco.regui.struts.accounts.AccountsForm.validatePCAccountId(AccountsForm.java:144)
    com.myco.regui.struts.accounts.AccountsForm._validatePCFields(AccountsForm.java:112)
    com.myco.regui.struts.accounts.AccountsForm.validate(AccountsForm.java:83)
    org.apache.struts.chain.commands.servlet.ValidateActionForm.validate(ValidateActionForm.java:57)
    org.apache.struts.chain.commands.AbstractValidateActionForm.execute(AbstractValidateActionForm.java:94)
    org.apache.struts.chain.commands.ActionCommandBase.execute(ActionCommandBase.java:48)
    org.apache.commons.chain.impl.ChainBase.execute(ChainBase.java:176)
    org.apache.commons.chain.generic.LookupCommand.execute(LookupCommand.java:303)
    org.apache.commons.chain.impl.ChainBase.execute(ChainBase.java:176)
    org.apache.struts.chain.ComposableRequestProcessor.process(ComposableRequestProcessor.java:272)
    org.apache.struts.action.ActionServlet.process(ActionServlet.java:1903)
    org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:462)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:736)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:851)
    weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
    weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
    weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:198)
    weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:41)
    com.myco.regui.servlets.filters.TransactionFilter.doFilter(TransactionFilter.java:23)
    .......

    You can configure these actions to occur when not all threads are stuck, but the number of stuck threads have exceeded a configured threshold:
    * Shut down the Work Manager if it has stuck threads. A Work Manager that is shut down will refuse new work and reject existing work in
    the queue by sending a rejection message. In a cluster, clustered clients will fail over to another cluster member.
    * Shut down the application if there are stuck threads in the application. The application is shutdown by bringing it into admin mode. All
    Work Managers belonging to the application are shut down, and behave as described above.
    * Mark the server instance as failed and shut it down it down if there are stuck threads in the server. In a cluster, clustered clients that
    are connected or attempting to connect will fail over to another cluster member.
    Link : [http://e-docs.bea.com/wls/docs103/config_wls/overload.html]

  • WebLogic 11g - Stuck Thread issue observed during MS-ADS authentication

    Hi
    We have weblogic server 10.3.5 installed on RHEL 5.6. For our application authentication, we are getting user credentials authenticated for application login page using Microsoft Active Directory Service (MS-ADS) configured under WebLogic Security Realm. We are getting the below error message now due to STUCK Thread observed at JVM level.
    Could you please help me to fix this issue and my thanks in advance.
    ]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
    Thread-6424 "[STUCK] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, suspended, waiting, priority=1, DAEMON> {
    -- Waiting for notification on: netscape.ldap.LDAPResponseListener@3a8ecc70[fat lock]
    java.lang.Object.wait(Object.java:???)
    java.lang.Object.wait(Object.java:485)
    netscape.ldap.LDAPMessageQueue.waitForMessage(LDAPMessageQueue.java:198)
    netscape.ldap.LDAPMessageQueue.waitFirstMessage(LDAPMessageQueue.java:100)
    netscape.ldap.LDAPConnection.sendRequest(LDAPConnection.java:1782)
    ^-- Holding lock: netscape.ldap.LDAPConnection@3a8ecc83[thin lock]
    netscape.ldap.LDAPConnection.simpleBind(LDAPConnection.java:1752)
    netscape.ldap.LDAPConnection.authenticate(LDAPConnection.java:1260)
    netscape.ldap.LDAPConnection.authenticate(LDAPConnection.java:1273)
    netscape.ldap.LDAPConnection.authenticate(LDAPConnection.java:1240)
    netscape.ldap.LDAPConnection.referralRebind(LDAPConnection.java:4955)
    netscape.ldap.LDAPConnection.performReferrals(LDAPConnection.java:5065)
    netscape.ldap.LDAPConnection.checkSearchMsg(LDAPConnection.java:2632)
    netscape.ldap.LDAPConnection.search(LDAPConnection.java:2500)
    weblogic.security.providers.authentication.LDAPAtnDelegate.getDNForUser(LDAPAtnDelegate.java:3667)
    weblogic.security.providers.authentication.LDAPAtnDelegate.authenticate(LDAPAtnDelegate.java:3513)
    weblogic.security.providers.authentication.LDAPAtnLoginModuleImpl.login(LDAPAtnLoginModuleImpl.java:145)
    com.bea.common.security.internal.service.LoginModuleWrapper$1.run(LoginModuleWrapper.java:110)
    com.bea.common.security.internal.service.LoginModuleWrapper.login(LoginModuleWrapper.java:101)
    sun.reflect.GeneratedMethodAccessor2808.invoke(Unknown Source)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    java.lang.reflect.Method.invoke(Method.java:575)
    javax.security.auth.login.LoginContext.invoke(LoginContext.java:720)
    javax.security.auth.login.LoginContext.access$000(LoginContext.java:186)
    javax.security.auth.login.LoginContext$4.run(LoginContext.java:683)
    javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
    javax.security.auth.login.LoginContext.login(LoginContext.java:566)
    com.bea.common.security.internal.service.JAASLoginServiceImpl.login(JAASLoginServiceImpl.java:77)
    sun.reflect.GeneratedMethodAccessor366.invoke(Unknown Source)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    java.lang.reflect.Method.invoke(Method.java:575)
    com.bea.common.security.internal.utils.Delegator$ProxyInvocationHandler.invoke(Delegator.java:57)
    $Proxy28.login(Unknown Source)
    weblogic.security.service.internal.WLSJAASLoginServiceImpl$ServiceImpl.login(WLSJAASLoginServiceImpl.java:81)
    com.bea.common.security.internal.service.JAASAuthenticationServiceImpl.authenticate(JAASAuthenticationServiceImpl.java:74)
    sun.reflect.GeneratedMethodAccessor364.invoke(Unknown Source)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    java.lang.reflect.Method.invoke(Method.java:575)
    com.bea.common.security.internal.utils.Delegator$ProxyInvocationHandler.invoke(Delegator.java:57)
    $Proxy46.authenticate(Unknown Source)
    weblogic.security.service.WLSJAASAuthenticationServiceWrapper.authenticate(WLSJAASAuthenticationServiceWrapper.java:29)
    weblogic.security.service.PrincipalAuthenticator.authenticate(PrincipalAuthenticator.java:339)
    weblogic.security.services.Authentication.doLogin(Authentication.java:127)
    weblogic.security.services.Authentication.login(Authentication.java:69)
    weblogic.security.services.Authentication.login(Authentication.java:51)
    in.sis.ps.view.template.user.UserAdapter.doLogin(UserAdapter.java:681)
    sun.reflect.GeneratedMethodAccessor2824.invoke(Unknown Source)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    java.lang.reflect.Method.invoke(Method.java:575)
    com.sun.el.parser.AstValue.invoke(Unknown Source)
    com.sun.el.MethodExpressionImpl.invoke(Unknown Source)
    org.apache.myfaces.trinidad.component.MethodExpressionMethodBinding.invoke(MethodExpressionMethodBinding.java:46)
    com.sun.faces.application.ActionListenerImpl.processAction(ActionListenerImpl.java:85)
    org.apache.myfaces.trinidad.component.UIXCommand.broadcast(UIXCommand.java:171)
    oracle.adf.view.rich.component.fragment.ContextSwitchingComponent$1.run(ContextSwitchingComponent.java:130)
    oracle.adf.view.rich.component.fragment.ContextSwitchingComponent._processPhase(ContextSwitchingComponent.java:447)
    oracle.adf.view.rich.component.fragment.ContextSwitchingComponent.broadcast(ContextSwitchingComponent.java:122)
    oracle.adf.view.rich.component.fragment.UIXInclude.broadcast(UIXInclude.java:106)
    oracle.adf.view.rich.component.fragment.ContextSwitchingComponent$1.run(ContextSwitchingComponent.java:130)
    oracle.adf.view.rich.component.fragment.ContextSwitchingComponent._processPhase(ContextSwitchingComponent.java:447)
    oracle.adf.view.rich.component.fragment.ContextSwitchingComponent.broadcast(ContextSwitchingComponent.java:122)
    oracle.adf.view.rich.component.fragment.UIXInclude.broadcast(UIXInclude.java:106)
    javax.faces.component.UIViewRoot.broadcastEvents(UIViewRoot.java:720)
    javax.faces.component.UIViewRoot.processApplication(UIViewRoot.java:1247)
    oracle.adfinternal.view.faces.lifecycle.LifecycleImpl._invokeApplication(LifecycleImpl.java:964)
    oracle.adfinternal.view.faces.lifecycle.LifecycleImpl._executePhase(LifecycleImpl.java:259)
    oracle.adfinternal.view.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:181)
    javax.faces.webapp.FacesServlet.service(FacesServlet.java:284)
    weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
    weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
    weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:206)
    weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    oracle.adf.model.servlet.ADFBindingFilter.doFilter(ADFBindingFilter.java:115)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    oracle.adfinternal.view.faces.webapp.rich.RegistrationFilter.doFilter(RegistrationFilter.java:65)
    org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:459)
    oracle.adfinternal.view.faces.activedata.AdsFilter.doFilter(AdsFilter.java:44)
    org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:459)
    org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl._doFilterImpl(TrinidadFilterImpl.java:253)
    org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl.doFilter(TrinidadFilterImpl.java:148)
    org.apache.myfaces.trinidad.webapp.TrinidadFilter.doFilter(TrinidadFilter.java:91)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:111)
    oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:299)
    oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:405)
    oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:94)
    oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:98)
    oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:70)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:86)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:25)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3687)
    weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3681)
    weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:308)
    weblogic.security.service.SecurityManager.runAs(SecurityManager.java:116)
    weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2213)
    weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2135)
    weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1420)
    weblogic.work.ExecuteThread.execute(ExecuteThread.java:203)
    weblogic.work.ExecuteThread.run(ExecuteThread.java:170)
    }

    Hi,
    make sure to keep your Authentication provider in provider specific select group member search as limited where as default would be unlimited.
    Once you made this changes then it will solve this issue.
    Regards,
    Kal

  • Weblogic 8.14: Stuck Threads during JMS - IO native write operations

    G'day!
    Hope somebody can advise a solution on the following...
    Recently message loads have started to increase on our weblogic 8.14 cluster (3 servers) and we started to see an almost daily occurrence of Stuck Thread issues. From thread dumps we could see these were caused by jrockit/net/SocketNativeIO.write processes. We concluded we had reached the max performance limit for the file based JMS stores and so we moved these to database based JMS stores. This certainly seemed to improve matters - but we have recently seen the issue return. Which is puzzling, as technically we thought all messages would be persisted to the database, and any file IO on the application server would be eliminated. Our database resides on a separate dedicated oracle server. We tracked a recent cause to a large message (200K) which caused the Struck Thread - resulting in a build up of pending messages. After bouncing the server the message processed ok. Any thoughts on how to overcome these issues - greatly appreciated. Here's some detail on the stuck thread from the thread dump: (+Why does weblogic use IO when the JMS store is on a database?+). All messages are persistant.
    "ExecuteThread: '6' for queue: 'weblogic.kernel.Default'" prio=5 id=0xc00 pid=27544 active, daemon
    at jrockit/net/SocketNativeIO.write(Native Method)@0xf3aeae10
    at jrockit/net/SocketNativeIO.write(Unknown Source)@0xf3aeae9c
    at jrockit/io/NativeIO.write(Unknown Source)@0xf3ae62c6
    at java/net/AbstractSocketImpl$2.write(Unknown Source)@0xf3aeada4
    at jrockit/io/NativeIOOutputStream.write(Unknown Source)@0xf3ae61ec
    at jrockit/io/NativeIOOutputStream.write(Unknown Source)@0xf3ae618d
    at java/io/DataOutputStream.write(Unknown Source)@0xf4e3ec25
    ^-- Holding lock: java/io/DataOutputStream@0x4309fc80[thin lock]
    at com/ibm/mq/MQInternalCommunications.send(MQInternalCommunications.java:1022)@0xed34364a
    ^-- Holding lock: com/ibm/mq/MQInternalCommunications@0x4309f7e0[thin lock]
    at com/ibm/mq/MQSESSIONClient.lowLevelComms(MQSESSIONClient.java:2832)@0xed342fbd
    ^-- Holding lock: java/lang/Integer@0x4309f8a0[thin lock]
    at com/ibm/mq/MQSESSIONClient.MQPUT(MQSESSIONClient.java:3844)@0xed34271b
    at com/ibm/mq/MQQueue.putMsg2(MQQueue.java:1486)@0xed342262
    ^-- Holding lock: com/ibm/mq/MQSPIQueue@0x4312ce80[thin lock]
    at com/ibm/mq/jms/MQMessageProducer.sendInternal(MQMessageProducer.java:1569)@0xed33d1ab
    at com/ibm/mq/jms/MQMessageProducer.send(MQMessageProducer.java:1012)@0xed860a49
    at com/ibm/mq/jms/MQMessageProducer.send(MQMessageProducer.java:1046)@0xed8605f4
    at weblogic/jms/adapter/JMSBaseConnection.sendInternal(JMSBaseConnection.java:667)@0xef7df0bf
    ^-- Holding lock: weblogic/jms/adapter/JMSBaseConnection@0x43093748[thin lock]
    at weblogic/jms/adapter/JMSBaseConnection.access$200(JMSBaseConnection.java:80)@0xef7dee28
    at weblogic/jms/adapter/JMSBaseConnection$6.run(JMSBaseConnection.java:647)@0xef7dee1d
    at weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)@0xf2fcbb88
    at weblogic/security/service/SecurityManager.runAs(SecurityManager.java:147)@0xf2fcbb0a
    at weblogic/jms/adapter/JMSBaseConnection.send(JMSBaseConnection.java:644)@0xef7decf3
    at weblogic/jms/adapter/JMSConnectionHandle.send(JMSConnectionHandle.java:144)@0xef7dec7c
    at jrockit/reflect/NativeMethodInvoker.invoke0(Native Method)@0xf4f10d10
    at jrockit/reflect/NativeMethodInvoker.invoke(Unknown Source)@0xf4f10e98
    at jrockit/reflect/VirtualNativeMethodInvoker.invoke(Unknown Source)@0xf4e86d70
    at java/lang/reflect/Method.invoke(Unknown Source)@0xf4f104c3
    at weblogic/connector/common/internal/ConnectionWrapper.invoke(ConnectionWrapper.java:149)@0xed96a498
    at $Proxy14.send(Unknown Source)@0xef7dec04
    at weblogic/jms/bridge/internal/MessagingBridge.onMessageInternal(MessagingBridge.java:1258)@0xed857b25
    at weblogic/jms/bridge/internal/MessagingBridge.onMessage(MessagingBridge.java:1181)@0xed857864
    at weblogic/jms/adapter/JMSBaseConnection$27.run(JMSBaseConnection.java:1943)@0xef7de623
    at weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)@0xf2fcbb88
    at weblogic/security/service/SecurityManager.runAs(SecurityManager.java:147)@0xf2fcbb0a
    at weblogic/jms/adapter/JMSBaseConnection.onMessage(JMSBaseConnection.java:1939)@0xef7de586
    at weblogic/jms/client/JMSSession.onMessage(JMSSession.java:2678)@0xeee7e30c
    at weblogic/jms/client/JMSSession.execute(JMSSession.java:2598)@0xeee7d25d
    at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:219)@0xf4e14070
    at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:178)@0xf4e1fddb
    at java/lang/Thread.startThreadFromVM(Unknown Source)@0xf4f3c9f3
    Thanks
    C.

    I'm not sure what operating system you're running, but I only have experience of WLS8.1 on Windows.
    We did extensive load testing of JMS before we went live, and found JDBC file stores to be rubbish in comparison to file store. When a message is persisted, it has to be written to the database, then when it is consumed, it must be deleted from the table. This simply killed our system when we load tested it.
    One thing we did do though, was use a second JMS server and second file store for one particularly heavy queue, where we did have IO issues. This was with a JMS distributed destination. We also use a synchronous write policy of cache-flush on the file store and we also have configured a paging store to page messages so the app server does not run out of memory.
    200K isn't by any means a huge message though, and shouldn't be any problem at all, we have 1mb messages quite regularly.
    I notice from the stack trace, that you have MQ series aswell. I'm not experienced with MQ, so can't give any information, but what I would try is the following.
    Using a JMS server with JMS File storage, develop a small java client to write as many messages to the queue and effectively stress the JMS subsystem just to establish if the filestore is the problem. This should be done without a connection to MQ and also use large messages. if you don't get any issues and WebLogic JMS with file store looks ok, then attach a consumer to the queue, which should also be a simple java client. Then you have production and consumption going on, which will result in writes and read's/deletes from the jms file store.
    Once you've stress tested JMS and you are satisfied that its ok or not, then add the MQseries connection and see how that affects things.
    You may well have already done all this, so please feel free to ignore me! I did a load of this kind of stuff before we went live to try and bash JMS to bits, but I didn't manage it! The only thing we did was add a second file store and second JMS server.
    I just looked back in my support cases, and found we'd had a similar issue - disk fragmentation seemed to be one of the factors here too. The JMS file store was heavily fragmented. That's windows for you though!!!
    Hope that helps.
    Pete

  • Thread are gettin stuck causing the WebLogic server to hang

    We are developing an application for an US client which inclueds the following technology.
    - WebLogic 9.2 JMS (Foreign Server, Queue, Connection Factor, persistent Store)
    - Oracle 10g & Oracle 9i
    - J2EE (MDB)
    Different application uses different databases.
    After the server run for a few day it is getting a stuck thread and reported the same as a Hogging Thread in server -> monitoring -> thread. At this point the server hangs and stopped working with the status in 'warning' state (in the same page itself).
    The problem remains until the managed server is restarted.
    We have taken thread dump of different times and one of the error that we have got is attached below:
    "[STUCK] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.rjvm.ResponseImpl@1279315 WAITING
         java.lang.Object.wait(Native Method)
         weblogic.rjvm.ResponseImpl.waitForData(ResponseImpl.java:82)
         weblogic.rjvm.ResponseImpl.getTxContext(ResponseImpl.java:109)
         weblogic.messaging.dispatcher.DispatcherProxy.unmarshalResponse(DispatcherProxy.java:238)
         weblogic.messaging.dispatcher.DispatcherProxy.dispatchSyncTranFuture(DispatcherProxy.java:131)
         weblogic.messaging.dispatcher.DispatcherWrapperState.dispatchSyncTran(DispatcherWrapperState.java:241)
         weblogic.messaging.dispatcher.DispatcherWrapperState.dispatchSyncNoTran(DispatcherWrapperState.java:266)
         weblogic.jms.dispatcher.DispatcherAdapter.dispatchSyncNoTran(DispatcherAdapter.java:59)
         weblogic.jms.client.JMSProducer.toFEProducer(JMSProducer.java:1061)
         weblogic.jms.client.JMSProducer.deliveryInternal(JMSProducer.java:670)
         weblogic.jms.client.JMSProducer.sendInternal(JMSProducer.java:455)
         weblogic.jms.client.JMSProducer.send(JMSProducer.java:337)
         weblogic.jms.client.WLProducerImpl.send(WLProducerImpl.java:966)
         com.ctb.dex.messagehost.InputMessageListener.onMessage(Unknown Source)
         weblogic.ejb.container.internal.MDListener.execute(MDListener.java:429)
         weblogic.ejb.container.internal.MDListener.transactionalOnMessage(MDListener.java:335)
         weblogic.ejb.container.internal.MDListener.onMessage(MDListener.java:291)
         weblogic.jms.client.JMSSession.onMessage(JMSSession.java:4072)
         weblogic.jms.client.JMSSession.execute(JMSSession.java:3962)
         weblogic.jms.client.JMSSession$UseForRunnable.run(JMSSession.java:4490)
         weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
         weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
         weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
    Any body has any idea why this is happening? Please let me know in case you need to know more about the problem. Mail me at [email protected].

    Tom, we are experiencing the similar stuck thread max time out errors on our weblogic server(weblogic 8.1 sp5). Below is the thread dump:
    Blocked lock chains
    ===================
    Chain 4:
    "ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" id=101 idx=0xca tid=19298 waiting for java/lang/String@0x9c1fb10 held by:
    "ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" id=102 idx=0xcc tid=19299 in chain 3
    Open lock chains
    ================
    Chain 1:
    "ExecuteThread: '33' for queue: 'weblogic.kernel.Default'" id=62 idx=0x80 tid=19223 waiting for java/lang/String@0xbd00b10 held by:
    "ExecuteThread: '3' for queue: 'weblogic.kernel.Default'" id=32 idx=0x44 tid=19192 (waiting on notification)
    Chain 2:
    "ExecuteThread: '45' for queue: 'weblogic.kernel.Default'" id=74 idx=0x98 tid=19235 waiting for java/lang/String@0x1d1083a8 held by:
    "ExecuteThread: '11' for queue: 'weblogic.kernel.Default'" id=40 idx=0x54 tid=19200 (waiting on notification)
    Chain 3:
    "ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" id=100 idx=0xc8 tid=19297 waiting for java/lang/String@0x9c1fb10 held by:
    "ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" id=102 idx=0xcc tid=19299 (active)
    Can you help us in resolving the issue? This causes the weblogic server to hung and eventually non responsive.
    Below are our java version details:
    java version "1.4.2_13"
    Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_13-b06)
    Java HotSpot(TM) Client VM (build 1.4.2_13-b06, mixed mode)
    Can you let me know how to proceed and what can be done to avoid this issue?
    Thanks a bunch Tom.
    Regards
    RV

  • Weblogic 10.3.4 stuck thread after attempting to passify ApplicationModule

    Using ADF 11.1.1.4
    Is there any reason that Weblogic should ever end up with a stuck thread when attempting to passify an application module. See the stack trace below.
    There is no indication of what business components if any were responsible. It goes right up to the panel stretch layout and jsp template.
    ]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
      oracle.jbo.common.Base64$Base64OutputStream.getDeflatedBytes(Base64.java:135)
      oracle.jbo.common.Base64$Base64OutputStream.toString(Base64.java:116)
      oracle.jbo.common.VariableValueManagerImpl.writeVarsToXML(VariableValueManagerImpl.java:819)
      oracle.jbo.server.ViewObjectImpl.doPassivateSettings(ViewObjectImpl.java:19100)
      oracle.jbo.server.ApplicationModuleImpl.passivateVOs(ApplicationModuleImpl.java:7712)
      oracle.jbo.server.ApplicationModuleImpl.doPassivateState(ApplicationModuleImpl.java:7522)
      oracle.jbo.server.ApplicationModuleImpl.doPassivateAMState(ApplicationModuleImpl.java:7509)
      oracle.jbo.server.Serializer.passivate(Serializer.java:213)
      oracle.jbo.server.DBSerializer.passivateRootAM(DBSerializer.java:290)
      oracle.jbo.server.DBSerializer.passivateRootAM(DBSerializer.java:267)
      oracle.jbo.server.ApplicationModuleImpl.passivateStateInternal(ApplicationModuleImpl.java:5913)
      oracle.jbo.server.ApplicationModuleImpl.passivateState(ApplicationModuleImpl.java:5783)
      oracle.jbo.server.ApplicationPoolMessageHandler.doPoolMessage(ApplicationPoolMessageHandler.java:381)
      oracle.jbo.server.ApplicationModuleImpl.doPoolMessage(ApplicationModuleImpl.java:8933)
      oracle.jbo.common.ampool.ApplicationPoolImpl.sendPoolMessage(ApplicationPoolImpl.java:4496)
      oracle.jbo.common.ampool.ApplicationPoolImpl.manageReferencingState(ApplicationPoolImpl.java:1445)
      oracle.jbo.common.ampool.ApplicationPoolImpl.recycleReferencedInstance(ApplicationPoolImpl.java:2544)
      oracle.jbo.common.ampool.ApplicationPoolImpl.prepareApplicationModule(ApplicationPoolImpl.java:2365)
      oracle.jbo.common.ampool.ApplicationPoolImpl.doCheckout(ApplicationPoolImpl.java:2270)
      oracle.jbo.common.ampool.ApplicationPoolImpl.useApplicationModule(ApplicationPoolImpl.java:3168)
      oracle.jbo.common.ampool.SessionCookieImpl.useApplicationModule(SessionCookieImpl.java:460)
      oracle.jbo.http.HttpSessionCookieImpl.useApplicationModule(HttpSessionCookieImpl.java:234)
      oracle.jbo.common.ampool.SessionCookieImpl.useApplicationModule(SessionCookieImpl.java:431)
      oracle.jbo.common.ampool.SessionCookieImpl.useApplicationModule(SessionCookieImpl.java:426)
      oracle.adf.model.bc4j.DCJboDataControl.initializeApplicationModule(DCJboDataControl.java:516)
      oracle.adf.model.bc4j.DCJboDataControl.getApplicationModule(DCJboDataControl.java:862)
      oracle.adf.model.binding.DCDataControl.setErrorHandler(DCDataControl.java:483)
      oracle.jbo.uicli.binding.JUApplication.setErrorHandler(JUApplication.java:261)
      oracle.adf.model.BindingContext.put(BindingContext.java:1300)
      oracle.adf.model.binding.DCDataControlReference.getDataControl(DCDataControlReference.java:174)
      oracle.adf.model.BindingContext.instantiateDataControl(BindingContext.java:1024)
      oracle.adf.model.dcframe.DataControlFrameImpl.doFindDataControl(DataControlFrameImpl.java:1282)
      oracle.adf.model.dcframe.DataControlFrameImpl.internalFindDataControl(DataControlFrameImpl.java:1186)
      oracle.adf.model.dcframe.DataControlFrameImpl.findDataControl(DataControlFrameImpl.java:1146)
      oracle.adf.model.BindingContext.internalFindDataControl(BindingContext.java:1149)
      oracle.adf.model.BindingContext.get(BindingContext.java:1102)
      oracle.adf.model.binding.DCParameter.evaluateValue(DCParameter.java:82)
      oracle.adf.model.binding.DCParameter.getValue(DCParameter.java:111)
      oracle.adf.model.binding.DCBindingContainer.getChildByName(DCBindingContainer.java:2708)
      oracle.adf.model.binding.DCBindingContainer.internalGet(DCBindingContainer.java:2756)
      oracle.adf.model.binding.DCExecutableBinding.get(DCExecutableBinding.java:115)
      oracle.adf.model.binding.DCUtil.findSpelObject(DCUtil.java:328)
      oracle.adf.model.binding.DCBindingContainer.evaluateParameterWithElCheck(DCBindingContainer.java:1460)
      oracle.adf.model.binding.DCBindingContainer.findDataControl(DCBindingContainer.java:1590)
      oracle.adf.model.binding.DCIteratorBinding.initDataControl(DCIteratorBinding.java:2470)
      oracle.adf.model.binding.DCIteratorBinding.getDataControl(DCIteratorBinding.java:2414)
      oracle.adf.model.binding.DCIteratorBinding.refresh(DCIteratorBinding.java:4313)
      oracle.adf.model.binding.DCExecutableBinding.refreshIfNeeded(DCExecutableBinding.java:341)
      oracle.adf.model.binding.DCIteratorBinding.getAllRowsInRange(DCIteratorBinding.java:2337)
      sun.reflect.GeneratedMethodAccessor437.invoke(Unknown Source)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      java.lang.reflect.Method.invoke(Method.java:597)
      oracle.jbo.common.JboBeanUtils.getProperty(JboBeanUtils.java:113)
      oracle.jbo.common.JboAbstractMap.internalGet(JboAbstractMap.java:97)
      oracle.adf.model.binding.DCExecutableBinding.internalGet(DCExecutableBinding.java:169)
      oracle.adf.model.binding.DCIteratorBinding.internalGet(DCIteratorBinding.java:4558)
      oracle.adf.model.binding.DCExecutableBinding.get(DCExecutableBinding.java:115)
      javax.el.MapELResolver.getValue(MapELResolver.java:164)
      com.sun.faces.el.DemuxCompositeELResolver._getValue(DemuxCompositeELResolver.java:173)
      com.sun.faces.el.DemuxCompositeELResolver.getValue(DemuxCompositeELResolver.java:200)
      com.sun.el.parser.AstValue.getValue(Unknown Source)
      com.sun.el.ValueExpressionImpl.getValue(Unknown Source)
      org.apache.myfaces.trinidad.bean.FacesBeanImpl.getProperty(FacesBeanImpl.java:68)
      org.apache.myfaces.trinidad.component.UIXComponentBase.getProperty(UIXComponentBase.java:1194)
      org.apache.myfaces.trinidad.component.UIXIterator.getValue(UIXIterator.java:706)
      org.apache.myfaces.trinidad.component.UIXCollection._flushCachedModel(UIXCollection.java:1386)
      org.apache.myfaces.trinidad.component.UIXCollection.__processFlattenedChildrenBegin(UIXCollection.java:1369)
      org.apache.myfaces.trinidad.component.UIXIterator.processFlattenedChildren(UIXIterator.java:156)
      org.apache.myfaces.trinidad.component.UIXComponent.processFlattenedChildren(UIXComponent.java:160)
      org.apache.myfaces.trinidad.component.UIXComponent.processFlattenedChildren(UIXComponent.java:290)
      org.apache.myfaces.trinidad.component.UIXComponent.processFlattenedChildren(UIXComponent.java:213)
      oracle.adfinternal.view.faces.renderkit.rich.PanelGroupLayoutRenderer.encodeAll(PanelGroupLayoutRenderer.java:315)
      oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1396)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:335)
      org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:767)
      javax.faces.component.UIComponent.encodeAll(UIComponent.java:937)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:399)
      oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2633)
      oracle.adfinternal.view.faces.renderkit.rich.ShowDetailItemRenderer.access$500(ShowDetailItemRenderer.java:31)
      oracle.adfinternal.view.faces.renderkit.rich.ShowDetailItemRenderer$ChildEncoderCallback.processComponent(ShowDetailItemRenderer.java:531)
      oracle.adfinternal.view.faces.renderkit.rich.ShowDetailItemRenderer$ChildEncoderCallback.processComponent(ShowDetailItemRenderer.java:504)
      org.apache.myfaces.trinidad.component.UIXComponent.processFlattenedChildren(UIXComponent.java:170)
      org.apache.myfaces.trinidad.component.UIXComponent.processFlattenedChildren(UIXComponent.java:290)
      org.apache.myfaces.trinidad.component.UIXComponent.encodeFlattenedChildren(UIXComponent.java:255)
      oracle.adfinternal.view.faces.renderkit.rich.ShowDetailItemRenderer._encodeChildren(ShowDetailItemRenderer.java:446)
      oracle.adfinternal.view.faces.renderkit.rich.ShowDetailItemRenderer.encodeAll(ShowDetailItemRenderer.java:115)
      oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1396)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:335)
      org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:767)
      javax.faces.component.UIComponent.encodeAll(UIComponent.java:937)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:399)
      oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2633)
      oracle.adf.view.rich.render.RichRenderer.encodeStretchedChild(RichRenderer.java:2004)
      oracle.adfinternal.view.faces.renderkit.rich.PanelAccordionRenderer._encodeNonScreenReaderDetailItem(PanelAccordionRenderer.java:1422)
      oracle.adfinternal.view.faces.renderkit.rich.PanelAccordionRenderer._encodeDetailItem(PanelAccordionRenderer.java:1483)
      oracle.adfinternal.view.faces.renderkit.rich.PanelAccordionRenderer.access$4800(PanelAccordionRenderer.java:52)
      oracle.adfinternal.view.faces.renderkit.rich.PanelAccordionRenderer$PanelEncoderCallback.processComponent(PanelAccordionRenderer.java:2503)
      oracle.adfinternal.view.faces.renderkit.rich.PanelAccordionRenderer$PanelEncoderCallback.processComponent(PanelAccordionRenderer.java:2447)
      org.apache.myfaces.trinidad.component.UIXComponent.processFlattenedChildren(UIXComponent.java:170)
      org.apache.myfaces.trinidad.component.UIXComponent.processFlattenedChildren(UIXComponent.java:290)
      org.apache.myfaces.trinidad.component.UIXComponent.encodeFlattenedChildren(UIXComponent.java:255)
      oracle.adfinternal.view.faces.renderkit.rich.PanelAccordionRenderer.encodeAll(PanelAccordionRenderer.java:456)
      oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1396)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:335)
      org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:767)
      javax.faces.component.UIComponent.encodeAll(UIComponent.java:937)
      oracle.adfinternal.view.faces.taglib.region.IncludeTag$FacetWrapper.encodeAll(IncludeTag.java:536)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:399)
      oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2633)
      oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer.encodeCenterFacet(PanelStretchLayoutRenderer.java:769)
      oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer._encodeCenterPane(PanelStretchLayoutRenderer.java:1140)
      oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer._encodeMiddlePanes(PanelStretchLayoutRenderer.java:348)
      oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer.encodeAll(PanelStretchLayoutRenderer.java:313)
      oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1396)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:335)
      org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:767)
      javax.faces.component.UIComponent.encodeAll(UIComponent.java:937)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:399)
      oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2633)
      oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer.encodeStartFacet(PanelStretchLayoutRenderer.java:805)
      oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer._encodeVerticalPane(PanelStretchLayoutRenderer.java:1369)
      oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer._encodeMiddlePanes(PanelStretchLayoutRenderer.java:347)
      oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer.encodeAll(PanelStretchLayoutRenderer.java:313)
      oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1396)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:335)
      org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:767)
      javax.faces.component.UIComponent.encodeAll(UIComponent.java:937)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:399)
      oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2633)
      oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer.encodeCenterFacet(PanelStretchLayoutRenderer.java:769)
      oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer._encodeCenterPane(PanelStretchLayoutRenderer.java:1140)
      oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer._encodeMiddlePanes(PanelStretchLayoutRenderer.java:348)
      oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer.encodeAll(PanelStretchLayoutRenderer.java:313)
      oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1396)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:335)
      org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:767)
      javax.faces.component.UIComponent.encodeAll(UIComponent.java:937)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:399)
      oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2633)
      oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer.encodeCenterFacet(PanelStretchLayoutRenderer.java:769)
      oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer._encodeCenterPane(PanelStretchLayoutRenderer.java:1140)
      oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer._encodeMiddlePanes(PanelStretchLayoutRenderer.java:348)
      oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer.encodeAll(PanelStretchLayoutRenderer.java:313)
      oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1396)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:335)
      org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:767)
      javax.faces.component.UIComponent.encodeAll(UIComponent.java:937)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:399)
      oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2633)
      oracle.adf.view.rich.render.RichRenderer.encodeStretchedChild(RichRenderer.java:2004)
      oracle.adfinternal.view.faces.renderkit.rich.PanelSplitterRenderer._renderPane(PanelSplitterRenderer.java:1353)
      oracle.adfinternal.view.faces.renderkit.rich.PanelSplitterRenderer.encodeAll(PanelSplitterRenderer.java:259)
      oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1396)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:335)
      org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:767)
      javax.faces.component.UIComponent.encodeAll(UIComponent.java:937)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:399)
      oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2633)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeAllChildren(CoreRenderer.java:415)
      oracle.adfinternal.view.faces.renderkit.rich.PageTemplateRenderer.encodeAll(PageTemplateRenderer.java:69)
      oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1396)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:335)
      org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:767)
      oracle.adf.view.rich.component.fragment.ContextSwitchingComponent.encodeEnd(ContextSwitchingComponent.java:155)
      javax.faces.component.UIComponent.encodeAll(UIComponent.java:937)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:399)
      oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2633)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeAllChildren(CoreRenderer.java:415)
      oracle.adfinternal.view.faces.renderkit.rich.FormRenderer.encodeAll(FormRenderer.java:220)
      oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1396)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:335)
      org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:767)
      javax.faces.component.UIComponent.encodeAll(UIComponent.java:937)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:399)
      oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2633)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeAllChildren(CoreRenderer.java:415)
      oracle.adfinternal.view.faces.renderkit.rich.DocumentRenderer.encodeAll(DocumentRenderer.java:1181)
      oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1396)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:335)
      org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:767)
      javax.faces.component.UIComponent.encodeAll(UIComponent.java:937)
      javax.faces.component.UIComponent.encodeAll(UIComponent.java:933)
      com.sun.faces.application.ViewHandlerImpl.doRenderView(ViewHandlerImpl.java:266)
      com.sun.faces.application.ViewHandlerImpl.renderView(ViewHandlerImpl.java:197)
      javax.faces.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:189)
      org.apache.myfaces.trinidadinternal.application.ViewHandlerImpl.renderView(ViewHandlerImpl.java:193)
      oracle.adfinternal.view.faces.lifecycle.LifecycleImpl._renderResponse(LifecycleImpl.java:777)
      oracle.adfinternal.view.faces.lifecycle.LifecycleImpl._executePhase(LifecycleImpl.java:293)
      oracle.adfinternal.view.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:213)
      javax.faces.webapp.FacesServlet.service(FacesServlet.java:266)
      weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
      weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
      weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:300)
      weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
      weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
      oracle.adf.model.servlet.ADFBindingFilter.doFilter(ADFBindingFilter.java:205)
      weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
      oracle.adfinternal.view.faces.webapp.rich.RegistrationFilter.doFilter(RegistrationFilter.java:106)
      org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:446)
      oracle.adfinternal.view.faces.activedata.AdsFilter.doFilter(AdsFilter.java:60)
      org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:446)
      org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl._doFilterImpl(TrinidadFilterImpl.java:271)
      org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl.doFilter(TrinidadFilterImpl.java:177)
      org.apache.myfaces.trinidad.webapp.TrinidadFilter.doFilter(TrinidadFilter.java:92)
      weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
      oracle.adf.library.webapp.LibraryFilter.doFilter(LibraryFilter.java:175)
      weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
      oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:111)
      java.security.AccessController.doPrivileged(Native Method)
      oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:313)
      oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:413)
      oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:94)
      oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:161)
      oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:71)
      weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
      oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:136)
      weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
      weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:27)
      weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
      weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3715)
      weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3681)
      weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
      weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
      weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2277)
      weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2183)
      weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1454)
      weblogic.work.ExecuteThread.execute(ExecuteThread.java:207)
      weblogic.work.ExecuteThread.run(ExecuteThread.java:176)
    SUBSYSTEM = WebLogicServer USERID = <WLS Kernel> SEVERITY = Error THREAD = [ACTIVE] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)' MSGID = BEA-000337 MACHINE = app-sahe.dmz TXID =  CONTEXTID = ed3140ecffa37389:3c925abd:13f6031c9fe:-8000-0000000000015dff TIMESTAMP = 1371715760129 
    WatchAlarmType: AutomaticReset
    WatchAlarmResetPeriod: 600000

    Was their any changes in any of configuration files or something else. If not, then try bouncing the system once again and see if its working.

  • Stuck thread in adf application with weblogic

    hi every body
    I am working in JDev 11.1.1.4 and weblogic 10.3.4
    when I deploy my Application in my production weblogic
    after some time (it does not constant it is variable) I got warnning on the server
    when I check the server I got some stuck thread
    any idea or notes how can I know what is the reason of these stuck threads?
    is there any objects have thread unsafe issue, resource contention or race condition in ADF?
    BR,
    Alaa

    actully I do that guys
    but what I get does not have any related to my application
    even those http request does not request the same page
    this is the stuck thread when I dump
    "[STUCK] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" id=61 idx=0xe8 tid=13369 prio=1 alive, waiting, native_blocked, daemon
    -- Waiting for notification on: oracle/adfinternal/controller/util/RequestLock@0x88e06b58[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
    at java/lang/Object.wait(Object.java:485)[inlined]
    at oracle/adfinternal/controller/util/RequestLock.lock(RequestLock.java:42)[inlined]
    at oracle/adfinternal/controller/state/RootViewPortContextImpl.lockViewPortRequestLock(RootViewPortContextImpl.java:604)[optimized]
    ^-- Lock released while waiting: oracle/adfinternal/controller/util/RequestLock@0x88e06b58[fat lock]
    at oracle/adfinternal/controller/state/ControllerState.initializeRequest(ControllerState.java:833)[inlined]
    at oracle/adfinternal/controller/state/ControllerState.initializeRequest(ControllerState.java:754)[inlined]
    at oracle/adfinternal/controller/application/AdfcConfigurator.beginRequest(AdfcConfigurator.java:50)[optimized]
    at org/apache/myfaces/trinidadinternal/config/GlobalConfiguratorImpl._startConfiguratorServiceRequest(GlobalConfiguratorImpl.java:562)[inlined]
    at org/apache/myfaces/trinidadinternal/config/GlobalConfiguratorImpl.beginRequest(GlobalConfiguratorImpl.java:212)[optimized]
    at org/apache/myfaces/trinidadinternal/webapp/TrinidadFilterImpl.doFilter(TrinidadFilterImpl.java:155)[optimized]
    at org/apache/myfaces/trinidad/webapp/TrinidadFilter.doFilter(TrinidadFilter.java:92)[optimized]
    at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[optimized]
    at oracle/adf/library/webapp/LibraryFilter.doFilter(LibraryFilter.java:175)[optimized]
    at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[optimized]
    at oracle/security/jps/ee/http/JpsAbsFilter$1.run(JpsAbsFilter.java:111)[optimized]
    at jrockit/vm/AccessController.doPrivileged(AccessController.java:254)[inlined]
    at oracle/security/jps/util/JpsSubject.doAsPrivileged(JpsSubject.java:313)[inlined]
    at oracle/security/jps/ee/util/JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:413)[inlined]
    at oracle/security/jps/ee/http/JpsAbsFilter.runJaasMode(JpsAbsFilter.java:94)[inlined]
    at oracle/security/jps/ee/http/JpsAbsFilter.doFilter(JpsAbsFilter.java:161)[optimized]
    at oracle/security/jps/ee/http/JpsFilter.doFilter(JpsFilter.java:71)[optimized]
    at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[optimized]
    at oracle/dms/servlet/DMSServletFilter.doFilter(DMSServletFilter.java:136)[optimized]
    at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[optimized]
    at weblogic/servlet/internal/RequestEventsFilter.doFilter(RequestEventsFilter.java:27)[optimized]
    at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[inlined]
    at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3715)[inlined]
    at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3681)[optimized]
    at weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)[optimized]
    at weblogic/security/service/SecurityManager.runAs(SecurityManager.java:120)[inlined]
    at weblogic/servlet/internal/WebAppServletContext.securedExecute(WebAppServletContext.java:2277)[inlined]
    at weblogic/servlet/internal/WebAppServletContext.execute(WebAppServletContext.java:2183)[optimized]
    at weblogic/servlet/internal/ServletRequestImpl.run(ServletRequestImpl.java:1454)[optimized]
    at weblogic/work/ExecuteThread.execute(ExecuteThread.java:207)[optimized]
    at weblogic/work/ExecuteThread.run(ExecuteThread.java:176)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
    any another ideas?
    BR,
    Alaa

  • Weblogic server + stuck threads

    Hi All,
    We have java application deployed on weblogic server 10.1.3.4 which is integrated with BPEL (SOA Oracle 11g). The application creates a task through BPEL. We have configured StuckThread Max Time to 600 secs. Now after certain time (5-7 days) the servers goes into warning state due to stuck threads and finally it gets crashed
    We noticed the OOM error in jrockit dump, could any one help me here as how to avoid this? your help is very much appreciated, thanks

    Regarding
    Sample of your Weblogic logs showing the java.lang.OutOfMemoryError along with Stack Trace if available
    Please find below
    [ACTIVE] ExecuteThread: '20' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <532776f80ba738ab:-48a8d4c8:133068f0cee:-8000-00000000000001a5> <1322080967160> <BEA-080003> <RuntimeException thrown by rmi server: javax.management.remote.rmi.RMIConnectionImpl.invoke(Ljavax.management.ObjectName;Ljava.lang.String;Ljava.rmi.MarshalledObject;[Ljava.lang.String;Ljavax.security.auth.Subject;)
    javax.management.RuntimeErrorException: java.lang.OutOfMemoryError: allocLargeObjectOrArray: [B, size 1335312.
    javax.management.RuntimeErrorException: java.lang.OutOfMemoryError: allocLargeObjectOrArray: [B, size 1335312
            at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:858)
            at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrowMaybeMBeanException(DefaultMBeanServerInterceptor.java:869)
            at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:838)
            at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
            at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$16.run(WLSMBeanServerInterceptorBase.java:449)
            at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:447)
            at weblogic.management.mbeanservers.internal.JMXContextInterceptor.invoke(JMXContextInterceptor.java:263)
            at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$16.run(WLSMBeanServerInterceptorBase.java:449)
            at weblogic.management.mbeanservers.internal.SecurityInterceptor.invoke(SecurityInterceptor.java:444)
            at weblogic.management.jmx.mbeanserver.WLSMBeanServer.invoke(WLSMBeanServer.java:323)
            at weblogic.management.mbeanservers.internal.JMXConnectorSubjectForwarder$11.run(JMXConnectorSubjectForwarder.java:661)
            at weblogic.management.mbeanservers.internal.JMXConnectorSubjectForwarder.invoke(JMXConnectorSubjectForwarder.java:654)
            at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
            at javax.management.remote.rmi.RMIConnectionImpl_WLSkel.invoke(Unknown Source)
            at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:668)
            at weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:118)
            at weblogic.work.ExecuteThread.execute(ExecuteThread.java:207)
            at weblogic.work.ExecuteThread.run(ExecuteThread.java:176)
    Caused By: java.lang.OutOfMemoryError: allocLargeObjectOrArray: [B, size 1335312
            at java.util.Arrays.copyOf(Arrays.java:2786)
            at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
            at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1847)
      at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
            at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1847)
            at java.io.ObjectOutputStream$BlockDataOutputStream.flush(ObjectOutputStream.java:1792)
            at java.io.ObjectOutputStream.flush(ObjectOutputStream.java:699)
            at java.io.ObjectOutputStream.close(ObjectOutputStream.java:720)
            at oracle.dms.jmx.app.ServerAggreMBeanImpl.getTableGroupInBytes(ServerAggreMBeanImpl.java:328)
            at oracle.dms.jmx.app.ServerAggreMBeanWrapper.invokeOperation(ServerAggreMBeanWrapper.java:71)
            at oracle.dms.jmx.app.AggreMBeanWrapper.invoke(AggreMBeanWrapper.java:167)
            at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
            at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
            at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$16.run(WLSMBeanServerInterceptorBase.java:449)
            at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:447)
            at weblogic.management.mbeanservers.internal.JMXContextInterceptor.invoke(JMXContextInterceptor.java:263)
            at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$16.run(WLSMBeanServerInterceptorBase.java:449)
            at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:447)
            at weblogic.management.mbeanservers.internal.SecurityInterceptor.invoke(SecurityInterceptor.java:444)
            at weblogic.management.jmx.mbeanserver.WLSMBeanServer.invoke(WLSMBeanServer.java:323)
            at weblogic.management.mbeanservers.internal.JMXConnectorSubjectForwarder$11$1.run(JMXConnectorSubjectForwarder.java:663)
            at weblogic.management.mbeanservers.internal.JMXConnectorSubjectForwarder$11.run(JMXConnectorSubjectForwarder.java:661)
            at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
            at weblogic.management.mbeanservers.internal.JMXConnectorSubjectForwarder.invoke(JMXConnectorSubjectForwarder.java:654)
            at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
            at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
            at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
            at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1367)
            at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
            at javax.management.remote.rmi.RMIConnectionImpl_WLSkel.invoke(Unknown Source)
            at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:667)
            at weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:522)
            at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
            at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:146)
            at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:518)
            at weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:118)
            at weblogic.work.ExecuteThread.execute(ExecuteThread.java:207)
            at weblogic.work.ExecuteThread.run(ExecuteThread.java:176)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               

  • Need help with STUCK Threads.  What's causing them?

    In our production environment, we have a soa suite cluster with 3 nodes. All 3 of the servers keep turning up stuck and hogging threads, and eventually they all grind to a halt where I can't even view composite instances through em. There is a bpel process that is using a dbadapter to poll for records every 5 seconds and then performs some service call outs and other dbadapters. This process is generating thousands of instances a day and I think it is the culprit. If I view the logs, I see one of the dbadapters that it calls out to fails occasionally with a PK violation. I looked at the developer's code, and they have no error handling in the process at all. In the logs, there will the a hit for the PK violation, then about 10 minutes later the a STUCK thread gets logged(the configured timeout is 600 seconds, or 10 minutes). Would not handling an error in a BPEL process hog/STUCK a thread? I need to be able to show what code is causing the issue so I can have a resolution path.
    I also see a note in the trace about something holding a lock:
    Holding lock: oracle.jdbc.driver.T4CConnection@1fbc3486[thin lock]....would that be an un-rolled back transaction?
    Here is the dump of the STUCK thread. Any help is greatly appreciated. Thanks!
    <Aug 5, 2011 10:30:02 AM EDT> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '64' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "634" seconds working on the request "oracle.soa.management.internal.ejb.impl.FacadeFinderBean_4vacyo_FacadeFinderBeanImpl", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
    Thread-517 "[STUCK] ExecuteThread: '64' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, priority=1, DAEMON> {
    jrockit.net.SocketNativeIO.readBytesPinned(SocketNativeIO.java:???)
    jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:24)
    java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)
    java.net.SocketInputStream.read(SocketInputStream.java:107)
    oracle.net.nt.MetricsEnabledInputStream.read(TcpNTAdapter.java:553)
    oracle.net.ns.Packet.receive(Packet.java:249)
    oracle.net.ns.DataPacket.receive(DataPacket.java:94)
    oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:168)
    oracle.net.ns.NetInputStream.read(NetInputStream.java:113)
    oracle.net.ns.NetInputStream.read(NetInputStream.java:96)
    oracle.net.ns.NetInputStream.read(NetInputStream.java:81)
    oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:120)
    oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:75)
    oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1081)
    oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1062)
    oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:588)
    oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:70)
    oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:854)
    oracle.jdbc.driver.T4CPreparedStatement.executeMaybeDescribe(T4CPreparedStatement.java:918)
    oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1252)
    oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3541)
    oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3605)
    ^-- Holding lock: oracle.jdbc.driver.T4CConnection@1fbc3486[thin lock]
    oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1667)
    weblogic.jdbc.wrapper.PreparedStatement.executeQuery(PreparedStatement.java:123)
    org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeSelect(DatabaseAccessor.java:887)
    org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:541)
    org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:530)
    org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:521)
    org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:202)
    org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:192)
    org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelectCall(DatasourceCallQueryMechanism.java:254)
    org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelect(DatasourceCallQueryMechanism.java:245)
    org.eclipse.persistence.queries.DataReadQuery.executeNonCursor(DataReadQuery.java:188)
    org.eclipse.persistence.queries.DataReadQuery.executeDatabaseQuery(DataReadQuery.java:141)
    org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:603)
    org.eclipse.persistence.queries.DataReadQuery.execute(DataReadQuery.java:115)
    org.eclipse.persistence.queries.DatabaseQuery.executeInUnitOfWork(DatabaseQuery.java:589)
    org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2898)
    org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1218)
    org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1190)
    org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1176)
    org.eclipse.persistence.internal.jpa.EJBQueryImpl.executeReadQuery(EJBQueryImpl.java:378)
    org.eclipse.persistence.internal.jpa.EJBQueryImpl.getSingleResult(EJBQueryImpl.java:701)
    weblogic.deployment.TransactionalEntityManagerProxyImpl$QueryImpl.getSingleResult(TransactionalEntityManagerProxyImpl.java:232)
    com.collaxa.cube.persistence.dao.impl.jpa.JpaCubeInstanceDaoImpl.getNumberOfInstances(JpaCubeInstanceDaoImpl.java:439)
    oracle.fabric.CubeServiceEngine.getNumberOfComponentInstances(CubeServiceEngine.java:1962)
    oracle.soa.management.internal.ejb.impl.FacadeFinderBeanImpl.getNumberOfComponentInstances(FacadeFinderBeanImpl.java:1618)
    oracle.soa.management.internal.ejb.impl.FacadeFinderBeanImpl.getComponentSnapShot(FacadeFinderBeanImpl.java:1653)
    sun.reflect.GeneratedMethodAccessor5852.invoke(Unknown Source)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    java.lang.reflect.Method.invoke(Method.java:575)
    com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.jee.intercept.MethodInvocationInvocationContext.proceed(MethodInvocationInvocationContext.java:104)
    oracle.security.jps.ee.ejb.JpsAbsInterceptor$1.run(JpsAbsInterceptor.java:88)
    oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:299)
    oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:406)
    oracle.security.jps.wls.JpsWeblogicEjbInterceptor.runJaasMode(JpsWeblogicEjbInterceptor.java:61)
    oracle.security.jps.ee.ejb.JpsAbsInterceptor.intercept(JpsAbsInterceptor.java:81)
    oracle.security.jps.ee.ejb.JpsInterceptor.intercept(JpsInterceptor.java:101)
    sun.reflect.GeneratedMethodAccessor863.invoke(Unknown Source)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    java.lang.reflect.Method.invoke(Method.java:575)
    com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
    com.bea.core.repackaged.springframework.jee.intercept.JeeInterceptorInterceptor.invoke(JeeInterceptorInterceptor.java:66)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
    com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:102)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.jee.spi.MethodInvocationVisitorImpl.visit(MethodInvocationVisitorImpl.java:37)
    weblogic.ejb.container.injection.EnvironmentInterceptorCallbackImpl.callback(EnvironmentInterceptorCallbackImpl.java:22)
    com.bea.core.repackaged.springframework.jee.spi.EnvironmentInterceptor.invoke(EnvironmentInterceptor.java:49)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:86)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
    com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:102)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:151)
    $Proxy309.getComponentSnapShot(Unknown Source)
    oracle.soa.management.internal.ejb.impl.FacadeFinderBean_4vacyo_FacadeFinderBeanImpl.getComponentSnapShot(FacadeFinderBean_4vacyo_FacadeFinderBeanImpl.java:1604)
    oracle.soa.management.internal.ejb.impl.FacadeFinderBean_4vacyo_FacadeFinderBeanImpl_WLSkel.invoke(Unknown Source)
    weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:567)
    weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:477)
    weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:350)
    weblogic.security.service.SecurityManager.runAs(SecurityManager.java:143)
    weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:425)
    weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:117)
    weblogic.work.ExecuteThread.execute(ExecuteThread.java:198)
    weblogic.work.ExecuteThread.run(ExecuteThread.java:165)
    >
    <Aug 5, 2011 10:30:02 AM EDT> <Notice> <Diagnostics> <BEA-320068> <Watch 'StuckThread' with severity 'Notice' on server 'soa_server1' has triggered at Aug 5, 2011 10:30:02 AM EDT. Notification details:
    WatchRuleType: Log
    WatchRule: (SEVERITY = 'Error') AND (MSGID = 'BEA-000337')
    WatchData: DATE = Aug 5, 2011 10:30:02 AM EDT SERVER = soa_server1 MESSAGE = [STUCK] ExecuteThread: '64' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "634" seconds working on the request "oracle.soa.management.internal.ejb.impl.FacadeFinderBean_4vacyo_FacadeFinderBeanImpl", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
    Thread-517 "[STUCK] ExecuteThread: '64' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, priority=1, DAEMON> {
    jrockit.net.SocketNativeIO.readBytesPinned(SocketNativeIO.java:???)
    jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:24)
    java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)
    java.net.SocketInputStream.read(SocketInputStream.java:107)
    oracle.net.nt.MetricsEnabledInputStream.read(TcpNTAdapter.java:553)
    oracle.net.ns.Packet.receive(Packet.java:249)
    oracle.net.ns.DataPacket.receive(DataPacket.java:94)
    oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:168)
    oracle.net.ns.NetInputStream.read(NetInputStream.java:113)
    oracle.net.ns.NetInputStream.read(NetInputStream.java:96)
    oracle.net.ns.NetInputStream.read(NetInputStream.java:81)
    oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:120)
    oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:75)
    oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1081)
    oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1062)
    oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:588)
    oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:70)
    oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:854)
    oracle.jdbc.driver.T4CPreparedStatement.executeMaybeDescribe(T4CPreparedStatement.java:918)
    oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1252)
    oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3541)
    oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3605)
    ^-- Holding lock: oracle.jdbc.driver.T4CConnection@1fbc3486[thin lock]
    oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1667)
    weblogic.jdbc.wrapper.PreparedStatement.executeQuery(PreparedStatement.java:123)
    org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeSelect(DatabaseAccessor.java:887)
    org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:541)
    org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:530)
    org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:521)
    org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:202)
    org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:192)
    org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelectCall(DatasourceCallQueryMechanism.java:254)
    org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelect(DatasourceCallQueryMechanism.java:245)
    org.eclipse.persistence.queries.DataReadQuery.executeNonCursor(DataReadQuery.java:188)
    org.eclipse.persistence.queries.DataReadQuery.executeDatabaseQuery(DataReadQuery.java:141)
    org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:603)
    org.eclipse.persistence.queries.DataReadQuery.execute(DataReadQuery.java:115)
    org.eclipse.persistence.queries.DatabaseQuery.executeInUnitOfWork(DatabaseQuery.java:589)
    org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2898)
    org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1218)
    org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1190)
    org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1176)
    org.eclipse.persistence.internal.jpa.EJBQueryImpl.executeReadQuery(EJBQueryImpl.java:378)
    org.eclipse.persistence.internal.jpa.EJBQueryImpl.getSingleResult(EJBQueryImpl.java:701)
    weblogic.deployment.TransactionalEntityManagerProxyImpl$QueryImpl.getSingleResult(TransactionalEntityManagerProxyImpl.java:232)
    com.collaxa.cube.persistence.dao.impl.jpa.JpaCubeInstanceDaoImpl.getNumberOfInstances(JpaCubeInstanceDaoImpl.java:439)
    oracle.fabric.CubeServiceEngine.getNumberOfComponentInstances(CubeServiceEngine.java:1962)
    oracle.soa.management.internal.ejb.impl.FacadeFinderBeanImpl.getNumberOfComponentInstances(FacadeFinderBeanImpl.java:1618)
    oracle.soa.management.internal.ejb.impl.FacadeFinderBeanImpl.getComponentSnapShot(FacadeFinderBeanImpl.java:1653)
    sun.reflect.GeneratedMethodAccessor5852.invoke(Unknown Source)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    java.lang.reflect.Method.invoke(Method.java:575)
    com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.jee.intercept.MethodInvocationInvocationContext.proceed(MethodInvocationInvocationContext.java:104)
    oracle.security.jps.ee.ejb.JpsAbsInterceptor$1.run(JpsAbsInterceptor.java:88)
    oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:299)
    oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:406)
    oracle.security.jps.wls.JpsWeblogicEjbInterceptor.runJaasMode(JpsWeblogicEjbInterceptor.java:61)
    oracle.security.jps.ee.ejb.JpsAbsInterceptor.intercept(JpsAbsInterceptor.java:81)
    oracle.security.jps.ee.ejb.JpsInterceptor.intercept(JpsInterceptor.java:101)
    sun.reflect.GeneratedMethodAccessor863.invoke(Unknown Source)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    java.lang.reflect.Method.invoke(Method.java:575)
    com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
    com.bea.core.repackaged.springframework.jee.intercept.JeeInterceptorInterceptor.invoke(JeeInterceptorInterceptor.java:66)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
    com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:102)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.jee.spi.MethodInvocationVisitorImpl.visit(MethodInvocationVisitorImpl.java:37)
    weblogic.ejb.container.injection.EnvironmentInterceptorCallbackImpl.callback(EnvironmentInterceptorCallbackImpl.java:22)
    com.bea.core.repackaged.springframework.jee.spi.EnvironmentInterceptor.invoke(EnvironmentInterceptor.java:49)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:86)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
    com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:102)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:151)
    $Proxy309.getComponentSnapShot(Unknown Source)
    oracle.soa.management.internal.ejb.impl.FacadeFinderBean_4vacyo_FacadeFinderBeanImpl.getComponentSnapShot(FacadeFinderBean_4vacyo_FacadeFinderBeanImpl.java:1604)
    oracle.soa.management.internal.ejb.impl.FacadeFinderBean_4vacyo_FacadeFinderBeanImpl_WLSkel.invoke(Unknown Source)
    weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:567)
    weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:477)
    weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:350)
    weblogic.security.service.SecurityManager.runAs(SecurityManager.java:143)
    weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:425)
    weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:117)
    weblogic.work.ExecuteThread.execute(ExecuteThread.java:198)
    weblogic.work.ExecuteThread.run(ExecuteThread.java:165)
    SUBSYSTEM = WebLogicServer USERID = <WLS Kernel> SEVERITY = Error THREAD = [ACTIVE] ExecuteThread: '96' for queue: 'weblogic.kernel.Default (self-tuning)' MSGID = BEA-000337 MACHINE = soaprod TXID = CONTEXTID = f5d0b85a929630a2:6eada0f7:1304e0a2bdc:-7ffb-00000000017e1605 TIMESTAMP = 1312554602698
    WatchAlarmType: AutomaticReset
    WatchAlarmResetPeriod: 600000
    >
    Edited by: Brutus35 on Aug 5, 2011 8:04 AM

    my impression is that you have trouble on the DB,
    monitor the sessions to see if there is some table locked, and increase the number of processes on the DB
    you can use SQLDeveloper for that
    anyway your TX should rollback after some time, check that you have a timeout in place, it's better to rollback a TX than having the entire server crash

  • SOA 11g: Intergation with BAM 11g failed with ThreadPool has stuck threads

    Hi,
    I have installed BAM and SOA server on local host. Now while integrating BAM with BAMAdapter it's going into long running state and finally failing with below error.
    <Notice> <Diagnostics> <mars.as.local> <soa_server1> <[ACTIVE] ExecuteThread: '14' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <c82d73b0a2a7776f:37a0afd7:130d1278d97:-8000-0000000000028a7f> <1309264042380> <BEA-320068> <Watch 'StuckThread' with severity 'Notice' on server 'soa_server1' has triggered at Jun 28, 2011 1:27:22 PM BST. Notification details:
    WatchRuleType: Log
    WatchRule: (SEVERITY = 'Error') AND ((MSGID = 'WL-000337') OR (MSGID = 'BEA-000337'))
    WatchData: DATE = Jun 28, 2011 1:27:22 PM BST SERVER = soa_server1 MESSAGE = *[STUCK] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "638" seconds working on the request "weblogic.servlet.internal.ServletRequestImpl@16e92176[*
    *POST /soa-infra/services/default/BAMInsert/bpelprocess1_client_ep HTTP/1.1*
    Connection: TE
    TE: trailers, deflate, gzip, compress
    User-Agent: Oracle HTTPClient Version 10h
    SOAPAction: "process"
    Accept-Encoding: gzip, x-gzip, compress, x-compress
    ECID-Context: 1.c82d73b0a2a7776f:2ced6156:130d1186ef6:-8000-00000000000046b2;kYjE0ZJOoOTLkKPOoLRKlSODoITT_G
    Content-type: text/xml; charset=UTF-8
    Content-Length: 270
    *]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:*
    *Thread-139 "[STUCK] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, suspended, sleeping, priority=1, DAEMON> {*
    *java.lang.Thread.sleep(Thread.java:???)*
    oracle.bam.common.remoting.BamEjbClient.getSession(BamEjbClient.java:973)
    oracle.bam.common.remoting.BamEjbClient.getADCSession(BamEjbClient.java:350)
    oracle.bam.adc.api.util.Context.<init>(Context.java:270)
    oracle.bam.adapter.adc.CachedConnection.<init>(CachedConnection.java:134)
    oracle.bam.adapter.adc.ADCManagedConnectionFactory.getCachedConnection(ADCManagedConnectionFactory.java:490)
    ^-- Holding lock: java.lang.String@13416588[fat lock]
    What could be root cause of this? Can anyone please help to find the solution?
    Even though both servers on same host why BAM connection is getting to much time and Putting server in warning state like :
    +[STUCK] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'+
    *Do i need to modify any config file at Admin or BAM level? if Yes, what parameter Do i need to change?*
    Please guide.
    Thanks,
    Sagar.

    Further Updates on this :
    I further debugged and found out few more things. This might help.
    In Weblogic Admin console Home > Summary of Deployments >oracle-bam(11.1.1)
    MDB application oracle-bam is NOT connected to messaging system with below error.
    EJBs
    MessageDispatcherBean - Error java.lang.IllegalArgumentException: Getting Deployment configuration...
    connectionFactoryJNDIName - Red Cross with warning
    destinationJNDIName - Red Cross with warning
    resourceAdapterJNDIName - Red Cross with warning
    Modules
    Also please check out another similar thread and please update.
    BAMAdapter Issue : java.rmi.ConnectException: Destination unreachable;
    Thanks,
    Sagar

  • Problem Deploying Composites to Oracle SOA 11.1.1.5 (STUCK Thread Issue)

    Hi All,
    I got stuck thread issues when deploying composites to Oracle SOA 11.1.1.5 running on Windows Server 2008 R2. Even the SimpleApproval composite which ships with the standard installation is having trouble deploying.
    The composites are deploying fine in Oracle SOA 11.1.1.5 running on Windows XP.
    I'm not so good with thread dumps, so I post it below for rescue.
    Please advice,
    Rommel
    <Jul 18, 2011 10:15:25 AM AST> <Error> <WebLogicServer> <BEA-000337> <[STUCK] Ex
    ecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)' has been bus
    y for "605" seconds working on the request "weblogic.servlet.internal.ServletReq
    uestImpl@32d047d4[
    POST /soa-infra/deployer HTTP/1.1
    Connection: TE
    TE: trailers, deflate, gzip, compress
    User-Agent: Oracle HTTPClient Version 10h
    Accept-Encoding: gzip, x-gzip, compress, x-compress
    ECID-Context:
    Authorization: Basic bmFnaGl3ZWJsb2dpY3NvYTp3ZWJsb2dpY3NvYTEwIQ==
    Content-type: application/octet-stream
    Content-Length: 25876
    ]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds
    . Stack trace:
    Thread-55 "[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-
    tuning)'" <alive, in native, suspended, priority=1, DAEMON> {
    jrockit.net.SocketNativeIO.readBytesPinned(SocketNativeIO.java:???)
    jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:24)
    java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)
    java.net.SocketInputStream.read(SocketInputStream.java:107)
    oracle.net.nt.MetricsEnabledInputStream.read(TcpNTAdapter.java:707)
    oracle.net.ns.Packet.receive(Packet.java:243)
    oracle.net.ns.DataPacket.receive(DataPacket.java:106)
    oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:309)
    oracle.net.ns.NetInputStream.read(NetInputStream.java:254)
    oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrap
    per.java:93)
    oracle.jdbc.driver.T4CMAREngine.getNBytes(T4CMAREngine.java:1573)
    oracle.jdbc.driver.T4C8TTILobd.unmarshalLobData(T4C8TTILobd.java:360)
    oracle.jdbc.driver.T4C8TTILob.readLOBD(T4C8TTILob.java:787)
    oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:303)
    oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:181)
    oracle.jdbc.driver.T4C8TTILob.read(T4C8TTILob.java:139)
    oracle.jdbc.driver.T4CConnection.getBytes(T4CConnection.java:2690)
    ^-- Holding lock: oracle.jdbc.driver.T4CConnection@32386377[thin lock]
    oracle.sql.BLOB.getBytes(BLOB.java:557)
    oracle.jdbc.driver.OracleBlobInputStream.needBytes(OracleBlobInputStream.jav
    a:188)
    oracle.jdbc.driver.OracleBufferedStream.readInternal(OracleBufferedStream.ja
    va:171)
    oracle.jdbc.driver.OracleBufferedStream.read(OracleBufferedStream.java:151)
    ^-- Holding lock: oracle.jdbc.driver.OracleBlobInputStream@341d0b87[thin loc
    k]
    java.io.BufferedInputStream.fill(BufferedInputStream.java:189)
    java.io.BufferedInputStream.read(BufferedInputStream.java:236)
    ^-- Holding lock: oracle.mds.internal.persistence.db.DBInputStream@341d0bb2[
    thin lock]
    oracle.xml.parser.v2.XMLReader.pushXMLReader(XMLReader.java:363)
    oracle.xml.parser.v2.XMLReader.pushXMLReader(XMLReader.java:230)
    oracle.xml.parser.v2.XMLParser.parse(XMLParser.java:217)
    oracle.fabric.common.wsdl.SchemaBuilder.parseXMLSrc(SchemaBuilder.java:1016)
    oracle.fabric.common.wsdl.SchemaBuilder.parseXSDSrc(SchemaBuilder.java:984)
    oracle.fabric.common.wsdl.SchemaBuilder.processSchemaImportsAndInclude(Schem
    aBuilder.java:755)
    oracle.fabric.common.wsdl.SchemaBuilder.loadEmbeddedSchemas(SchemaBuilder.ja
    va:474)
    oracle.fabric.common.wsdl.SchemaBuilder.loadSchemasFromWSDL(SchemaBuilder.ja
    va:423)
    oracle.fabric.common.wsdl.SchemaBuilder.loadAllSchemas(SchemaBuilder.java:21
    4)
    oracle.fabric.common.wsdl.SchemaManager.loadAllXSD(SchemaManager.java:209)
    ^-- Holding lock: oracle.fabric.common.wsdl.SchemaManager@36279195[thin lock
    oracle.fabric.common.wsdl.SchemaManager.getXSDEntityResolver(SchemaManager.j
    ava:424)
    oracle.fabric.composite.model.CompositeModel.loadImports(CompositeModel.java
    :361)
    oracle.fabric.composite.model.CompositeModel.getWSDLManager(CompositeModel.j
    ava:196)
    oracle.integration.platform.blocks.adapter.AbstractAdapterBindingComponent.g
    etWSDLDefinition(AbstractAdapterBindingComponent.java:168)
    oracle.integration.platform.blocks.adapter.AdapterReference.loadDefinition(A
    dapterReference.java:543)
    oracle.integration.platform.blocks.adapter.AdapterReference.load(AdapterRefe
    rence.java:377)
    oracle.integration.platform.blocks.adapter.AdapterReference.load(AdapterRefe
    rence.java:73)
    oracle.integration.platform.blocks.deploy.CompositeDeploymentConnection.depl
    oyReferences(CompositeDeploymentConnection.java:177)
    oracle.integration.platform.blocks.deploy.CompositeDeploymentConnection.depl
    oy(CompositeDeploymentConnection.java:81)
    oracle.integration.platform.blocks.deploy.CompositeDeploymentManagerImpl.ini
    tDeployment(CompositeDeploymentManagerImpl.java:143)
    oracle.integration.platform.blocks.deploy.CompositeDeploymentManagerImpl.loa
    d(CompositeDeploymentManagerImpl.java:61)
    sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:?
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:27
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
    .java:25)
    java.lang.reflect.Method.invoke(Method.java:575)
    org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopU
    tils.java:306)
    org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint
    (ReflectiveMethodInvocation.java:182)
    org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
    iveMethodInvocation.java:148)
    oracle.integration.platform.blocks.deploy.DeploymentEventPublisher.invoke(De
    ploymentEventPublisher.java:57)
    org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
    iveMethodInvocation.java:148)
    org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopPro
    xy.java:151)
    $Proxy309.load(Unknown Source)
    oracle.integration.platform.blocks.deploy.StandaloneCompositeDeploymentCoord
    inatorImpl.coordinateCompositeDeployment(StandaloneCompositeDeploymentCoordinato
    rImpl.java:54)
    oracle.integration.platform.blocks.deploy.servlet.BaseDeployProcessor.deploy
    NewComposite(BaseDeployProcessor.java:365)
    oracle.integration.platform.blocks.deploy.servlet.BaseDeployProcessor.deploy
    SARs(BaseDeployProcessor.java:123)
    ^-- Holding lock: java.lang.Object@3621c97c[thin lock]
    oracle.integration.platform.blocks.deploy.servlet.DeployProcessor.doDeployWo
    rk(DeployProcessor.java:122)
    oracle.integration.platform.blocks.deploy.servlet.DeployProcessor.doDeployWo
    rk(DeployProcessor.java:107)
    oracle.integration.platform.blocks.deploy.servlet.DeployProcessor.doDeploy(D
    eployProcessor.java:96)
    oracle.integration.platform.blocks.deploy.servlet.DeployProcessor.process(De
    ployProcessor.java:68)
    oracle.integration.platform.blocks.deploy.servlet.CompositeDeployerServlet.d
    oPostInsideLoggingSession(CompositeDeployerServlet.java:141)
    oracle.integration.platform.blocks.deploy.servlet.CompositeDeployerServlet.d
    oPost(CompositeDeployerServlet.java:119)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:700)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:815)
    weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSe
    curityHelper.java:224)
    weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelpe
    r.java:108)
    weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:206)
    weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:111)
    oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:299)
    oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java
    :405)
    oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:94)
    oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:98)
    oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:70)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:111)
    oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:299)
    oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java
    :405)
    oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:94)
    oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:98)
    oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:70)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:86)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapR
    un(WebAppServletContext.java:3687)
    weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(W
    ebAppServletContext.java:3681)
    weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubjec
    t.java:308)
    weblogic.security.service.SecurityManager.runAs(SecurityManager.java:116)
    weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletC
    ontext.java:2213)
    weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.
    java:2135)
    weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:142
    0)
    weblogic.work.ExecuteThread.execute(ExecuteThread.java:203)
    weblogic.work.ExecuteThread.run(ExecuteThread.java:170)
    >

    I just noted that this is a database connectivity issue because when I point my soa related data sources (mds, soa_infra, etc...)to a local xe database, the deployment is doing fine.

  • ST3B14 : xdo  ReportService.getReportDefinition gives stuck thread

    We use the following code in List LOV source VVO to get the template names associated with the report:
    private static List<String> mTemplates = null;
    private static ReportService mReportService = null;
    private static int mCurrentRow = 0;
    private void initReportService() { 
    try { 
    mReportService = PPBIPUtil.getReportService();
    } catch (Throwable t) {
    t.printStackTrace();
    protected void executeQueryForCollection(Object qc, Object[] params,
    int noUserParams) {
    System.out.println("Nitesh : Entering method executeQueryForCollection");
    String reportPath =
    "/Financials/Receivables/BFBPrintSummary.xdo";
    try {
    ReportDefinition reportDefinition =
    mReportService.getReportDefinition(reportPath);
    System.out.println("Nitesh : Getting the templates");
    mTemplates = reportDefinition.getTemplateIds().getItem();
    System.out.println("Nitesh : mTemplates.size = "+ mTemplates.size());
    I have a local ST3B14 deployment pointing to the st3b14 debug db and I see that when the xdo is not uploaded to the BIP server, the parameterVO UI comes up fine, obviously the template LOV is blank as expected as it cannot find the report. However, when I upload the report the highlighted code :
    ReportDefinition reportDefinition =
    mReportService.getReportDefinition(reportPath);
    gives a stuck thread and execution hangs at this point. Here is what I see in the Adminserver logs:
    oracle.j2ee.ws.client.jaxws.WsClientProxyInvocationHandler.invoke(WsClientProxyInvocationHandler.java:142)
    $Proxy237.getReportDefinition(Unknown Source)
    oracle.xdo.service.client.ReportService.getReportDefinition(ReportService.java:307)
    oracle.apps.financials.receivables.transactions.balanceForwardBills.publicEssModel.view.BFBSummaryReportTemplateVVOImpl.executeQueryForCollection(BFBSummaryReportTemplateVVOImpl.java:55)
    oracle.jbo.server.ViewRowSetImpl.execute(ViewRowSetImpl.java:1036)
    ^-- Holding lock: oracle.jbo.JboSyncLock@75080cf[thin lock]
    oracle.jbo.server.ViewRowSetImpl.execute(ViewRowSetImpl.java:1021)
    oracle.jbo.server.ViewRowSetIteratorImpl.ensureRefreshed(ViewRowSetIteratorImpl.java:2738)
    ^-- Holding lock: oracle.jbo.JboSyncLock@75080cf[thin lock]
    oracle.jbo.server.ViewRowSetIteratorImpl.ensureRefreshed(ViewRowSetIteratorImpl.java:2730)
    oracle.jbo.server.ViewRowSetIteratorImpl.getAllRowsInRangeInternal(ViewRowSetIteratorImpl.java:2107)
    ^-- Holding lock: oracle.jbo.JboSyncLock@75080cf[thin lock]
    oracle.jbo.server.ViewRowSetIteratorImpl.getAllRowsInRange(ViewRowSetIteratorImpl.java:2165)
    Ess server logs
    oracle.jdbc.driver.T4CPreparedStatement.fetch(T4CPreparedStatement.java:995)
    oracle.jdbc.driver.OracleResultSetImpl.close_or_fetch_from_next(OracleResultSetImpl.java:244)
    ^-- Holding lock: oracle.jdbc.driver.T4CConnection@13be4a9[thin lock]
    oracle.jdbc.driver.OracleResultSetImpl.next(OracleResultSetImpl.java:187)
    ^-- Holding lock: oracle.jdbc.driver.T4CConnection@13be4a9[thin lock]
    weblogic.jdbc.wrapper.ResultSet_oracle_jdbc_driver_OracleResultSetImpl.next(Unknown Source)
    Has somebody encountered this? Also, where I can find the XDO source code jar/library that I can import in Jdev to debug further.
    thanks
    Nitesh
    Edited by: user595705 on Nov 2, 2010 3:57 PM

    Hi,
    Any updates on this?How did u solve it?I am encountering a similar issue on my junits.
    Thanks

Maybe you are looking for