Stuck Threads Leading to server hang
Hi ,
We have been facing a problem with Stuck Threads in our production environment. We are currently running version 8.1. Though the stuck Threads do point to JDBC issues, we suspect that these problems may be related to some Weblogic/OS configuration issue as well. These probelms eventually lead to a server hang.
The general pattern of the problem is that we see messages like :
Dec 24, 2006 8:56:09 AM EST> <Debug> <Common> <BEA-000610> < RP(rtqdbConnectionPool):timeoutInactiveRes (20)>
####<Dec 25, 2006 8:54:51 AM EST> <Error> <WebLogicServer> <hygeia> <hygeia-rtq-vm7> <weblogic.health.CoreHealthMonitor> <<WLS Kernel>> <> <BEA-000337> <UNKNOWN_SESSION has been busy for "896" seconds working on the request "Http Request: /quote.do", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.> ==================== This email/fax message is for the sole use of the intended
Sometimes a deadlock Exception would be thrown from the Database,
nested exception is java.sql.SQLException: ORA-00060: deadlock detected while waiting for resource
But this is not as often as we get the Stuck Thread messages.
This is a typical Thread dump from one of our servers when it complained of a stuck thread.
///*************Thread Dump Start**********************///
Full thread dump Java HotSpot(TM) Server VM (1.4.2_08-b03 mixed mode):
"idle thread" daemon prio=5 tid=0x00d2c150 nid=0x7f in Object.wait() [94e7f000..94e7fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xb2010000> (a org.xbill.Task.WorkerThread)
at org.xbill.Task.WorkerThread.run(WorkerThread.java:130)
- locked <0xb2010000> (a org.xbill.Task.WorkerThread)
"Thread-28" daemon prio=5 tid=0x00c6a7c0 nid=0x7d in Object.wait() [92c7f000..92c7fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xbce667f8> (a com.geico.util.LockManager)
at com.geico.util.LockManager.waitForWork(LockManager.java:85)
- locked <0xbce667f8> (a com.geico.util.LockManager)
at com.geico.mail.Mail.run(Mail.java:259)
at java.lang.Thread.run(Thread.java:534)
"CacheManager-idle" daemon prio=5 tid=0x00c7d950 nid=0x7c in Object.wait() [92d7f000..92d7fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xbce66878> (a com.geico.util.LockManager)
at com.geico.util.LockManager.waitForWork(LockManager.java:85)
- locked <0xbce66878> (a com.geico.util.LockManager)
at com.geico.cache.CacheManager.run(CacheManager.java:353)
at java.lang.Thread.run(Thread.java:534)
"ExecuteThread: '2' for queue: 'weblogic.kernel.Non-Blocking'" daemon prio=5 tid=0x006a6c90 nid=0x7a in Object.wait() [92e7f000..92e7fc28]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:429)
at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
- locked <0xbc2a5db8> (a weblogic.kernel.ExecuteThread)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '1' for queue: 'weblogic.kernel.Non-Blocking'" daemon prio=5 tid=0x0120c568 nid=0x79 in Object.wait() [92f7f000..92f7fc28]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:429)
at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
- locked <0xbc2a5e38> (a weblogic.kernel.ExecuteThread)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '0' for queue: 'weblogic.kernel.Non-Blocking'" daemon prio=5 tid=0x006a6af0 nid=0x78 in Object.wait() [9307f000..9307fc28]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:429)
at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
- locked <0xbc2a5eb8> (a weblogic.kernel.ExecuteThread)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '0' for queue: 'JMSStore<AptSoftFileStore-morpheus-rtq-vm7>.ioThreadPool'" daemon prio=5 tid=0x0052a198 nid=0x77 in Object.wait() [9317f000..9317fc28]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:429)
at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
- locked <0xbc2a5f38> (a weblogic.kernel.ExecuteThread)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"Thread-26" daemon prio=5 tid=0x00635d58 nid=0x75 in Object.wait() [9327f000..9327fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xbb97ad20> (a com.geico.util.LockManager)
at java.lang.Object.wait(Object.java:429)
at com.geico.util.LockManager.waitForWork(LockManager.java:63)
- locked <0xbb97ad20> (a com.geico.util.LockManager)
at com.geico.util.providers.StorageProvider.run(StorageProvider.java:158)
at java.lang.Thread.run(Thread.java:534)
"Thread-25" daemon prio=5 tid=0x00fac430 nid=0x72 waiting on condition [9337f000..9337fc28]
at java.lang.Thread.sleep(Native Method)
at org.apache.commons.pool.impl.GenericObjectPool$Evictor.run(GenericObjectPool.java:1080)
at java.lang.Thread.run(Thread.java:534)
"ListenThread.Default" prio=5 tid=0x014ec6e8 nid=0x71 runnable [9347f000..9347fc28]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
- locked <0xbb97ae60> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at java.net.ServerSocket.accept(ServerSocket.java:419)
at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:26)
at weblogic.t3.srvr.ListenThread.accept(ListenThread.java:735)
at weblogic.t3.srvr.ListenThread.run(ListenThread.java:301)
"Thread-24" daemon prio=5 tid=0x011e9668 nid=0x70 waiting on condition [9357f000..9357fc28]
at java.lang.Thread.sleep(Native Method)
at com.geico.log.HeartbeatLogger$1.run(HeartbeatLogger.java:133)
"Thread-22" daemon prio=5 tid=0x011e8610 nid=0x6f waiting on condition [9367f000..9367fc28]
at java.lang.Thread.sleep(Native Method)
at com.geico.log.StreamLogger.pause(StreamLogger.java:263)
at com.geico.log.StreamLogger.access$300(StreamLogger.java:15)
at com.geico.log.StreamLogger$WorkerThread.run(StreamLogger.java:217)
at java.lang.Thread.run(Thread.java:534)
"filewatcher" daemon prio=5 tid=0x0111ca50 nid=0x6e waiting on condition [9377f000..9377fc28]
at java.lang.Thread.sleep(Native Method)
at com.geico.watcher.FileWatcher.sleep(FileWatcher.java:135)
at com.geico.watcher.FileWatcher.run(FileWatcher.java:118)
at java.lang.Thread.run(Thread.java:534)
"Thread-18" prio=5 tid=0x00de2388 nid=0x6d in Object.wait() [9437e000..9437fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xa20f1a60> (a weblogic.jms.backend.BEConsumerReceiveRequest)
at java.lang.Object.wait(Object.java:429)
at weblogic.jms.dispatcher.Request.sleepTillNotified(Request.java:240)
at weblogic.jms.dispatcher.Request.wrappedFiniteStateMachine(Request.java:685)
- locked <0xa20f1a60> (a weblogic.jms.backend.BEConsumerReceiveRequest)
at weblogic.jms.dispatcher.DispatcherImpl.dispatchAsyncInternal(DispatcherImpl.java:132)
at weblogic.jms.dispatcher.DispatcherImpl.dispatchAsync(DispatcherImpl.java:123)
at weblogic.jms.dispatcher.Request.dispatchAsync(Request.java:878)
at weblogic.jms.frontend.FEConsumer.receive(FEConsumer.java:452)
at weblogic.jms.frontend.FEConsumer.invoke(FEConsumer.java:643)
at weblogic.jms.dispatcher.Request.wrappedFiniteStateMachine(Request.java:643)
at weblogic.jms.dispatcher.DispatcherImpl.dispatchSyncTran(DispatcherImpl.java:192)
at weblogic.jms.client.JMSSession.receiveMessage(JMSSession.java:456)
at weblogic.jms.client.JMSConsumer.receive(JMSConsumer.java:421)
at org.springframework.jms.core.JmsTemplate.doReceive(JmsTemplate.java:800)
at org.springframework.jms.core.JmsTemplate.doReceive(JmsTemplate.java:783)
at org.springframework.jms.core.JmsTemplate$8.doInJms(JmsTemplate.java:741)
at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:582)
at org.springframework.jms.core.JmsTemplate.receive(JmsTemplate.java:739)
at org.springframework.jms.core.JmsTemplate.receive(JmsTemplate.java:731)
at com.geico.uierrorlog.UiErrorUtil.getMessage(UiErrorUtil.java:89)
at com.geico.uierrorlog.UiErrorDaemon.getMessage(UiErrorDaemon.java:108)
at com.geico.uierrorlog.UiErrorDaemon.run(UiErrorDaemon.java:166)
at java.lang.Thread.run(Thread.java:534)
"StatisticsThread" daemon prio=5 tid=0x00d0c798 nid=0x6c in Object.wait() [9447f000..9447fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xb8ee7b18> (a com.geico.tangosol.stats.StatsDaemon)
at com.geico.tangosol.stats.StatsDaemon.run(StatsDaemon.java:144)
- locked <0xb8ee7b18> (a com.geico.tangosol.stats.StatsDaemon)
at com.tangosol.util.Daemon$1.run(Daemon.java:63)
"StatisticsThread" daemon prio=5 tid=0x01202c18 nid=0x6b in Object.wait() [9457f000..9457fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xb8ee7bd8> (a com.geico.tangosol.stats.StatsDaemon)
at com.geico.tangosol.stats.StatsDaemon.run(StatsDaemon.java:144)
- locked <0xb8ee7bd8> (a com.geico.tangosol.stats.StatsDaemon)
at com.tangosol.util.Daemon$1.run(Daemon.java:63)
"StatisticsThread" daemon prio=5 tid=0x00b33f18 nid=0x6a in Object.wait() [9467f000..9467fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xb8ee7c90> (a com.geico.tangosol.stats.StatsDaemon)
at com.geico.tangosol.stats.StatsDaemon.run(StatsDaemon.java:144)
- locked <0xb8ee7c90> (a com.geico.tangosol.stats.StatsDaemon)
at com.tangosol.util.Daemon$1.run(Daemon.java:63)
"DistributedCache:DistributedRTQOverflow" daemon prio=5 tid=0x009895e8 nid=0x69 in Object.wait() [9477f000..9477fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xb8ee7d40> (a com.tangosol.coherence.component.util.daemon.QueueProcessor$Queue)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
- locked <0xb8ee7d40> (a com.tangosol.coherence.component.util.daemon.QueueProcessor$Queue)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"DistributedCache:PoolStatsDistributedCache:EventDispatcher" daemon prio=5 tid=0x00dff060 nid=0x68 in Object.wait() [9487f000..9487fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xb8ee7f18> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
- locked <0xb8ee7f18> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"DistributedCache:PoolStatsDistributedCache" daemon prio=5 tid=0x006f3e98 nid=0x67 in Object.wait() [9497f000..9497fc28]
at java.lang.Object.wait(Native Method)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
- locked <0xb8ee7fe8> (a com.tangosol.coherence.component.util.daemon.QueueProcessor$Queue)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"StatisticsThread" daemon prio=5 tid=0x0090d8d0 nid=0x66 in Object.wait() [94a7f000..94a7fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xb8ee81c8> (a com.geico.tangosol.stats.StatsDaemon)
at com.geico.tangosol.stats.StatsDaemon.run(StatsDaemon.java:144)
- locked <0xb8ee81c8> (a com.geico.tangosol.stats.StatsDaemon)
at com.tangosol.util.Daemon$1.run(Daemon.java:63)
"DistributedCache:dist-RTQSession:EventDispatcher" daemon prio=5 tid=0x00a05110 nid=0x65 in Object.wait() [94b7f000..94b7fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xb8ee8278> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
- locked <0xb8ee8278> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"DistributedCache:dist-RTQSession" daemon prio=5 tid=0x0036e600 nid=0x64 in Object.wait() [94c7f000..94c7fc28]
at java.lang.Object.wait(Native Method)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
- locked <0xb8ee8348> (a com.tangosol.coherence.component.util.daemon.QueueProcessor$Queue)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"ReplicatedCache" daemon prio=5 tid=0x00e9f9f8 nid=0x63 in Object.wait() [94d7f000..94d7fc28]
at java.lang.Object.wait(Native Method)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
- locked <0xb8ee8520> (a com.tangosol.coherence.component.util.daemon.QueueProcessor$Queue)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"Scheduler_Worker-9" prio=5 tid=0x00339820 nid=0x61 in Object.wait() [94f7f000..94f7fc28]
at java.lang.Object.wait(Native Method)
at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:423)
- locked <0xb8ee8788> (a java.lang.Object)
at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:53)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:514)
"Scheduler_Worker-8" prio=5 tid=0x0033af58 nid=0x60 in Object.wait() [9507f000..9507fc28]
at java.lang.Object.wait(Native Method)
at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:423)
- locked <0xb8ee8788> (a java.lang.Object)
at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:53)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:514)
"Scheduler_Worker-7" prio=5 tid=0x0033a878 nid=0x5f in Object.wait() [9517f000..9517fc28]
at java.lang.Object.wait(Native Method)
at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:423)
- locked <0xb8ee8788> (a java.lang.Object)
at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:53)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:514)
"Scheduler_Worker-6" prio=5 tid=0x00935488 nid=0x5e in Object.wait() [9527f000..9527fc28]
at java.lang.Object.wait(Native Method)
at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:423)
- locked <0xb8ee8788> (a java.lang.Object)
at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:53)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:514)
"Scheduler_Worker-5" prio=5 tid=0x00bb2190 nid=0x5d in Object.wait() [9537f000..9537fc28]
at java.lang.Object.wait(Native Method)
at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:423)
- locked <0xb8ee8788> (a java.lang.Object)
at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:53)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:514)
"Scheduler_Worker-4" prio=5 tid=0x010b9038 nid=0x5c in Object.wait() [9547f000..9547fc28]
at java.lang.Object.wait(Native Method)
at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:423)
- locked <0xb8ee8788> (a java.lang.Object)
at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:53)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:514)
"Scheduler_Worker-3" prio=5 tid=0x00a34678 nid=0x5b in Object.wait() [9557f000..9557fc28]
at java.lang.Object.wait(Native Method)
at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:423)
- locked <0xb8ee8788> (a java.lang.Object)
at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:53)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:514)
"Scheduler_Worker-2" prio=5 tid=0x014f9860 nid=0x5a in Object.wait() [9567f000..9567fc28]
at java.lang.Object.wait(Native Method)
at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:423)
- locked <0xb8ee8788> (a java.lang.Object)
at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:53)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:514)
"Scheduler_Worker-1" prio=5 tid=0x0044e530 nid=0x59 in Object.wait() [9577f000..9577fc28]
at java.lang.Object.wait(Native Method)
at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:423)
- locked <0xb8ee8788> (a java.lang.Object)
at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:53)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:514)
"Scheduler_Worker-0" prio=5 tid=0x0110b8e8 nid=0x58 in Object.wait() [9587f000..9587fc28]
at java.lang.Object.wait(Native Method)
at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:423)
- locked <0xb8ee8788> (a java.lang.Object)
at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:53)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:514)
"StatisticsThread" daemon prio=5 tid=0x011352b8 nid=0x57 in Object.wait() [9597f000..9597fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xb8ee8b20> (a com.geico.tangosol.stats.StatsDaemon)
at com.geico.tangosol.stats.StatsDaemon.run(StatsDaemon.java:144)
- locked <0xb8ee8b20> (a com.geico.tangosol.stats.StatsDaemon)
at com.tangosol.util.Daemon$1.run(Daemon.java:63)
"DistributedCache:dist-rtq-sessionheader:EventDispatcher" daemon prio=5 tid=0x004222a0 nid=0x56 in Object.wait() [95a7f000..95a7fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xb8ee8bd8> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
- locked <0xb8ee8bd8> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"DistributedCache:dist-rtq-sessionheader" daemon prio=5 tid=0x01278fa0 nid=0x55 in Object.wait() [95b7f000..95b7fc28]
at java.lang.Object.wait(Native Method)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
- locked <0xb8ee8ca8> (a com.tangosol.coherence.component.util.daemon.QueueProcessor$Queue)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"DistributedCache:dist-hibernate-default:EventDispatcher" daemon prio=5 tid=0x015273f0 nid=0x54 in Object.wait() [95c7f000..95c7fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xb8ee8e80> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
- locked <0xb8ee8e80> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"DistributedCache:dist-hibernate-default" daemon prio=5 tid=0x00ea1328 nid=0x53 in Object.wait() [9607f000..9607fc28]
at java.lang.Object.wait(Native Method)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
- locked <0xb8ee8f50> (a com.tangosol.coherence.component.util.daemon.QueueProcessor$Queue)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"Invocation:Management" daemon prio=5 tid=0x003d8968 nid=0x52 in Object.wait() [95d7f000..95d7fc28]
at java.lang.Object.wait(Native Method)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
- locked <0xb8448698> (a com.tangosol.coherence.component.util.daemon.QueueProcessor$Queue)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"Invocation:Management:EventDispatcher" daemon prio=5 tid=0x007e6040 nid=0x51 in Object.wait() [95e7f000..95e7fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xb84487f0> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
- locked <0xb84487f0> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"TcpRingListener" daemon prio=6 tid=0x007e6f40 nid=0x50 runnable [95f7f000..95f7fc28]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
- locked <0xb84488d0> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at java.net.ServerSocket.accept(ServerSocket.java:419)
at com.tangosol.coherence.component.net.socket.TcpSocketAccepter.accept(TcpSocketAccepter.CDB:17)
at com.tangosol.coherence.component.util.daemon.TcpRingListener.acceptConnection(TcpRingListener.CDB:9)
at com.tangosol.coherence.component.util.daemon.TcpRingListener.onNotify(TcpRingListener.CDB:1)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:34)
at java.lang.Thread.run(Thread.java:534)
"Cluster" daemon prio=5 tid=0x00c94738 nid=0x4e in Object.wait() [9617f000..9617fc28]
at java.lang.Object.wait(Native Method)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
- locked <0xb8448ac0> (a com.tangosol.coherence.component.net.Cluster$ClusterService$Queue)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"PacketPublisher" daemon prio=6 tid=0x00d83418 nid=0x4d runnable [9627f000..9627fc28]
at java.lang.Object.wait(Native Method)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
- locked <0xb8448c78> (a com.tangosol.coherence.component.net.Cluster$PacketPublisher$Queue)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"PacketReceiver" daemon prio=7 tid=0x00b3cd70 nid=0x4c in Object.wait() [9637f000..9637fc28]
at java.lang.Object.wait(Native Method)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
- locked <0xb8448dc0> (a com.tangosol.coherence.component.net.Cluster$PacketReceiver$Queue)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"PacketListenerN" daemon prio=8 tid=0x003d7cd0 nid=0x4b runnable [9647f000..9647fc28]
at java.net.PlainDatagramSocketImpl.receive(Native Method)
at java.net.DatagramSocket.receive(DatagramSocket.java:711)
- locked <0xb84494c8> (a java.net.DatagramPacket)
- locked <0xb84494e8> (a java.net.MulticastSocket)
at com.tangosol.coherence.component.net.socket.UdpSocket.receive(UdpSocket.CDB:18)
at com.tangosol.coherence.component.net.UdpPacket.receive(UdpPacket.CDB:17)
at com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onWait(PacketListener.CDB:5)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"PacketListener1" daemon prio=8 tid=0x00df5538 nid=0x4a runnable [9657f000..9657fc28]
at java.net.PlainDatagramSocketImpl.receive(Native Method)
at java.net.DatagramSocket.receive(DatagramSocket.java:711)
- locked <0xb8449cb8> (a java.net.DatagramPacket)
- locked <0xb8449cd8> (a java.net.DatagramSocket)
at com.tangosol.coherence.component.net.socket.UdpSocket.receive(UdpSocket.CDB:18)
at com.tangosol.coherence.component.net.UdpPacket.receive(UdpPacket.CDB:17)
at com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onWait(PacketListener.CDB:5)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"Logger@13785534 3.1.2/343" daemon prio=3 tid=0x006b7358 nid=0x49 in Object.wait() [9667f000..9667fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xb8449e90> (a com.tangosol.coherence.component.application.console.Coherence$Logger$Queue)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
- locked <0xb8449e90> (a com.tangosol.coherence.component.application.console.Coherence$Logger$Queue)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"ExecuteThread: '5' for queue: 'JMS.TimerClientPool'" daemon prio=5 tid=0x011384b0 nid=0x48 in Object.wait() [9677f000..9677fc28]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:429)
at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
- locked <0xb82c91f8> (a weblogic.kernel.ExecuteThread)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '4' for queue: 'JMS.TimerClientPool'" daemon prio=5 tid=0x01137bc8 nid=0x47 in Object.wait() [9687f000..9687fc28]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:429)
at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
- locked <0xb82c9278> (a weblogic.kernel.ExecuteThread)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '3' for queue: 'JMS.TimerClientPool'" daemon prio=5 tid=0x0111d4e0 nid=0x46 in Object.wait() [9697f000..9697fc28]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:429)
at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
- locked <0xb82c92f8> (a weblogic.kernel.ExecuteThread)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '2' for queue: 'JMS.TimerClientPool'" daemon prio=5 tid=0x010dd218 nid=0x45 in Object.wait() [96a7f000..96a7fc28]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:429)
at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
- locked <0xb82c9378> (a weblogic.kernel.ExecuteThread)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '1' for queue: 'JMS.TimerClientPool'" daemon prio=5 tid=0x00bb1c98 nid=0x44 in Object.wait() [96b7f000..96b7fc28]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:429)
at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
- locked <0xb82c93f8> (a weblogic.kernel.ExecuteThread)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '0' for queue: 'JMS.TimerClientPool'" daemon prio=5 tid=0x00979ac8 nid=0x43 in Object.wait() [96c7f000..96c7fc28]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:429)
at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
- locked <0xb82c9478> (a weblogic.kernel.ExecuteThread)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"Thread-12" daemon prio=5 tid=0x00200610 nid=0x42 waiting on condition [96d7f000..96d7fc28]
at java.lang.Thread.sleep(Native Method)
at com.geico.log.HeartbeatLogger$1.run(HeartbeatLogger.java:133)
"Thread-10" daemon prio=5 tid=0x00200470 nid=0x41 waiting on condition [96e7f000..96e7fc28]
at java.lang.Thread.sleep(Native Method)
at com.geico.log.StreamLogger.pause(StreamLogger.java:267)
at com.geico.log.StreamLogger.access$300(StreamLogger.java:19)
at com.geico.log.StreamLogger$WorkerThread.run(StreamLogger.java:221)
at java.lang.Thread.run(Thread.java:534)
"ClockDaemonThread-0" daemon prio=5 tid=0x00ab6a48 nid=0x40 in Object.wait() [96f7f000..96f7fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xb8051b20> (a com.geico.util.concurrent.ClockDaemon)
at com.geico.util.concurrent.ClockDaemon.nextTask(ClockDaemon.java:328)
- locked <0xb8051b20> (a com.geico.util.concurrent.ClockDaemon)
at com.geico.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:362)
at java.lang.Thread.run(Thread.java:534)
"Thread-9" prio=5 tid=0x008912e0 nid=0x3f in Object.wait() [9ad7f000..9ad7fc28]
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:429)
- locked <0xb8051bc0> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:382)
"ExecuteThread: '0' for queue: 'JMS.TimerTreePool'" daemon prio=5 tid=0x001d7fa8 nid=0x3e in Object.wait() [9ae7f000..9ae7fc28]
at java.lang.Object.wait(Native Method)
at weblogic.jms.backend.BETimerTree.execute(BETimerTree.java:146)
- locked <0xb8051c90> (a weblogic.jms.backend.BETimerTree)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)
"Thread-8" prio=5 tid=0x00247418 nid=0x3d in Object.wait() [9af7f000..9af7fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xb8051d20> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:429)
- locked <0xb8051d20> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:382)
"Thread-7" prio=5 tid=0x001d8a28 nid=0x3c in Object.wait() [9b07f000..9b07fc28]
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:429)
- locked <0xb8051dc0> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:382)
"ExecuteThread: '0' for queue: 'weblogic.cluster.MulticastManager'" daemon prio=5 tid=0x00da0a60 nid=0x3b runnable [9b17f000..9b17fc28]
at java.net.PlainDatagramSocketImpl.receive(Native Method)
at java.net.DatagramSocket.receive(DatagramSocket.java:711)
- locked <0xa22dc3e8> (a java.net.DatagramPacket)
- locked <0xb7558130> (a java.net.MulticastSocket)
at weblogic.cluster.FragmentSocket.receive(FragmentSocket.java:169)
at weblogic.cluster.MulticastManager.execute(MulticastManager.java:395)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)
"SSLListenThread.Administrator" prio=5 tid=0x00c7db08 nid=0x3a runnable [9b27f000..9b27fc28]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
- locked <0xb75582a0> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at javax.net.ssl.impl.SSLServerSocketImp
Hi,
I guess this issue is related to your connectivity and hyperion services. Can you take a restart and try to execute the reports. Also try to check all your Hyperion services are up and running before executing the FR Reports. You can also check the Reporting services by login to Weblogic server (if you have applied it in your environment). In weblogic it should be in Running mode and not in Admin mode. If it is not in Running mode then try to check the Data Source setting from Connection Pool in Weblogic Server.
Soumya
Similar Messages
-
How does WebLogic clean up stuck threads?
Hi,
Using WebLogic 9.2.2 on Linux OS. In our logs, we are finding some stuck threads and it seems to be due to something WebLogic is doing. While we don't understand the cause (maybe the stack trace helps), does WebLogic do anything to remove stuck threads from its pool, or is there anything we can do to configure it to?
<Jul 21, 2009 2:50:21 PM UTC> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "605" seconds working on the request "Http Request: /RoutingEngine/pcFlow.do", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-69 "[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, priority=1, DAEMON> {
jrockit.net.SocketNativeIO.readBytesPinned(Unknown Source)
jrockit.net.SocketNativeIO.socketRead(Unknown Source)
java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)
java.net.SocketInputStream.read(SocketInputStream.java:107)
java.io.BufferedInputStream.fill(BufferedInputStream.java:189)
java.io.BufferedInputStream.read(BufferedInputStream.java:234)
^-- Holding lock: java.io.BufferedInputStream@3537712[thin lock]
weblogic.net.http.MessageHeader.isHTTP(MessageHeader.java:214)
weblogic.net.http.MessageHeader.parseHeader(MessageHeader.java:141)
weblogic.net.http.HttpClient.parseHTTP(HttpClient.java:473)
^-- Holding lock: java.io.BufferedInputStream@3537712[thin lock]
weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:332)
^-- Holding lock: java.io.BufferedInputStream@3537712[thin lock]
weblogic.net.http.SOAPHttpURLConnection.getInputStream(SOAPHttpURLConnection.java:36)
weblogic.wsee.util.dom.DOMParser.getWebLogicDocumentImpl(DOMParser.java:92)
weblogic.wsee.util.dom.DOMParser.getDocument(DOMParser.java:56)
weblogic.wsee.wsdl.WsdlReader.getDocument(WsdlReader.java:311)
weblogic.wsee.wsdl.WsdlReader.getDocument(WsdlReader.java:305)
weblogic.wsee.wsdl.WsdlSchema.parse(WsdlSchema.java:120)
weblogic.wsee.wsdl.WsdlSchemaImport.parse(WsdlSchemaImport.java:55)
weblogic.wsee.wsdl.WsdlSchema.parse(WsdlSchema.java:82)
weblogic.wsee.wsdl.WsdlSchema.parse(WsdlSchema.java:62)
weblogic.wsee.wsdl.WsdlTypes.parse(WsdlTypes.java:145)
weblogic.wsee.wsdl.WsdlDefinitions.parseChild(WsdlDefinitions.java:452)
weblogic.wsee.wsdl.WsdlExtensible.parse(WsdlExtensible.java:84)
weblogic.wsee.wsdl.WsdlDefinitions.parse(WsdlDefinitions.java:378)
weblogic.wsee.wsdl.WsdlDefinitions.parse(WsdlDefinitions.java:358)
weblogic.wsee.wsdl.WsdlFactory.parse(WsdlFactory.java:50)
weblogic.wsee.jaxrpc.ServiceImpl.loadWsdlDefinition(ServiceImpl.java:437)
weblogic.wsee.jaxrpc.ServiceImpl.<init>(ServiceImpl.java:95)
com.myco.regui.provisioning.client.ProvisioningAccountService_Impl.<init>(ProvisioningAccountService_Impl.java:21)
com.myco.regui.provisioning.client.ProvisioningAccountService_Impl.<init>(ProvisioningAccountService_Impl.java:17)
com.myco.regui.struts.accounts.AccountsForm.contactWPS(AccountsForm.java:208)
com.myco.regui.struts.accounts.AccountsForm.validateAccountid(AccountsForm.java:161)
com.myco.regui.struts.accounts.AccountsForm.validatePCAccountId(AccountsForm.java:144)
com.myco.regui.struts.accounts.AccountsForm._validatePCFields(AccountsForm.java:112)
com.myco.regui.struts.accounts.AccountsForm.validate(AccountsForm.java:83)
org.apache.struts.chain.commands.servlet.ValidateActionForm.validate(ValidateActionForm.java:57)
org.apache.struts.chain.commands.AbstractValidateActionForm.execute(AbstractValidateActionForm.java:94)
org.apache.struts.chain.commands.ActionCommandBase.execute(ActionCommandBase.java:48)
org.apache.commons.chain.impl.ChainBase.execute(ChainBase.java:176)
org.apache.commons.chain.generic.LookupCommand.execute(LookupCommand.java:303)
org.apache.commons.chain.impl.ChainBase.execute(ChainBase.java:176)
org.apache.struts.chain.ComposableRequestProcessor.process(ComposableRequestProcessor.java:272)
org.apache.struts.action.ActionServlet.process(ActionServlet.java:1903)
org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:462)
javax.servlet.http.HttpServlet.service(HttpServlet.java:736)
javax.servlet.http.HttpServlet.service(HttpServlet.java:851)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:198)
weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:41)
com.myco.regui.servlets.filters.TransactionFilter.doFilter(TransactionFilter.java:23)
.......You can configure these actions to occur when not all threads are stuck, but the number of stuck threads have exceeded a configured threshold:
* Shut down the Work Manager if it has stuck threads. A Work Manager that is shut down will refuse new work and reject existing work in
the queue by sending a rejection message. In a cluster, clustered clients will fail over to another cluster member.
* Shut down the application if there are stuck threads in the application. The application is shutdown by bringing it into admin mode. All
Work Managers belonging to the application are shut down, and behave as described above.
* Mark the server instance as failed and shut it down it down if there are stuck threads in the server. In a cluster, clustered clients that
are connected or attempting to connect will fail over to another cluster member.
Link : [http://e-docs.bea.com/wls/docs103/config_wls/overload.html] -
WorkManager and ignore-stuck-thread setting
In WL 10.3, using an application scoped work manager with the following .war deployment descriptors:
web.xml:
<resource-ref>
<res-ref-name>wm/WorkManagerABC</res-ref-name>
<res-type>commonj.work.WorkManager</res-type>
<res-auth>Container</res-auth>
<res-sharing-scope>Shareable</res-sharing-scope>
</resource-ref>
weblogic.xml:
<work-manager>
<name>WorkManagerABC</name>
<fair-share-request-class>
<name>abcClass</name>
<fair-share>1000</fair-share>
</fair-share-request-class>
<ignore-stuck-threads>true</ignore-stuck-threads>
<min-threads-constraint>
<name>MinThreadsCountTen</name>
<count>10</count>
</min-threads-constraint>
<max-threads-constraint>
<name>MaxThreadsCountTwenty</name>
<count>20</count>
</max-threads-constraint>
</work-manager>
<wl-dispatch-policy>WorkManagerABC</wl-dispatch-policy>
( This particular app is meant to be a socket listener using javax.nio ). This results in WL seeing "stuck threads" after the default 600 seconds. In our clusters, our clusters are configured that 3 stuck threads cause a server restart by nm.
So I created the same objects at the domain level ( workmanager, min/max/fair-share ), with the workmanager set to ignore stuck threads.
web.xml
<resource-ref>
<res-ref-name>WorkManagerABC</res-ref-name>
<res-type>commonj.work.WorkManager</res-type>
<res-auth>Container</res-auth>
<res-sharing-scope>Shareable</res-sharing-scope>
</resource-ref>
weblogic.xml
<weblogic-version>10.3</weblogic-version>
<!-- no work manager defined -->
<wl-dispatch-policy>WorkManagerEni</wl-dispatch-policy>
The app deploys and simple web pages are processed within the WorkManagerABC wm. Although WL still marks 4 threads as hoggers, none of them are stuck. So no restart, and the server Health stays OK.
So a workManager at the server level obeys the "Ignore Stuck Threads" while an application scoped workmanager does not.I wasn't too disappointed that the application-scoped workmanager didn't obey the stuck threads; I'd rather have the work manager configurable at the server console level without having to involve customized deployment plans to override hard-coded thread restraint levels in the app's deployment descriptors.
I think having it defined within the server console will make it simpler to manage the different performances in production versus qa, etc., than having it defined within the app and then adjusting it in each environment. -
Where we can know about the stuck threads without taking thread dump.
Hi Everyone,
Where we can View/know about the stuck threads without taking thread dump.
Thank YouHi!
WebLogic Server diagnoses a thread as stuck if it is continually working (not idle) for a set period of time. You can tune stuck threads in the Weblogic Server.
You can tune a server’s thread detection behavior by changing the length of time before a thread is diagnosed as stuck, and by changing the frequency with which the server checks for stuck threads.
It can be configured using below mentioned 2 parameters. These can be configured from Admin console -> Servers ->Configuration > Tuning tab.
1)Stuck Thread Max Time
2)Stuck Thread Timer Interval
Although you can change the criteria WebLogic Server uses to determine whether a thread is stuck, you cannot change the default behavior of setting the “warning” and “critical” health states when all threads in a particular execute queue become stuck.
There is a parameter "Stuck Thread Count" which can be configured from Console
(Servers -> Configuration ->Overload -> Stuck Thread count) which helps to transition the Server to FAILED state once the stuck threads reaches the value.
You can also use "OverloadProtectionMBean" for tuning. In Weblogic Server 9.x and later, it is recommended that you use the "ServerFailureTriggerMBean" in the "OverloadProtectionMBean".The ServerFailureTriggerMBean transitions the server to a FAILED state after the specified number of stuck threads are detected.The OverloadProtectionMBean has options to suspend or shutdown a failed server.
WebLogic Server checks for stuck threads periodically. If all application threads are stuck, a server instance marks itself failed, if configured to do so, exits. You can configure Node Manager or a third-party high-availability solution to restart the server instance for automatic failure recovery.
You can configure these actions to occur when not all threads are stuck, but the number of stuck threads have exceeded a configured threshold:
- Shut down the Work Manager if it has stuck threads. A Work Manager that is shut down will refuse new work and reject existing work in the queue by sending a rejection message. In a cluster, clustered clients will fail over to another cluster member.
- Shut down the application if there are stuck threads in the application. The application is shutdown by bringing it into ADMIN mode. All Work Managers belonging to the application are shut down, and behave as described above.
- Mark the server instance as failed and shut it down it down if there are stuck threads in the server. In a cluster, clustered clients that are connected or attempting to connect will fail over to another cluster member.
You can configure the "ServerFailureTriggerMBean" in the "OverloadProtectionMBean".
Below is documentation link for "ServerFailureTriggerMBean" methods.
http://download.oracle.com/docs/cd/E12839_01/apirefs.1111/e13945/weblogic/management/configuration/ServerFailureTriggerMBean.html
You can use getStuckThreadCount() method to check the number of stuck threads and transition the server to Failed State once itreaches the limit.
getStuckThreadCount:
int getStuckThreadCount() - The number of stuck threads after which the server is transitioned into FAILED state. There are options inOverloadProtectionMBean to suspend and shutdown a FAILED server. By default, the server continues to run in FAILED state. If the StuckThreadCount value is set to zero then the server never transitions into FAILED server irrespective of the number of stuck threads.
Returns:
The StuckThreadCount value
Default Value:
0
Maximum Value:
java.lang.Integer.MAX_VALUE
Minimum Value:
0
Below is documentation link for "OverloadProtectionMBean" methods.
http://download.oracle.com/docs/cd/E11035_01/wls100/javadocs_mhome/weblogic/management/configuration/OverloadProtectionMBean.html
In the Admin console, you can set the "FailureAction" under Servers->Configuration->Overload to force shutdown the managed server once the server is in Failed state.
The OverloadProtectionMBean has a method getFailureAction to achieve the same.
getFailureAction:
String getFailureAction() - Enable automatic forceshutdown of the server on failed state. The server self-health monitoring detects fatal failures and mark the server as failed. The server can be restarted using NodeManager or a HA agent.
Valid Values:
OverloadProtectionMBean.NO_ACTION, OverloadProtectionMBean.FORCE_SHUTDOWN, OverloadProtectionMBean.ADMIN_STATE
If you start the managed servers using node manager, you can enable "Auto Kill if Failed" and "Auto Restart" in the Admin console, under Servers-> configuration->Health Monitoring. Node Manager will take care of restarting the managed server if you enable "Auto Restart".
You can also configure the "Stuck Thread Count" and "Failure Action" to "Force Immediate shutdown of the Server" from Admin console under servers-> configuration-> Overload. This will help you to shutdown the server when the stuck thread count is reached. But there is no way to release the threads once they are stuck from the configuration.
Suppose if you set the value of the Stuck Thread Count to 20, The server will be transitioned to failed state once the count reaches 20 and if you enable the Failure Action, the server self-health monitoring detects fatal failures and mark the server as failed.
Depending on how you are starting the servers (custom scripts or Node Manager or startup scripts,...), you can restart the servers.
For more details on this please refer to below link:
http://www.oracle.com/technetwork/articles/entarch/workload-management-088692.html
Hope this helps.
Thanks,
Cris -
Thread are gettin stuck causing the WebLogic server to hang
We are developing an application for an US client which inclueds the following technology.
- WebLogic 9.2 JMS (Foreign Server, Queue, Connection Factor, persistent Store)
- Oracle 10g & Oracle 9i
- J2EE (MDB)
Different application uses different databases.
After the server run for a few day it is getting a stuck thread and reported the same as a Hogging Thread in server -> monitoring -> thread. At this point the server hangs and stopped working with the status in 'warning' state (in the same page itself).
The problem remains until the managed server is restarted.
We have taken thread dump of different times and one of the error that we have got is attached below:
"[STUCK] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.rjvm.ResponseImpl@1279315 WAITING
java.lang.Object.wait(Native Method)
weblogic.rjvm.ResponseImpl.waitForData(ResponseImpl.java:82)
weblogic.rjvm.ResponseImpl.getTxContext(ResponseImpl.java:109)
weblogic.messaging.dispatcher.DispatcherProxy.unmarshalResponse(DispatcherProxy.java:238)
weblogic.messaging.dispatcher.DispatcherProxy.dispatchSyncTranFuture(DispatcherProxy.java:131)
weblogic.messaging.dispatcher.DispatcherWrapperState.dispatchSyncTran(DispatcherWrapperState.java:241)
weblogic.messaging.dispatcher.DispatcherWrapperState.dispatchSyncNoTran(DispatcherWrapperState.java:266)
weblogic.jms.dispatcher.DispatcherAdapter.dispatchSyncNoTran(DispatcherAdapter.java:59)
weblogic.jms.client.JMSProducer.toFEProducer(JMSProducer.java:1061)
weblogic.jms.client.JMSProducer.deliveryInternal(JMSProducer.java:670)
weblogic.jms.client.JMSProducer.sendInternal(JMSProducer.java:455)
weblogic.jms.client.JMSProducer.send(JMSProducer.java:337)
weblogic.jms.client.WLProducerImpl.send(WLProducerImpl.java:966)
com.ctb.dex.messagehost.InputMessageListener.onMessage(Unknown Source)
weblogic.ejb.container.internal.MDListener.execute(MDListener.java:429)
weblogic.ejb.container.internal.MDListener.transactionalOnMessage(MDListener.java:335)
weblogic.ejb.container.internal.MDListener.onMessage(MDListener.java:291)
weblogic.jms.client.JMSSession.onMessage(JMSSession.java:4072)
weblogic.jms.client.JMSSession.execute(JMSSession.java:3962)
weblogic.jms.client.JMSSession$UseForRunnable.run(JMSSession.java:4490)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
Any body has any idea why this is happening? Please let me know in case you need to know more about the problem. Mail me at [email protected].Tom, we are experiencing the similar stuck thread max time out errors on our weblogic server(weblogic 8.1 sp5). Below is the thread dump:
Blocked lock chains
===================
Chain 4:
"ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" id=101 idx=0xca tid=19298 waiting for java/lang/String@0x9c1fb10 held by:
"ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" id=102 idx=0xcc tid=19299 in chain 3
Open lock chains
================
Chain 1:
"ExecuteThread: '33' for queue: 'weblogic.kernel.Default'" id=62 idx=0x80 tid=19223 waiting for java/lang/String@0xbd00b10 held by:
"ExecuteThread: '3' for queue: 'weblogic.kernel.Default'" id=32 idx=0x44 tid=19192 (waiting on notification)
Chain 2:
"ExecuteThread: '45' for queue: 'weblogic.kernel.Default'" id=74 idx=0x98 tid=19235 waiting for java/lang/String@0x1d1083a8 held by:
"ExecuteThread: '11' for queue: 'weblogic.kernel.Default'" id=40 idx=0x54 tid=19200 (waiting on notification)
Chain 3:
"ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" id=100 idx=0xc8 tid=19297 waiting for java/lang/String@0x9c1fb10 held by:
"ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" id=102 idx=0xcc tid=19299 (active)
Can you help us in resolving the issue? This causes the weblogic server to hung and eventually non responsive.
Below are our java version details:
java version "1.4.2_13"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_13-b06)
Java HotSpot(TM) Client VM (build 1.4.2_13-b06, mixed mode)
Can you let me know how to proceed and what can be done to avoid this issue?
Thanks a bunch Tom.
Regards
RV -
Page stuck while load and leads to stuck thread
Hi
I am facing an issue from a long time.
When i run the page of my application, when it opens for first time it get stuck and get loading and take very much time and after that it leads to stuck thread issue. All the component of my page opens but there is a table which gets value from OIM APIs also loads for the fiirst time. As in coding i have done it will pass null value to the table and wont fetch data for the first time. it will only fetch when value will be passed in a text box and submit was hit.
So my page gets hanged or loading in the table section everytime when it loads. for that i have to login in weblogic server, run the workmanager, forcefully shut down the server instance and run the page again. But sometime that doesnt work too.
Please help its a bug trouble for me and creating a huge problem.
Thanks
SudeepI'm not sure I understand your issue - but if you want to delay the fetching of information into the table until you actually have a value to pass to it see:
http://blogs.oracle.com/shay/2010/08/delay_method_execution_when_us.html -
12.1.3 Servicebus deployment hangs stuck threads
Hi,
I'm not sure what the issue is but on a new standalone SOA environment, most service bus deployments keep hanging.
(Deployments using sbconfig.jar import).
Not sure what the issue is - I see stuck threads on the AdminServer and the osb managed server and the /servicebus console just hangs forever.
All the dependencies (mainly connection pool jndi names etc.) exist on the target server.
Occasionally this environment also gives me the error shown in the screenshot "Exception occurred while downloading files"
When I import the same sbconfig.jar on a different server (installed locally on my desktop), the deployment goes through perfectly fine.Hi,
Can You try to delete or rename the data folder present in the Domain_home/Server/AdminServer and then try to restart the server.
Admin Server will create the ldap files again and then you can try to use the same password and check if it works.
You can rename the data .tmp and cache folder
Else You can also create a New Admin user and use them to start the server.
Find the link with more information for the same
Recover WebLogic admin password - weblogicexpert
Hope it helps -
Stuck thread in Oracle Weblogic server
Dear All,
We have a stuck thread in Oracle WLS 10.3.3. We have deployed 7 ear files in 5th managed server(psMS5). We have stuck thread in one of the application (psinsurance). We have allocated 1GB heap space for the psMS5 managed server. We are using this in Development environment. Please check the below message from psMS5 managed server log file. Kindly provide reason for stuck thread and solution.
WatchTime: Sep 23, 2011 10:28:29 AM IST
WatchDomainName: dev_domain
WatchServerName: psMS5
WatchSeverityLevel: Warning
WatchName: MS5-Watch
WatchRuleType: Log
WatchRule: ((MESSAGE LIKE '%has been busy for "%') AND (MSGID = 'BEA-000337') AND (SERVER = 'psMS5'))
WatchData: DATE = Sep 23, 2011 10:28:28 AM IST SERVER = psMS5 MESSAGE = [STUCK] ExecuteThread: '22' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "629" seconds working on the request "weblogic.servlet.internal.ServletRequestImpl@1cac6525[
POST */psinsurance/app/lending/insurance/PS00738D.jspx*?_adf.ctrl-state=24f0li74e_4 HTTP/1.1
Accept: image/jpeg, application/x-ms-application, image/gif, application/xaml+xml, image/pjpeg, application/x-ms-xbap, */*
Referer: http://sis-prosun11/psinsurance/faces/PS00003?jsessionid=r2BrT8TpqZBsnLhCw2Y8KDpTV7hMpgnLDFW54P87ZpHwn8JMy88s%21168932459%211316753022074&_afrWindowMode=0&_afrLoop=244724092027000&_adf.ctrl-state=24f0li74e_4
Accept-Language: en-US
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; Tablet PC 2.0)
Content-Type: application/x-www-form-urlencoded
Accept-Encoding: gzip, deflate
Content-Length: 554
Cache-Control: no-cache
Cookie: JSESSIONID=LkJcT8TfMT64SyQJT4Ls4sXhfDl0pTS7CXJVQ9KcryHGWJ1dhyfz!1373322659
Connection: Keep-Alive
Proxy-Client-IP: 192.168.9.179
X-Forwarded-For: 192.168.9.179
X-WebLogic-KeepAliveSecs: 30
X-WebLogic-Force-JVMID: 1373322659
]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-237 "[STUCK] ExecuteThread: '22' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, suspended, priority=1, DAEMON> {
java.io.UnixFileSystem.getBooleanAttributes0(UnixFileSystem.java:???)
java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:228)
java.io.File.exists(File.java:729)
weblogic.utils.classloaders.ClasspathClassFinder.getFileSource(ClasspathClassFinder.java:383)
weblogic.utils.classloaders.ClasspathClassFinder.getSourcesInternal(ClasspathClassFinder.java:333)
weblogic.utils.classloaders.ClasspathClassFinder.getSource(ClasspathClassFinder.java:316)
weblogic.utils.classloaders.JarClassFinder.getSource(JarClassFinder.java:58)
weblogic.utils.classloaders.AbstractClassFinder.getClassSource(AbstractClassFinder.java:22)
weblogic.utils.classloaders.MultiClassFinder.getClassSource(MultiClassFinder.java:56)
weblogic.utils.classloaders.CodeGenClassFinder.getClassSource(CodeGenClassFinder.java:26)
weblogic.utils.classloaders.GenericClassLoader.findLocalClass(GenericClassLoader.java:268)
*^-- Holding lock: weblogic.utils.classloaders.GenericClassLoader@1e3295a1[thin lock]*
weblogic.utils.classloaders.GenericClassLoader.findClass(GenericClassLoader.java:237)
java.lang.ClassLoader.loadClass(ClassLoader.java:292)
*^-- Holding lock: weblogic.utils.classloaders.GenericClassLoader@1e3295a1[thin lock]*
java.lang.ClassLoader.loadClass(ClassLoader.java:292)
java.lang.ClassLoader.loadClass(ClassLoader.java:248)
weblogic.utils.classloaders.GenericClassLoader.loadClass(GenericClassLoader.java:177)
weblogic.utils.classloaders.ChangeAwareClassLoader.loadClass(ChangeAwareClassLoader.java:37)
*^-- Holding lock: weblogic.utils.classloaders.ChangeAwareClassLoader@1e553474[thin lock]*
java.lang.Class.getDeclaredConstructors0(Class.java:???)
java.lang.Class.privateGetDeclaredConstructors(Class.java:2370)
java.lang.Class.getConstructor0(Class.java:2699)
java.lang.Class.newInstance0(Class.java:318)
java.lang.Class.newInstance(Class.java:305)
org.apache.myfaces.trinidadinternal.renderkit.ClassRendererInstantiator.instantiate(ClassRendererInstantiator.java:48)
org.apache.myfaces.trinidadinternal.renderkit.RenderKitBase.findRenderer(RenderKitBase.java:146)
org.apache.myfaces.trinidadinternal.renderkit.RenderKitDecorator.findRenderer(RenderKitDecorator.java:84)
oracle.adfinternal.view.faces.unified.renderkit.UnifiedRenderKit._getRenderer(UnifiedRenderKit.java:218)
oracle.adfinternal.view.faces.unified.renderkit.UnifiedRenderKit.findRenderer(UnifiedRenderKit.java:156)
org.apache.myfaces.trinidadinternal.renderkit.RenderKitBase.getRenderer(RenderKitBase.java:129)
org.apache.myfaces.trinidad.component.UIXComponentBase._getRendererImpl(UIXComponentBase.java:1143)
org.apache.myfaces.trinidad.component.UIXComponentBase.getRenderer(UIXComponentBase.java:1173)
org.apache.myfaces.trinidad.component.UIXComponentBase.getClientId(UIXComponentBase.java:316)
javax.faces.webapp.UIComponentClassicTagBase.doStartTag(UIComponentClassicTagBase.java:1095)
org.apache.myfaces.trinidad.webapp.UIXComponentELTag.doStartTag(UIXComponentELTag.java:70)
oracle.adfinternal.view.faces.unified.taglib.input.UnifiedSelectBooleanCheckboxTag.doStartTag(UnifiedSelectBooleanCheckboxTag.java:45)
oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
oracle.jsp.runtime.tree.OracleJspIterationTagNode.executeHandler(OracleJspIterationTagNode.java:41)
oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
oracle.jsp.runtime.tree.OracleJspIterationTagNode.executeHandler(OracleJspIterationTagNode.java:41)
oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
oracle.jsp.runtime.tree.OracleJspNode.execute(OracleJspNode.java:73)
oracle.jsp.runtimev2.ShortCutServlet._jspService(ShortCutServlet.java:57)
oracle.jsp.runtime.OracleJspBase.service(OracleJspBase.java:29)
oracle.jsp.runtimev2.JspPageTable.compileAndServe(JspPageTable.java:650)
oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.java:330)
oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.java:680)
oracle.jsp.runtimev2.JspServlet.service(JspServlet.java:653)
javax.servlet.http.HttpServlet.service(HttpServlet.java:815)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:206)
weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.adf.library.webapp.LibraryFilter.doFilter(LibraryFilter.java:147)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
weblogic.servlet.internal.RequestDispatcherImpl.invokeServlet(RequestDispatcherImpl.java:520)
weblogic.servlet.internal.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:112)
com.sun.faces.context.ExternalContextImpl.dispatch(ExternalContextImpl.java:402)
org.apache.myfaces.trinidad.context.ExternalContextDecorator.dispatch(ExternalContextDecorator.java:44)
org.apache.myfaces.trinidad.context.ExternalContextDecorator.dispatch(ExternalContextDecorator.java:44)
org.apache.myfaces.trinidad.context.ExternalContextDecorator.dispatch(ExternalContextDecorator.java:44)
org.apache.myfaces.trinidad.context.ExternalContextDecorator.dispatch(ExternalContextDecorator.java:44)
org.apache.myfaces.trinidadinternal.context.FacesContextFactoryImpl$OverrideDispatch.dispatch(FacesContextFactoryImpl.java:261)
com.sun.faces.application.ViewHandlerImpl.executePageToBuildView(ViewHandlerImpl.java:428)
com.sun.faces.application.ViewHandlerImpl.renderView(ViewHandlerImpl.java:132)
javax.faces.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:189)
org.apache.myfaces.trinidadinternal.application.ViewHandlerImpl.renderView(ViewHandlerImpl.java:155)
oracle.adfinternal.view.faces.lifecycle.LifecycleImpl._renderResponse(LifecycleImpl.java:706)
oracle.adfinternal.view.faces.lifecycle.LifecycleImpl._executePhase(LifecycleImpl.java:244)
oracle.adfinternal.view.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:199)
javax.faces.webapp.FacesServlet.service(FacesServlet.java:246)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:206)
weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.adf.model.servlet.ADFBindingFilter.doFilter(ADFBindingFilter.java:134)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.adfinternal.view.faces.webapp.rich.RegistrationFilter.doFilter(RegistrationFilter.java:69)
org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:411)
oracle.adfinternal.view.faces.activedata.AdsFilter.doFilter(AdsFilter.java:44)
org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:411)
org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl._doFilterImpl(TrinidadFilterImpl.java:210)
org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl.doFilter(TrinidadFilterImpl.java:126)
org.apache.myfaces.trinidad.webapp.TrinidadFilter.doFilter(TrinidadFilter.java:91)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:94)
oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:299)
oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:406)
oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:82)
oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:70)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.adf.library.webapp.LibraryFilter.doFilter(LibraryFilter.java:147)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.dms.wls.DMSServletFilter.doFilter(DMSServletFilter.java:114)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:25)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.doIt(WebAppServletContext.java:3656)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3650)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:308)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:117)
weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2204)
weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2126)
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1412)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:198)
weblogic.work.ExecuteThread.run(ExecuteThread.java:165)
SUBSYSTEM = WebLogicServer USERID = <WLS Kernel> SEVERITY = Error THREAD = [ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)' MSGID = BEA-000337 MACHINE = sis-prosun11.sis.ad TXID = CONTEXTID = 11d1def534ea1be0:3bbe257e:13286031cbf:-7ffd-0000000000002fe7 TIMESTAMP = 1316753908672
WatchAlarmType: AutomaticReset
WatchAlarmResetPeriod: 60000
SMTPNotificationName: MS5-Health-NotificationI found something interesting to the question I posted from the below link:
http://www.mail-archive.com/[email protected]/msg03217.html -
"ThreadPool has stuck threads" - Warning while starting server soa 11.1.1.3
Hi,
When we are trying to restart the server its taking lot of time (nearly 2 hrs) and admin server status is showing as "Warning". when we check Monitoring -> Health for admin server we can see as "Thread Pool as stuck threads".
When clicked on "threadpool", can see "ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" and current request is for "weblogic.kernel.WorkManagerWrapper$1@6ce46ce4" thread.
Any solution for what to do.
Thanks,
RRFurther Updates on this :
I further debugged and found out few more things. This might help.
In Weblogic Admin console Home > Summary of Deployments >oracle-bam(11.1.1)
MDB application oracle-bam is NOT connected to messaging system with below error.
EJBs
MessageDispatcherBean - Error java.lang.IllegalArgumentException: Getting Deployment configuration...
connectionFactoryJNDIName - Red Cross with warning
destinationJNDIName - Red Cross with warning
resourceAdapterJNDIName - Red Cross with warning
Modules
Also please check out another similar thread and please update.
BAMAdapter Issue : java.rmi.ConnectException: Destination unreachable;
Thanks,
Sagar -
Weblogic server + stuck threads
Hi All,
We have java application deployed on weblogic server 10.1.3.4 which is integrated with BPEL (SOA Oracle 11g). The application creates a task through BPEL. We have configured StuckThread Max Time to 600 secs. Now after certain time (5-7 days) the servers goes into warning state due to stuck threads and finally it gets crashed
We noticed the OOM error in jrockit dump, could any one help me here as how to avoid this? your help is very much appreciated, thanksRegarding
Sample of your Weblogic logs showing the java.lang.OutOfMemoryError along with Stack Trace if available
Please find below
[ACTIVE] ExecuteThread: '20' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <532776f80ba738ab:-48a8d4c8:133068f0cee:-8000-00000000000001a5> <1322080967160> <BEA-080003> <RuntimeException thrown by rmi server: javax.management.remote.rmi.RMIConnectionImpl.invoke(Ljavax.management.ObjectName;Ljava.lang.String;Ljava.rmi.MarshalledObject;[Ljava.lang.String;Ljavax.security.auth.Subject;)
javax.management.RuntimeErrorException: java.lang.OutOfMemoryError: allocLargeObjectOrArray: [B, size 1335312.
javax.management.RuntimeErrorException: java.lang.OutOfMemoryError: allocLargeObjectOrArray: [B, size 1335312
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:858)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrowMaybeMBeanException(DefaultMBeanServerInterceptor.java:869)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:838)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$16.run(WLSMBeanServerInterceptorBase.java:449)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:447)
at weblogic.management.mbeanservers.internal.JMXContextInterceptor.invoke(JMXContextInterceptor.java:263)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$16.run(WLSMBeanServerInterceptorBase.java:449)
at weblogic.management.mbeanservers.internal.SecurityInterceptor.invoke(SecurityInterceptor.java:444)
at weblogic.management.jmx.mbeanserver.WLSMBeanServer.invoke(WLSMBeanServer.java:323)
at weblogic.management.mbeanservers.internal.JMXConnectorSubjectForwarder$11.run(JMXConnectorSubjectForwarder.java:661)
at weblogic.management.mbeanservers.internal.JMXConnectorSubjectForwarder.invoke(JMXConnectorSubjectForwarder.java:654)
at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
at javax.management.remote.rmi.RMIConnectionImpl_WLSkel.invoke(Unknown Source)
at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:668)
at weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:118)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:207)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:176)
Caused By: java.lang.OutOfMemoryError: allocLargeObjectOrArray: [B, size 1335312
at java.util.Arrays.copyOf(Arrays.java:2786)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1847)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1847)
at java.io.ObjectOutputStream$BlockDataOutputStream.flush(ObjectOutputStream.java:1792)
at java.io.ObjectOutputStream.flush(ObjectOutputStream.java:699)
at java.io.ObjectOutputStream.close(ObjectOutputStream.java:720)
at oracle.dms.jmx.app.ServerAggreMBeanImpl.getTableGroupInBytes(ServerAggreMBeanImpl.java:328)
at oracle.dms.jmx.app.ServerAggreMBeanWrapper.invokeOperation(ServerAggreMBeanWrapper.java:71)
at oracle.dms.jmx.app.AggreMBeanWrapper.invoke(AggreMBeanWrapper.java:167)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$16.run(WLSMBeanServerInterceptorBase.java:449)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:447)
at weblogic.management.mbeanservers.internal.JMXContextInterceptor.invoke(JMXContextInterceptor.java:263)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$16.run(WLSMBeanServerInterceptorBase.java:449)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:447)
at weblogic.management.mbeanservers.internal.SecurityInterceptor.invoke(SecurityInterceptor.java:444)
at weblogic.management.jmx.mbeanserver.WLSMBeanServer.invoke(WLSMBeanServer.java:323)
at weblogic.management.mbeanservers.internal.JMXConnectorSubjectForwarder$11$1.run(JMXConnectorSubjectForwarder.java:663)
at weblogic.management.mbeanservers.internal.JMXConnectorSubjectForwarder$11.run(JMXConnectorSubjectForwarder.java:661)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.management.mbeanservers.internal.JMXConnectorSubjectForwarder.invoke(JMXConnectorSubjectForwarder.java:654)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1367)
at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
at javax.management.remote.rmi.RMIConnectionImpl_WLSkel.invoke(Unknown Source)
at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:667)
at weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:522)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:146)
at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:518)
at weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:118)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:207)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:176) -
WL Server Stuck Thread on Network Read
I am experiencing an issue with the Weblogic deployment at my organisation.
Firstly an introduction; My Application works by receiving a Web Service (SOAP) Request from a client, the Web Service request is then processed and a internal message packet is generated and sent to a C++ Application ( which is listening on a particular port). The C++ application then processes the information, and returns the response to the web service Layer, which then process the Web Service request back to the calling client.
Currently we are having a situation where some packets are sent to the C++ application as expected. But then we are having situations where some packets NEVER reach the C++ side and the calling client time outs the entire request ( configured timeout is set for 180 seconds) and on the WebLogic Layer we have noticed a significant amount of Stuck ( and Hogging ) Threads. It should be noted that even after a stuck thread is recorded some requests are still being processed normally.
A deeper investigation into the Weblogic Thread dump shows the Weblogic thread is stuck at:
_at sun/nio/ch/IOUtil.read(Ljava/io/FileDescriptor;Ljava/nio/ByteBuffer;JLsun/nio/ch/NativeDispatcher;Ljava/lang/Object;)I(IOUtil.java)[inlined]_+
_at sun/nio/ch/SocketChannelImpl.read(SocketChannelImpl.java:236)[optimized]_+
+_^-- Holding lock: java/lang/Object@0x1a27db60[thin lock]_+
_at com/reso/number/service/comm/SoaSocket.receive(SoaSocket.java:128)[optimized]_+
_at com/reso/number/service/event/OTEvent.submitEvent(TEvent.java:219)[optimized]_+
_at com/reso/number/service/event/SIMQueryEvent.process(QQEvent.java:368)[optimized]_+
It seems that a write is sent to the C++ Application and the thread is 'WAITING' for the response.
Now, the problem is that due to the Infrastructure in place and architectural setup ( We are using Solaris Zone for Node deployment and we have the WebLogic AS on the same node/zone as the C++ application) we are unable to do a network trace to determine whether WebLogic actually sends the request.
Now my questions: i) Are there any known ways to trace the networks packets sent by Java ( maybe via the JVM) ii) Could Garbage collection cause the symptoms being observed? iii)Any recommended ways to resolve this kind of problem?
Thanks for all the help.Increasing the Stuck Thread Max Time will not help. " Stuck Thread Max Time" tells us, after how much amount of time the thread should mark as stuck. Because by default it is 600 sec, any thread which takes more than 600sec for execution will mark as stuck. If you mention 700 sec, after 700sec the thread is marked as stuck. Even though the thread is marked stuck it still process.
Here we need to find why the thread is stuck at
]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-15 "[STUCK] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, priority=1, DAEMON> {
sun.nio.ch.SocketDispatcher.read1(SocketDispatcher.java:???)
sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:46)
sun.nio.ch.IOUtil.read(IOUtil.java:228)
sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:159)
^-- Holding lock: java.lang.Object@16a47a8b[thin lock]
weblogic.socket.NIOSocketMuxer$NIOInputStream.read(NIOSocketMuxer.java:505)
weblogic.servlet.internal.PostInputStream.read(PostInputStream.java:156)
weblogic.servlet.internal.ServletInputStreamImpl.read(ServletInputStreamImpl.java:228)
weblogic.servlet.internal.ServletRequestImpl$RequestParameters.mergePostParams(ServletRequestImpl.java:2062)
You may have to increase the buffer size at sun.nio. -
Hi, From my server log I noticed that there are some stuck threads. After looked at the thread dump thread 18 is blocking. MainPortalServlet.java is extends PortalServlet and line 96 is super.service(request,response); Please help me to analyse why it is blocking. Thanks.
"ExecuteThread: '20' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x0090e6b0 nid=0x3b waiting for monitor entry [0x600fd000..0x600ffc28]
at com.bea.wlw.netui.script.el.util.ParseUtils.parse(ParseUtils.java:100)
- waiting to lock <0x8d3a3f18> (a com.bea.wlw.netui.script.el.util.ParseUtils$Cache)
at com.bea.wlw.netui.script.el.ExpressionEvaluatorImpl.containsExpression(ExpressionEvaluatorImpl.java:239)
at com.bea.wlw.netui.pageflow.ProcessPopulate.populate(ProcessPopulate.java:295)
at com.bea.wlw.netui.pageflow.PageFlowRequestProcessor.processPopulate(PageFlowRequestProcessor.java:402)
at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:254)
at com.bea.wlw.netui.pageflow.PageFlowRequestProcessor.process(PageFlowRequestProcessor.java:691)
at com.bea.wlw.netui.pageflow.AutoRegisterActionServlet.process(AutoRegisterActionServlet.java:527)
at com.bea.wlw.netui.pageflow.PageFlowActionServlet.process(PageFlowActionServlet.java:152)
at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:507)
at com.bea.wlw.netui.pageflow.PageFlowUtils.strutsLookup(PageFlowUtils.java:1786)
at com.bea.portlet.adapter.scopedcontent.ScopedContentCommonSupport.executeAction(ScopedContentCommonSupport.java:561)
at com.bea.portlet.adapter.scopedcontent.ScopedContentCommonSupport.processActionInternal(ScopedContentCommonSupport.java:121)
at com.bea.portlet.adapter.scopedcontent.PageFlowStubImpl.processAction(PageFlowStubImpl.java:98)
at com.bea.netuix.servlets.controls.content.NetuiContent.raiseScopedAction(NetuiContent.java:154)
at com.bea.netuix.servlets.controls.content.NetuiContent.raiseScopedAction(NetuiContent.java:113)
at com.bea.netuix.servlets.controls.content.NetuiContent.handlePostbackData(NetuiContent.java:228)
at com.bea.netuix.nf.ControlLifecycle$3.visit(ControlLifecycle.java:171)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:356)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:126)
at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:105)
at com.bea.netuix.nf.Lifecycle.runInbound(Lifecycle.java:173)
at com.bea.netuix.nf.Lifecycle.run(Lifecycle.java:137)
at com.bea.netuix.servlets.manager.UIServlet.runLifecycle(UIServlet.java:324)
at com.bea.netuix.servlets.manager.UIServlet.processControlTree(UIServlet.java:220)
at com.bea.netuix.servlets.manager.PortalServlet.doPost(PortalServlet.java:820)
at com.bea.netuix.servlets.manager.UIServlet.service(UIServlet.java:150)
at com.cd.servlets.portal.MainPortalServlet.service(MainPortalServlet.java:96)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:28)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
at com.bea.p13n.servlets.PortalServletFilter.doFilter(PortalServletFilter.java:293)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:7053)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3902)
at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2773)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)
"ExecuteThread: '19' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x0090dab8 nid=0x3a runnable [0x601fb000..0x601ffc28]
at java.util.HashMap.getEntry(HashMap.java:361)
at java.util.LinkedHashMap.get(LinkedHashMap.java:266)
at com.bea.wlw.netui.script.el.util.ParseUtils.parse(ParseUtils.java:84)
at com.bea.wlw.netui.script.el.util.ParseUtils.evaluate(ParseUtils.java:123)
at com.bea.wlw.netui.script.el.ExpressionEvaluatorImpl.evaluateStrict(ExpressionEvaluatorImpl.java:87)
at com.bea.wlw.netui.tags.AbstractBaseTag.evaluateExpressionInternal(AbstractBaseTag.java:573)
at com.bea.wlw.netui.tags.AbstractBaseTag.evaluateExpression(AbstractBaseTag.java:362)
at com.bea.wlw.netui.tags.databinding.script.GetData.doEndTag(GetData.java:249)
at jsp_servlet._portlets.___portlet1._jspService(_portlet1.jsp:27)
at weblogic.servlet.jsp.JspBase.service(JspBase.java:33)
at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:28)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
at com.bea.p13n.servlets.PortalServletFilter.doFilter(PortalServletFilter.java:293)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
at com.bea.wlw.netui.pageflow.PageFlowJspFilter.doFilter(PageFlowJspFilter.java:250)
- locked <0x78ccf6c8> (a portlets.Portlet1Controller)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:652)
at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:431)
at com.bea.wlw.netui.pageflow.scoping.internal.ScopedRequestDispatcher.include(ScopedRequestDispatcher.java:120)
at com.bea.netuix.servlets.controls.content.JspContent.beginRender(JspContent.java:534)
at com.bea.netuix.servlets.controls.content.NetuiContent.beginRender(NetuiContent.java:425)
at com.bea.netuix.nf.ControlLifecycle$1.visit(ControlLifecycle.java:495)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:543)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:247)
at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:105)
at com.bea.netuix.nf.Lifecycle.run(Lifecycle.java:356)
at com.bea.netuix.nf.UIControl.render(UIControl.java:536)
at com.bea.netuix.servlets.controls.PresentationContext.render(PresentationContext.java:405)
at com.bea.netuix.servlets.util.RenderToolkit.renderChild(RenderToolkit.java:123)
at com.bea.netuix.servlets.jsp.taglib.RenderChild.doStartTag(RenderChild.java:58)
at jsp_servlet._framework._skeletons._cd.__flowlayout._jspService(flowlayout.jsp:30)
at weblogic.servlet.jsp.JspBase.service(JspBase.java:33)
at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:348)
at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:646)
at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:431)
at com.bea.netuix.servlets.controls.JspRenderer.renderAlt(JspRenderer.java:194)
at com.bea.netuix.servlets.controls.JspRenderer.beginRender(JspRenderer.java:96)
at com.bea.netuix.nf.ControlLifecycle$1.visit(ControlLifecycle.java:491)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:543)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:247)
at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:105)
at com.bea.netuix.nf.Lifecycle.run(Lifecycle.java:356)
at com.bea.netuix.nf.UIControl.render(UIControl.java:536)
at com.bea.netuix.servlets.controls.PresentationContext.render(PresentationContext.java:405)
at com.bea.netuix.servlets.util.RenderToolkit.renderChild(RenderToolkit.java:123)
at com.bea.netuix.servlets.jsp.taglib.RenderChild.doStartTag(RenderChild.java:58)
at jsp_servlet._framework._skeletons._cd.__flowlayout._jspService(flowlayout.jsp:30)
at weblogic.servlet.jsp.JspBase.service(JspBase.java:33)
at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:348)
at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:646)
at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:431)
at com.bea.netuix.servlets.controls.JspRenderer.renderAlt(JspRenderer.java:194)
at com.bea.netuix.servlets.controls.JspRenderer.beginRender(JspRenderer.java:96)
at com.bea.netuix.nf.ControlLifecycle$1.visit(ControlLifecycle.java:491)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:543)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:247)
at com.bea.netuix.nf.Lifecycle.runOutbound(Lifecycle.java:204)
at com.bea.netuix.nf.Lifecycle.run(Lifecycle.java:153)
at com.bea.netuix.servlets.manager.UIServlet.runLifecycle(UIServlet.java:324)
at com.bea.netuix.servlets.manager.UIServlet.processControlTree(UIServlet.java:220)
at com.bea.netuix.servlets.manager.PortalServlet.doPost(PortalServlet.java:820)
at com.bea.netuix.servlets.manager.PortalServlet.doGet(PortalServlet.java:671)
at com.bea.netuix.servlets.manager.UIServlet.service(UIServlet.java:147)
at com.cd.servlets.portal.MainPortalServlet.service(MainPortalServlet.java:116)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:28)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
at com.bea.p13n.servlets.PortalServletFilter.doFilter(PortalServletFilter.java:293)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:7053)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3902)
at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2773)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)
"ExecuteThread: '18' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x0090cec0 nid=0x39 runnable [0x602fd000..0x602ffc28]
at java.util.LinkedHashMap.transfer(LinkedHashMap.java:224)
at java.util.HashMap.resize(HashMap.java:452)
at java.util.LinkedHashMap.addEntry(LinkedHashMap.java:399)
at java.util.HashMap.put(HashMap.java:392)
at com.bea.wlw.netui.script.el.util.ParseUtils.parse(ParseUtils.java:100)
- locked <0x8d3a3f18> (a com.bea.wlw.netui.script.el.util.ParseUtils$Cache)
at com.bea.wlw.netui.script.el.ExpressionEvaluatorImpl.containsExpression(ExpressionEvaluatorImpl.java:239)
at com.bea.wlw.netui.pageflow.ProcessPopulate.populate(ProcessPopulate.java:295)
at com.bea.wlw.netui.pageflow.PageFlowRequestProcessor.processPopulate(PageFlowRequestProcessor.java:402)
at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:254)
at com.bea.wlw.netui.pageflow.PageFlowRequestProcessor.process(PageFlowRequestProcessor.java:691)
at com.bea.wlw.netui.pageflow.AutoRegisterActionServlet.process(AutoRegisterActionServlet.java:527)
at com.bea.wlw.netui.pageflow.PageFlowActionServlet.process(PageFlowActionServlet.java:152)
at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:507)
at com.bea.wlw.netui.pageflow.PageFlowUtils.strutsLookup(PageFlowUtils.java:1786)
at com.bea.portlet.adapter.scopedcontent.ScopedContentCommonSupport.executeAction(ScopedContentCommonSupport.java:561)
at com.bea.portlet.adapter.scopedcontent.ScopedContentCommonSupport.processActionInternal(ScopedContentCommonSupport.java:121)
at com.bea.portlet.adapter.scopedcontent.PageFlowStubImpl.processAction(PageFlowStubImpl.java:98)
at com.bea.netuix.servlets.controls.content.NetuiContent.raiseScopedAction(NetuiContent.java:154)
at com.bea.netuix.servlets.controls.content.NetuiContent.raiseScopedAction(NetuiContent.java:113)
at com.bea.netuix.servlets.controls.content.NetuiContent.handlePostbackData(NetuiContent.java:228)
at com.bea.netuix.nf.ControlLifecycle$3.visit(ControlLifecycle.java:171)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:356)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:126)
at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:105)
at com.bea.netuix.nf.Lifecycle.runInbound(Lifecycle.java:173)
at com.bea.netuix.nf.Lifecycle.run(Lifecycle.java:137)
at com.bea.netuix.servlets.manager.UIServlet.runLifecycle(UIServlet.java:324)
at com.bea.netuix.servlets.manager.UIServlet.processControlTree(UIServlet.java:220)
at com.bea.netuix.servlets.manager.PortalServlet.doPost(PortalServlet.java:820)
at com.bea.netuix.servlets.manager.UIServlet.service(UIServlet.java:150)
at com.cd.servlets.portal.MainPortalServlet.service(MainPortalServlet.java:96)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:28)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
at com.bea.p13n.servlets.PortalServletFilter.doFilter(PortalServletFilter.java:293)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:7053)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3902)
at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2773)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)Hi user9503787,
Can you please confirm if the Weblogic instance / java process you took the Thread Dump from was showing high CPU utilization?
Your Thread Dump data is quite interesting and a perfect example of a non Thread safe HashMap combined with a read/write race condition:
Thread #23 (hanging for 600 seconds+) is showing a potential infinite looping condition in at java.util.HashMap.getEntry(HashMap.java:361)
Thread #22 (hanging for 600 seconds+) is showing a potential infinite looping condition while performing a resize() following a put() operation
Weblogic Portal has some known issues which regards to usage of non Thread safe HashHap data structures; which under load can cause infinite looping and sharp surge of CPU (due to corruption of the internal HashMap index preventing exiting the internal loop); ultimately hanging the whole JVM.
We had the same issue for one of our Weblogic Portal 10.0 production environment and Oracle provided a patch (via Collections.synchronizedMap() ).
Please open a SR and Oracle, share these 2 Threads and discuss with the engineer about a patch / know issues regarding non Thread safe HashMap and infinite looping.
Regards,
P-H
http://javaeesupportpatterns.blogspot.com/ -
Stuck threads reading blob column from db table
WLS 10.3.5, JDK 1.6u29, Oracle 11g RAC, ojdbc6 latest driver
We're having problems with stuck threads trying to read a blob column from a DB table. The query to extract the blob is a simple select, without any locking such as "for update" clauses or whatever. The blob's size is <= 100k.
The thread dump shows the following stack trace:
+"[STUCK] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" RUNNABLE native+
+ java.net.SocketInputStream.socketRead0(Native Method)+
+ java.net.SocketInputStream.read(SocketInputStream.java:129)+
+ oracle.net.ns.Packet.receive(Packet.java:300)+
+ oracle.net.ns.DataPacket.receive(DataPacket.java:106)+
+ oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:315)+
+ oracle.net.ns.NetInputStream.read(NetInputStream.java:260)+
+ oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:105)+
+ oracle.jdbc.driver.T4CMAREngine.getNBytes(T4CMAREngine.java:1517)+
+ oracle.jdbc.driver.T4C8TTILobd.unmarshalLobData(T4C8TTILobd.java:476)+
+ oracle.jdbc.driver.T4C8TTILob.readLOBD(T4C8TTILob.java:770)+
+ oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:361)+
+ oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)+
+ oracle.jdbc.driver.T4C8TTILob.read(T4C8TTILob.java:146)+
+ oracle.jdbc.driver.T4CConnection.getBytes(T4CConnection.java:2392)+
+ oracle.sql.BLOB.getBytes(BLOB.java:348)+
+ oracle.sql.BLOB.getBytes(BLOB.java:222)+
+ weblogic.jdbc.wrapper.Blob_oracle_sql_BLOB.getBytes(Unknown Source)+
+ com.ibatis.sqlmap.engine.type.BlobTypeHandlerCallback.getResult(BlobTypeHandlerCallback.java:33)+
+ com.ibatis.sqlmap.engine.type.CustomTypeHandler.getResult(CustomTypeHandler.java:52)+
+ com.ibatis.sqlmap.engine.mapping.result.ResultMap.getPrimitiveResultMappingValue(ResultMap.java:619)+
+ com.ibatis.sqlmap.engine.mapping.result.ResultMap.getResults(ResultMap.java:345)+
+ com.ibatis.sqlmap.engine.execution.SqlExecutor.handleResults(SqlExecutor.java:384)+
+ com.ibatis.sqlmap.engine.execution.SqlExecutor.handleMultipleResults(SqlExecutor.java:300)+
+ com.ibatis.sqlmap.engine.execution.SqlExecutor.executeQuery(SqlExecutor.java:189)+
+ com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.sqlExecuteQuery(MappedStatement.java:221)+
+ com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.executeQueryWithCallback(MappedStatement.java:189)+
+ com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.executeQueryForList(MappedStatement.java:139)+
+ com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryForList(SqlMapExecutorDelegate.java:567)+
+ com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryForList(SqlMapExecutorDelegate.java:541)+
+ com.ibatis.sqlmap.engine.impl.SqlMapSessionImpl.queryForList(SqlMapSessionImpl.java:118)+
+ com.ibatis.sqlmap.engine.impl.SqlMapClientImpl.queryForList(SqlMapClientImpl.java:94)+
+ com.ibatis.dao.client.template.SqlMapDaoTemplate.queryForList(SqlMapDaoTemplate.java:282)"[STUCK] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" RUNNABLE native+
+ java.net.SocketInputStream.socketRead0(Native Method)+
+ java.net.SocketInputStream.read(SocketInputStream.java:129)+
+ oracle.net.ns.Packet.receive(Packet.java:300)+
+ oracle.net.ns.DataPacket.receive(DataPacket.java:106)+
+ oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:315)+
+ oracle.net.ns.NetInputStream.read(NetInputStream.java:260)+
+ oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:105)+
+ oracle.jdbc.driver.T4CMAREngine.getNBytes(T4CMAREngine.java:1517)+
+ oracle.jdbc.driver.T4C8TTILobd.unmarshalLobData(T4C8TTILobd.java:476)+
+ oracle.jdbc.driver.T4C8TTILob.readLOBD(T4C8TTILob.java:770)+
+ oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:361)+
+ oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)+
+ oracle.jdbc.driver.T4C8TTILob.read(T4C8TTILob.java:146)+
+ oracle.jdbc.driver.T4CConnection.getBytes(T4CConnection.java:2392)+
+ oracle.sql.BLOB.getBytes(BLOB.java:348)+
+ oracle.sql.BLOB.getBytes(BLOB.java:222)+
+ weblogic.jdbc.wrapper.Blob_oracle_sql_BLOB.getBytes(Unknown Source)+
+ com.ibatis.sqlmap.engine.type.BlobTypeHandlerCallback.getResult(BlobTypeHandlerCallback.java:33)+
+ com.ibatis.sqlmap.engine.type.CustomTypeHandler.getResult(CustomTypeHandler.java:52)+
+ com.ibatis.sqlmap.engine.mapping.result.ResultMap.getPrimitiveResultMappingValue(ResultMap.java:619)+
+ com.ibatis.sqlmap.engine.mapping.result.ResultMap.getResults(ResultMap.java:345)+
+ com.ibatis.sqlmap.engine.execution.SqlExecutor.handleResults(SqlExecutor.java:384)+
+ com.ibatis.sqlmap.engine.execution.SqlExecutor.handleMultipleResults(SqlExecutor.java:300)+
+ com.ibatis.sqlmap.engine.execution.SqlExecutor.executeQuery(SqlExecutor.java:189)+
+ com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.sqlExecuteQuery(MappedStatement.java:221)+
com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.executeQueryWithCallback(MappedStatement.java:189)
+ com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.executeQueryForList(MappedStatement.java:139)+
+ com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryForList(SqlMapExecutorDelegate.java:567)+
+ com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryForList(SqlMapExecutorDelegate.java:541)+
+ com.ibatis.sqlmap.engine.impl.SqlMapSessionImpl.queryForList(SqlMapSessionImpl.java:118)+
+ com.ibatis.sqlmap.engine.impl.SqlMapClientImpl.queryForList(SqlMapClientImpl.java:94)+
+ com.ibatis.dao.client.template.SqlMapDaoTemplate.queryForList(SqlMapDaoTemplate.java:282)+
Some threads eventually end (after 1-2 hours), most of them remain there for days.
Any hint would be quite useful, thanks.Threads are executing the actual allocated request from the Weblogic Kernel. Most of the problems happen when the Thread execution is reaching the application or business layer.
At this point your application Java code module is sending or receiving data from external sources such as a an Oracle database for example. Any problem with such external system will cause the Thread to hang and wait for data to come back.
Other situations can occur such as internal deadlock, infinite looping, heavy IO contention on your server etc.
Doesn't loo like a driver issue.
http://docs.oracle.com/cd/E21764_01/doc.1111/e14770/weblogic_server_issues.htm#autoId2
Check at the Database end
Cheers ... -
ST3B14 : xdo ReportService.getReportDefinition gives stuck thread
We use the following code in List LOV source VVO to get the template names associated with the report:
private static List<String> mTemplates = null;
private static ReportService mReportService = null;
private static int mCurrentRow = 0;
private void initReportService() {
try {
mReportService = PPBIPUtil.getReportService();
} catch (Throwable t) {
t.printStackTrace();
protected void executeQueryForCollection(Object qc, Object[] params,
int noUserParams) {
System.out.println("Nitesh : Entering method executeQueryForCollection");
String reportPath =
"/Financials/Receivables/BFBPrintSummary.xdo";
try {
ReportDefinition reportDefinition =
mReportService.getReportDefinition(reportPath);
System.out.println("Nitesh : Getting the templates");
mTemplates = reportDefinition.getTemplateIds().getItem();
System.out.println("Nitesh : mTemplates.size = "+ mTemplates.size());
I have a local ST3B14 deployment pointing to the st3b14 debug db and I see that when the xdo is not uploaded to the BIP server, the parameterVO UI comes up fine, obviously the template LOV is blank as expected as it cannot find the report. However, when I upload the report the highlighted code :
ReportDefinition reportDefinition =
mReportService.getReportDefinition(reportPath);
gives a stuck thread and execution hangs at this point. Here is what I see in the Adminserver logs:
oracle.j2ee.ws.client.jaxws.WsClientProxyInvocationHandler.invoke(WsClientProxyInvocationHandler.java:142)
$Proxy237.getReportDefinition(Unknown Source)
oracle.xdo.service.client.ReportService.getReportDefinition(ReportService.java:307)
oracle.apps.financials.receivables.transactions.balanceForwardBills.publicEssModel.view.BFBSummaryReportTemplateVVOImpl.executeQueryForCollection(BFBSummaryReportTemplateVVOImpl.java:55)
oracle.jbo.server.ViewRowSetImpl.execute(ViewRowSetImpl.java:1036)
^-- Holding lock: oracle.jbo.JboSyncLock@75080cf[thin lock]
oracle.jbo.server.ViewRowSetImpl.execute(ViewRowSetImpl.java:1021)
oracle.jbo.server.ViewRowSetIteratorImpl.ensureRefreshed(ViewRowSetIteratorImpl.java:2738)
^-- Holding lock: oracle.jbo.JboSyncLock@75080cf[thin lock]
oracle.jbo.server.ViewRowSetIteratorImpl.ensureRefreshed(ViewRowSetIteratorImpl.java:2730)
oracle.jbo.server.ViewRowSetIteratorImpl.getAllRowsInRangeInternal(ViewRowSetIteratorImpl.java:2107)
^-- Holding lock: oracle.jbo.JboSyncLock@75080cf[thin lock]
oracle.jbo.server.ViewRowSetIteratorImpl.getAllRowsInRange(ViewRowSetIteratorImpl.java:2165)
Ess server logs
oracle.jdbc.driver.T4CPreparedStatement.fetch(T4CPreparedStatement.java:995)
oracle.jdbc.driver.OracleResultSetImpl.close_or_fetch_from_next(OracleResultSetImpl.java:244)
^-- Holding lock: oracle.jdbc.driver.T4CConnection@13be4a9[thin lock]
oracle.jdbc.driver.OracleResultSetImpl.next(OracleResultSetImpl.java:187)
^-- Holding lock: oracle.jdbc.driver.T4CConnection@13be4a9[thin lock]
weblogic.jdbc.wrapper.ResultSet_oracle_jdbc_driver_OracleResultSetImpl.next(Unknown Source)
Has somebody encountered this? Also, where I can find the XDO source code jar/library that I can import in Jdev to debug further.
thanks
Nitesh
Edited by: user595705 on Nov 2, 2010 3:57 PMHi,
Any updates on this?How did u solve it?I am encountering a similar issue on my junits.
Thanks -
Hi,
When try to access Help --> About Hyperion@Workspace on the Workspace Console, the workspace server hang and require to restart the services.
We are on Hyperion System 9.3.1 and here are the config,
1. Two load balanced Workspace servers. Weblogic 9.1 is running on these
2. One HFM App server
3. One HFM Web server
4. One Shared Services Server
Siteminder SSO has been implemented to log in to workspace console.
The workspace throw's the following error message in the workspace.log file,
####<Jan 13, 2010 3:19:13 PM CST> <Error> <WebLogicServer> <BL-NT-HYDW02> <Workspace> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1263417553179> <BEA-000337> <[STUCK] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "675" seconds working on the request "Http Request: /workspace/modules/com/hyperion/tools/cds/repository/bpm/abtdialog/Adf.jsp", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-72 "[STUCK] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, blocked, priority=1, DAEMON> {
-- Blocked trying to get lock: com.hyperion.tools.config.BPMContextBean@71fff3d[fat lock]
jrockit.vm.Threads.waitForSignal(Native Method)
jrockit.vm.Locks.fatLockBlockOrSpin(Unknown Source)
jrockit.vm.Locks.fatLockReacquire(Unknown Source)
jrockit.vm.Locks.lockFat(Unknown Source)
jrockit.vm.Locks.monitorEnterSecondStage(Unknown Source)
jrockit.vm.Locks.monitorEnter(Unknown Source)
com.hyperion.tools.config.BPMContextBean.readAllAppInfo(Unknown Source)
jsp_servlet._modules._com._hyperion._tools._cds._repository._bpm._abtdialog.__adf._jspService(__adf.java:380)
weblogic.servlet.jsp.JspBase.service(JspBase.java:34)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:225)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:127)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:272)
weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
com.hyperion.bpm.LocalizationServletFilter.doFilter(LocalizationServletFilter.java:76)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
com.hyperion.i18n.filter.Utf8Filter.doFilter(Unknown Source)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3151)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:1973)
weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:1880)
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1310)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:207)
weblogic.work.ExecuteThread.run(ExecuteThread.java:179)
jrockit.vm.RNI.c2java(Native Method)
Appreciated for any help...
Thanks,
Raghu.If you disable SiteMinder does the hanging problem go away? When working with SiteMinder in 11.1.1.2 it had quite a few quirks and I assume 9.3.1 integration wasn't too much better.
Regards,
John A. Booth
http://www.metavero.com
Maybe you are looking for
-
Web-based Chiness Chess Game Application(17-11-02007 OPEN)
The following is access link: http://carltam.no-ip.info:8083/Game/showLogin.html The game is a turn-based one and is implemented on using server/multi-client model(server program, applet) Technology used: signed applet, multi-threading, socket, objec
-
Authorization Based on Key Figure in an IP Layout
Hi, I have a situvation where there are 5 editable key figures in my IP layout, out of which 4 are open for all users and the other should be editable only for a few users. Can i have that kind of authorization in the same layout, where one key figur
-
Creating the New Process Chains in BW System
Hi SDN, What is the best practice of creating the New Process Chain? 1. Whether to create the Process Chain in BW Development first and trasferring it through transfer request OR 2. Create directly the Process Chain in the BW Production and schedule
-
Hi, This might not be the right place, but Sun used to sell a real "hard real time", kernel. I cant seem to find any reference to this on their site anymore. Does it still exist ?, can we still buy it ?, where do I go to find out more information ?.
-
FI IDCO standard ALE configuration
Hi all, I need some help on a solution to set up for FI doc ALe ditstribution using standard tools. The goal os is to send Acc doc from an R/3 to antoher R/3 system via trfc idoc, using standard messages, functions, ... The acc docs should be sent 1/