12.1.3 Servicebus deployment hangs   stuck threads

Hi,
I'm not sure what the issue is but on a new standalone SOA environment, most service bus deployments keep hanging.
(Deployments using sbconfig.jar import).
Not sure what the issue is - I see stuck threads on the AdminServer and the osb managed server and the /servicebus console just hangs forever.
All the dependencies (mainly connection pool jndi names etc.) exist on the target server.
Occasionally this environment also gives me the error shown in the screenshot "Exception occurred while downloading files"
When I import the same sbconfig.jar on a different server (installed locally on my desktop), the deployment goes through perfectly fine.

Hi,
Can You try to delete or rename the data folder present in the Domain_home/Server/AdminServer and then try to restart the server.
Admin Server will create the ldap files again and then you can try to use the same password and check if it works.
You can rename the data .tmp and cache folder
Else You can also create a New Admin user and use them to start the server.
Find the link with more information for the same
Recover WebLogic admin password - weblogicexpert
Hope it helps

Similar Messages

  • Stuck Threads Leading to server hang

    Hi ,
    We have been facing a problem with Stuck Threads in our production environment. We are currently running version 8.1. Though the stuck Threads do point to JDBC issues, we suspect that these problems may be related to some Weblogic/OS configuration issue as well. These probelms eventually lead to a server hang.
    The general pattern of the problem is that we see messages like :
    Dec 24, 2006 8:56:09 AM EST> <Debug> <Common> <BEA-000610> < RP(rtqdbConnectionPool):timeoutInactiveRes (20)>
    ####<Dec 25, 2006 8:54:51 AM EST> <Error> <WebLogicServer> <hygeia> <hygeia-rtq-vm7> <weblogic.health.CoreHealthMonitor> <<WLS Kernel>> <> <BEA-000337> <UNKNOWN_SESSION has been busy for "896" seconds working on the request "Http Request: /quote.do", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.> ==================== This email/fax message is for the sole use of the intended
    Sometimes a deadlock Exception would be thrown from the Database,
    nested exception is java.sql.SQLException: ORA-00060: deadlock detected while waiting for resource
    But this is not as often as we get the Stuck Thread messages.
    This is a typical Thread dump from one of our servers when it complained of a stuck thread.
    ///*************Thread Dump Start**********************///
    Full thread dump Java HotSpot(TM) Server VM (1.4.2_08-b03 mixed mode):
    "idle thread" daemon prio=5 tid=0x00d2c150 nid=0x7f in Object.wait() [94e7f000..94e7fc28]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xb2010000> (a org.xbill.Task.WorkerThread)
         at org.xbill.Task.WorkerThread.run(WorkerThread.java:130)
         - locked <0xb2010000> (a org.xbill.Task.WorkerThread)
    "Thread-28" daemon prio=5 tid=0x00c6a7c0 nid=0x7d in Object.wait() [92c7f000..92c7fc28]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xbce667f8> (a com.geico.util.LockManager)
         at com.geico.util.LockManager.waitForWork(LockManager.java:85)
         - locked <0xbce667f8> (a com.geico.util.LockManager)
         at com.geico.mail.Mail.run(Mail.java:259)
         at java.lang.Thread.run(Thread.java:534)
    "CacheManager-idle" daemon prio=5 tid=0x00c7d950 nid=0x7c in Object.wait() [92d7f000..92d7fc28]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xbce66878> (a com.geico.util.LockManager)
         at com.geico.util.LockManager.waitForWork(LockManager.java:85)
         - locked <0xbce66878> (a com.geico.util.LockManager)
         at com.geico.cache.CacheManager.run(CacheManager.java:353)
         at java.lang.Thread.run(Thread.java:534)
    "ExecuteThread: '2' for queue: 'weblogic.kernel.Non-Blocking'" daemon prio=5 tid=0x006a6c90 nid=0x7a in Object.wait() [92e7f000..92e7fc28]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xbc2a5db8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '1' for queue: 'weblogic.kernel.Non-Blocking'" daemon prio=5 tid=0x0120c568 nid=0x79 in Object.wait() [92f7f000..92f7fc28]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xbc2a5e38> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '0' for queue: 'weblogic.kernel.Non-Blocking'" daemon prio=5 tid=0x006a6af0 nid=0x78 in Object.wait() [9307f000..9307fc28]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xbc2a5eb8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '0' for queue: 'JMSStore<AptSoftFileStore-morpheus-rtq-vm7>.ioThreadPool'" daemon prio=5 tid=0x0052a198 nid=0x77 in Object.wait() [9317f000..9317fc28]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xbc2a5f38> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "Thread-26" daemon prio=5 tid=0x00635d58 nid=0x75 in Object.wait() [9327f000..9327fc28]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xbb97ad20> (a com.geico.util.LockManager)
         at java.lang.Object.wait(Object.java:429)
         at com.geico.util.LockManager.waitForWork(LockManager.java:63)
         - locked <0xbb97ad20> (a com.geico.util.LockManager)
         at com.geico.util.providers.StorageProvider.run(StorageProvider.java:158)
         at java.lang.Thread.run(Thread.java:534)
    "Thread-25" daemon prio=5 tid=0x00fac430 nid=0x72 waiting on condition [9337f000..9337fc28]
         at java.lang.Thread.sleep(Native Method)
         at org.apache.commons.pool.impl.GenericObjectPool$Evictor.run(GenericObjectPool.java:1080)
         at java.lang.Thread.run(Thread.java:534)
    "ListenThread.Default" prio=5 tid=0x014ec6e8 nid=0x71 runnable [9347f000..9347fc28]
         at java.net.PlainSocketImpl.socketAccept(Native Method)
         at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
         - locked <0xbb97ae60> (a java.net.PlainSocketImpl)
         at java.net.ServerSocket.implAccept(ServerSocket.java:448)
         at java.net.ServerSocket.accept(ServerSocket.java:419)
         at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:26)
         at weblogic.t3.srvr.ListenThread.accept(ListenThread.java:735)
         at weblogic.t3.srvr.ListenThread.run(ListenThread.java:301)
    "Thread-24" daemon prio=5 tid=0x011e9668 nid=0x70 waiting on condition [9357f000..9357fc28]
         at java.lang.Thread.sleep(Native Method)
         at com.geico.log.HeartbeatLogger$1.run(HeartbeatLogger.java:133)
    "Thread-22" daemon prio=5 tid=0x011e8610 nid=0x6f waiting on condition [9367f000..9367fc28]
         at java.lang.Thread.sleep(Native Method)
         at com.geico.log.StreamLogger.pause(StreamLogger.java:263)
         at com.geico.log.StreamLogger.access$300(StreamLogger.java:15)
         at com.geico.log.StreamLogger$WorkerThread.run(StreamLogger.java:217)
         at java.lang.Thread.run(Thread.java:534)
    "filewatcher" daemon prio=5 tid=0x0111ca50 nid=0x6e waiting on condition [9377f000..9377fc28]
         at java.lang.Thread.sleep(Native Method)
         at com.geico.watcher.FileWatcher.sleep(FileWatcher.java:135)
         at com.geico.watcher.FileWatcher.run(FileWatcher.java:118)
         at java.lang.Thread.run(Thread.java:534)
    "Thread-18" prio=5 tid=0x00de2388 nid=0x6d in Object.wait() [9437e000..9437fc28]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xa20f1a60> (a weblogic.jms.backend.BEConsumerReceiveRequest)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.jms.dispatcher.Request.sleepTillNotified(Request.java:240)
         at weblogic.jms.dispatcher.Request.wrappedFiniteStateMachine(Request.java:685)
         - locked <0xa20f1a60> (a weblogic.jms.backend.BEConsumerReceiveRequest)
         at weblogic.jms.dispatcher.DispatcherImpl.dispatchAsyncInternal(DispatcherImpl.java:132)
         at weblogic.jms.dispatcher.DispatcherImpl.dispatchAsync(DispatcherImpl.java:123)
         at weblogic.jms.dispatcher.Request.dispatchAsync(Request.java:878)
         at weblogic.jms.frontend.FEConsumer.receive(FEConsumer.java:452)
         at weblogic.jms.frontend.FEConsumer.invoke(FEConsumer.java:643)
         at weblogic.jms.dispatcher.Request.wrappedFiniteStateMachine(Request.java:643)
         at weblogic.jms.dispatcher.DispatcherImpl.dispatchSyncTran(DispatcherImpl.java:192)
         at weblogic.jms.client.JMSSession.receiveMessage(JMSSession.java:456)
         at weblogic.jms.client.JMSConsumer.receive(JMSConsumer.java:421)
         at org.springframework.jms.core.JmsTemplate.doReceive(JmsTemplate.java:800)
         at org.springframework.jms.core.JmsTemplate.doReceive(JmsTemplate.java:783)
         at org.springframework.jms.core.JmsTemplate$8.doInJms(JmsTemplate.java:741)
         at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:582)
         at org.springframework.jms.core.JmsTemplate.receive(JmsTemplate.java:739)
         at org.springframework.jms.core.JmsTemplate.receive(JmsTemplate.java:731)
         at com.geico.uierrorlog.UiErrorUtil.getMessage(UiErrorUtil.java:89)
         at com.geico.uierrorlog.UiErrorDaemon.getMessage(UiErrorDaemon.java:108)
         at com.geico.uierrorlog.UiErrorDaemon.run(UiErrorDaemon.java:166)
         at java.lang.Thread.run(Thread.java:534)
    "StatisticsThread" daemon prio=5 tid=0x00d0c798 nid=0x6c in Object.wait() [9447f000..9447fc28]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xb8ee7b18> (a com.geico.tangosol.stats.StatsDaemon)
         at com.geico.tangosol.stats.StatsDaemon.run(StatsDaemon.java:144)
         - locked <0xb8ee7b18> (a com.geico.tangosol.stats.StatsDaemon)
         at com.tangosol.util.Daemon$1.run(Daemon.java:63)
    "StatisticsThread" daemon prio=5 tid=0x01202c18 nid=0x6b in Object.wait() [9457f000..9457fc28]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xb8ee7bd8> (a com.geico.tangosol.stats.StatsDaemon)
         at com.geico.tangosol.stats.StatsDaemon.run(StatsDaemon.java:144)
         - locked <0xb8ee7bd8> (a com.geico.tangosol.stats.StatsDaemon)
         at com.tangosol.util.Daemon$1.run(Daemon.java:63)
    "StatisticsThread" daemon prio=5 tid=0x00b33f18 nid=0x6a in Object.wait() [9467f000..9467fc28]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xb8ee7c90> (a com.geico.tangosol.stats.StatsDaemon)
         at com.geico.tangosol.stats.StatsDaemon.run(StatsDaemon.java:144)
         - locked <0xb8ee7c90> (a com.geico.tangosol.stats.StatsDaemon)
         at com.tangosol.util.Daemon$1.run(Daemon.java:63)
    "DistributedCache:DistributedRTQOverflow" daemon prio=5 tid=0x009895e8 nid=0x69 in Object.wait() [9477f000..9477fc28]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xb8ee7d40> (a com.tangosol.coherence.component.util.daemon.QueueProcessor$Queue)
         at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
         - locked <0xb8ee7d40> (a com.tangosol.coherence.component.util.daemon.QueueProcessor$Queue)
         at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
         at java.lang.Thread.run(Thread.java:534)
    "DistributedCache:PoolStatsDistributedCache:EventDispatcher" daemon prio=5 tid=0x00dff060 nid=0x68 in Object.wait() [9487f000..9487fc28]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xb8ee7f18> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
         at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
         - locked <0xb8ee7f18> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
         at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
         at java.lang.Thread.run(Thread.java:534)
    "DistributedCache:PoolStatsDistributedCache" daemon prio=5 tid=0x006f3e98 nid=0x67 in Object.wait() [9497f000..9497fc28]
         at java.lang.Object.wait(Native Method)
         at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
         - locked <0xb8ee7fe8> (a com.tangosol.coherence.component.util.daemon.QueueProcessor$Queue)
         at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
         at java.lang.Thread.run(Thread.java:534)
    "StatisticsThread" daemon prio=5 tid=0x0090d8d0 nid=0x66 in Object.wait() [94a7f000..94a7fc28]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xb8ee81c8> (a com.geico.tangosol.stats.StatsDaemon)
         at com.geico.tangosol.stats.StatsDaemon.run(StatsDaemon.java:144)
         - locked <0xb8ee81c8> (a com.geico.tangosol.stats.StatsDaemon)
         at com.tangosol.util.Daemon$1.run(Daemon.java:63)
    "DistributedCache:dist-RTQSession:EventDispatcher" daemon prio=5 tid=0x00a05110 nid=0x65 in Object.wait() [94b7f000..94b7fc28]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xb8ee8278> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
         at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
         - locked <0xb8ee8278> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
         at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
         at java.lang.Thread.run(Thread.java:534)
    "DistributedCache:dist-RTQSession" daemon prio=5 tid=0x0036e600 nid=0x64 in Object.wait() [94c7f000..94c7fc28]
         at java.lang.Object.wait(Native Method)
         at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
         - locked <0xb8ee8348> (a com.tangosol.coherence.component.util.daemon.QueueProcessor$Queue)
         at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
         at java.lang.Thread.run(Thread.java:534)
    "ReplicatedCache" daemon prio=5 tid=0x00e9f9f8 nid=0x63 in Object.wait() [94d7f000..94d7fc28]
         at java.lang.Object.wait(Native Method)
         at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
         - locked <0xb8ee8520> (a com.tangosol.coherence.component.util.daemon.QueueProcessor$Queue)
         at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
         at java.lang.Thread.run(Thread.java:534)
    "Scheduler_Worker-9" prio=5 tid=0x00339820 nid=0x61 in Object.wait() [94f7f000..94f7fc28]
         at java.lang.Object.wait(Native Method)
         at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:423)
         - locked <0xb8ee8788> (a java.lang.Object)
         at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:53)
         at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:514)
    "Scheduler_Worker-8" prio=5 tid=0x0033af58 nid=0x60 in Object.wait() [9507f000..9507fc28]
         at java.lang.Object.wait(Native Method)
         at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:423)
         - locked <0xb8ee8788> (a java.lang.Object)
         at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:53)
         at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:514)
    "Scheduler_Worker-7" prio=5 tid=0x0033a878 nid=0x5f in Object.wait() [9517f000..9517fc28]
         at java.lang.Object.wait(Native Method)
         at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:423)
         - locked <0xb8ee8788> (a java.lang.Object)
         at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:53)
         at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:514)
    "Scheduler_Worker-6" prio=5 tid=0x00935488 nid=0x5e in Object.wait() [9527f000..9527fc28]
         at java.lang.Object.wait(Native Method)
         at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:423)
         - locked <0xb8ee8788> (a java.lang.Object)
         at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:53)
         at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:514)
    "Scheduler_Worker-5" prio=5 tid=0x00bb2190 nid=0x5d in Object.wait() [9537f000..9537fc28]
         at java.lang.Object.wait(Native Method)
         at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:423)
         - locked <0xb8ee8788> (a java.lang.Object)
         at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:53)
         at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:514)
    "Scheduler_Worker-4" prio=5 tid=0x010b9038 nid=0x5c in Object.wait() [9547f000..9547fc28]
         at java.lang.Object.wait(Native Method)
         at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:423)
         - locked <0xb8ee8788> (a java.lang.Object)
         at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:53)
         at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:514)
    "Scheduler_Worker-3" prio=5 tid=0x00a34678 nid=0x5b in Object.wait() [9557f000..9557fc28]
         at java.lang.Object.wait(Native Method)
         at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:423)
         - locked <0xb8ee8788> (a java.lang.Object)
         at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:53)
         at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:514)
    "Scheduler_Worker-2" prio=5 tid=0x014f9860 nid=0x5a in Object.wait() [9567f000..9567fc28]
         at java.lang.Object.wait(Native Method)
         at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:423)
         - locked <0xb8ee8788> (a java.lang.Object)
         at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:53)
         at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:514)
    "Scheduler_Worker-1" prio=5 tid=0x0044e530 nid=0x59 in Object.wait() [9577f000..9577fc28]
         at java.lang.Object.wait(Native Method)
         at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:423)
         - locked <0xb8ee8788> (a java.lang.Object)
         at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:53)
         at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:514)
    "Scheduler_Worker-0" prio=5 tid=0x0110b8e8 nid=0x58 in Object.wait() [9587f000..9587fc28]
         at java.lang.Object.wait(Native Method)
         at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:423)
         - locked <0xb8ee8788> (a java.lang.Object)
         at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:53)
         at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:514)
    "StatisticsThread" daemon prio=5 tid=0x011352b8 nid=0x57 in Object.wait() [9597f000..9597fc28]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xb8ee8b20> (a com.geico.tangosol.stats.StatsDaemon)
         at com.geico.tangosol.stats.StatsDaemon.run(StatsDaemon.java:144)
         - locked <0xb8ee8b20> (a com.geico.tangosol.stats.StatsDaemon)
         at com.tangosol.util.Daemon$1.run(Daemon.java:63)
    "DistributedCache:dist-rtq-sessionheader:EventDispatcher" daemon prio=5 tid=0x004222a0 nid=0x56 in Object.wait() [95a7f000..95a7fc28]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xb8ee8bd8> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
         at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
         - locked <0xb8ee8bd8> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
         at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
         at java.lang.Thread.run(Thread.java:534)
    "DistributedCache:dist-rtq-sessionheader" daemon prio=5 tid=0x01278fa0 nid=0x55 in Object.wait() [95b7f000..95b7fc28]
         at java.lang.Object.wait(Native Method)
         at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
         - locked <0xb8ee8ca8> (a com.tangosol.coherence.component.util.daemon.QueueProcessor$Queue)
         at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
         at java.lang.Thread.run(Thread.java:534)
    "DistributedCache:dist-hibernate-default:EventDispatcher" daemon prio=5 tid=0x015273f0 nid=0x54 in Object.wait() [95c7f000..95c7fc28]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xb8ee8e80> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
         at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
         - locked <0xb8ee8e80> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
         at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
         at java.lang.Thread.run(Thread.java:534)
    "DistributedCache:dist-hibernate-default" daemon prio=5 tid=0x00ea1328 nid=0x53 in Object.wait() [9607f000..9607fc28]
         at java.lang.Object.wait(Native Method)
         at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
         - locked <0xb8ee8f50> (a com.tangosol.coherence.component.util.daemon.QueueProcessor$Queue)
         at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
         at java.lang.Thread.run(Thread.java:534)
    "Invocation:Management" daemon prio=5 tid=0x003d8968 nid=0x52 in Object.wait() [95d7f000..95d7fc28]
         at java.lang.Object.wait(Native Method)
         at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
         - locked <0xb8448698> (a com.tangosol.coherence.component.util.daemon.QueueProcessor$Queue)
         at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
         at java.lang.Thread.run(Thread.java:534)
    "Invocation:Management:EventDispatcher" daemon prio=5 tid=0x007e6040 nid=0x51 in Object.wait() [95e7f000..95e7fc28]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xb84487f0> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
         at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
         - locked <0xb84487f0> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
         at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
         at java.lang.Thread.run(Thread.java:534)
    "TcpRingListener" daemon prio=6 tid=0x007e6f40 nid=0x50 runnable [95f7f000..95f7fc28]
         at java.net.PlainSocketImpl.socketAccept(Native Method)
         at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
         - locked <0xb84488d0> (a java.net.PlainSocketImpl)
         at java.net.ServerSocket.implAccept(ServerSocket.java:448)
         at java.net.ServerSocket.accept(ServerSocket.java:419)
         at com.tangosol.coherence.component.net.socket.TcpSocketAccepter.accept(TcpSocketAccepter.CDB:17)
         at com.tangosol.coherence.component.util.daemon.TcpRingListener.acceptConnection(TcpRingListener.CDB:9)
         at com.tangosol.coherence.component.util.daemon.TcpRingListener.onNotify(TcpRingListener.CDB:1)
         at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:34)
         at java.lang.Thread.run(Thread.java:534)
    "Cluster" daemon prio=5 tid=0x00c94738 nid=0x4e in Object.wait() [9617f000..9617fc28]
         at java.lang.Object.wait(Native Method)
         at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
         - locked <0xb8448ac0> (a com.tangosol.coherence.component.net.Cluster$ClusterService$Queue)
         at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
         at java.lang.Thread.run(Thread.java:534)
    "PacketPublisher" daemon prio=6 tid=0x00d83418 nid=0x4d runnable [9627f000..9627fc28]
         at java.lang.Object.wait(Native Method)
         at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
         - locked <0xb8448c78> (a com.tangosol.coherence.component.net.Cluster$PacketPublisher$Queue)
         at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
         at java.lang.Thread.run(Thread.java:534)
    "PacketReceiver" daemon prio=7 tid=0x00b3cd70 nid=0x4c in Object.wait() [9637f000..9637fc28]
         at java.lang.Object.wait(Native Method)
         at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
         - locked <0xb8448dc0> (a com.tangosol.coherence.component.net.Cluster$PacketReceiver$Queue)
         at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
         at java.lang.Thread.run(Thread.java:534)
    "PacketListenerN" daemon prio=8 tid=0x003d7cd0 nid=0x4b runnable [9647f000..9647fc28]
         at java.net.PlainDatagramSocketImpl.receive(Native Method)
         at java.net.DatagramSocket.receive(DatagramSocket.java:711)
         - locked <0xb84494c8> (a java.net.DatagramPacket)
         - locked <0xb84494e8> (a java.net.MulticastSocket)
         at com.tangosol.coherence.component.net.socket.UdpSocket.receive(UdpSocket.CDB:18)
         at com.tangosol.coherence.component.net.UdpPacket.receive(UdpPacket.CDB:17)
         at com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onWait(PacketListener.CDB:5)
         at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
         at java.lang.Thread.run(Thread.java:534)
    "PacketListener1" daemon prio=8 tid=0x00df5538 nid=0x4a runnable [9657f000..9657fc28]
         at java.net.PlainDatagramSocketImpl.receive(Native Method)
         at java.net.DatagramSocket.receive(DatagramSocket.java:711)
         - locked <0xb8449cb8> (a java.net.DatagramPacket)
         - locked <0xb8449cd8> (a java.net.DatagramSocket)
         at com.tangosol.coherence.component.net.socket.UdpSocket.receive(UdpSocket.CDB:18)
         at com.tangosol.coherence.component.net.UdpPacket.receive(UdpPacket.CDB:17)
         at com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onWait(PacketListener.CDB:5)
         at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
         at java.lang.Thread.run(Thread.java:534)
    "Logger@13785534 3.1.2/343" daemon prio=3 tid=0x006b7358 nid=0x49 in Object.wait() [9667f000..9667fc28]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xb8449e90> (a com.tangosol.coherence.component.application.console.Coherence$Logger$Queue)
         at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
         - locked <0xb8449e90> (a com.tangosol.coherence.component.application.console.Coherence$Logger$Queue)
         at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
         at java.lang.Thread.run(Thread.java:534)
    "ExecuteThread: '5' for queue: 'JMS.TimerClientPool'" daemon prio=5 tid=0x011384b0 nid=0x48 in Object.wait() [9677f000..9677fc28]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xb82c91f8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '4' for queue: 'JMS.TimerClientPool'" daemon prio=5 tid=0x01137bc8 nid=0x47 in Object.wait() [9687f000..9687fc28]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xb82c9278> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '3' for queue: 'JMS.TimerClientPool'" daemon prio=5 tid=0x0111d4e0 nid=0x46 in Object.wait() [9697f000..9697fc28]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xb82c92f8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '2' for queue: 'JMS.TimerClientPool'" daemon prio=5 tid=0x010dd218 nid=0x45 in Object.wait() [96a7f000..96a7fc28]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xb82c9378> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '1' for queue: 'JMS.TimerClientPool'" daemon prio=5 tid=0x00bb1c98 nid=0x44 in Object.wait() [96b7f000..96b7fc28]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xb82c93f8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '0' for queue: 'JMS.TimerClientPool'" daemon prio=5 tid=0x00979ac8 nid=0x43 in Object.wait() [96c7f000..96c7fc28]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xb82c9478> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "Thread-12" daemon prio=5 tid=0x00200610 nid=0x42 waiting on condition [96d7f000..96d7fc28]
         at java.lang.Thread.sleep(Native Method)
         at com.geico.log.HeartbeatLogger$1.run(HeartbeatLogger.java:133)
    "Thread-10" daemon prio=5 tid=0x00200470 nid=0x41 waiting on condition [96e7f000..96e7fc28]
         at java.lang.Thread.sleep(Native Method)
         at com.geico.log.StreamLogger.pause(StreamLogger.java:267)
         at com.geico.log.StreamLogger.access$300(StreamLogger.java:19)
         at com.geico.log.StreamLogger$WorkerThread.run(StreamLogger.java:221)
         at java.lang.Thread.run(Thread.java:534)
    "ClockDaemonThread-0" daemon prio=5 tid=0x00ab6a48 nid=0x40 in Object.wait() [96f7f000..96f7fc28]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xb8051b20> (a com.geico.util.concurrent.ClockDaemon)
         at com.geico.util.concurrent.ClockDaemon.nextTask(ClockDaemon.java:328)
         - locked <0xb8051b20> (a com.geico.util.concurrent.ClockDaemon)
         at com.geico.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:362)
         at java.lang.Thread.run(Thread.java:534)
    "Thread-9" prio=5 tid=0x008912e0 nid=0x3f in Object.wait() [9ad7f000..9ad7fc28]
         at java.lang.Object.wait(Native Method)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <0xb8051bc0> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "ExecuteThread: '0' for queue: 'JMS.TimerTreePool'" daemon prio=5 tid=0x001d7fa8 nid=0x3e in Object.wait() [9ae7f000..9ae7fc28]
         at java.lang.Object.wait(Native Method)
         at weblogic.jms.backend.BETimerTree.execute(BETimerTree.java:146)
         - locked <0xb8051c90> (a weblogic.jms.backend.BETimerTree)
         at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)
    "Thread-8" prio=5 tid=0x00247418 nid=0x3d in Object.wait() [9af7f000..9af7fc28]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xb8051d20> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <0xb8051d20> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "Thread-7" prio=5 tid=0x001d8a28 nid=0x3c in Object.wait() [9b07f000..9b07fc28]
         at java.lang.Object.wait(Native Method)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <0xb8051dc0> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "ExecuteThread: '0' for queue: 'weblogic.cluster.MulticastManager'" daemon prio=5 tid=0x00da0a60 nid=0x3b runnable [9b17f000..9b17fc28]
         at java.net.PlainDatagramSocketImpl.receive(Native Method)
         at java.net.DatagramSocket.receive(DatagramSocket.java:711)
         - locked <0xa22dc3e8> (a java.net.DatagramPacket)
         - locked <0xb7558130> (a java.net.MulticastSocket)
         at weblogic.cluster.FragmentSocket.receive(FragmentSocket.java:169)
         at weblogic.cluster.MulticastManager.execute(MulticastManager.java:395)
         at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)
    "SSLListenThread.Administrator" prio=5 tid=0x00c7db08 nid=0x3a runnable [9b27f000..9b27fc28]
         at java.net.PlainSocketImpl.socketAccept(Native Method)
         at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
         - locked <0xb75582a0> (a java.net.PlainSocketImpl)
         at java.net.ServerSocket.implAccept(ServerSocket.java:448)
         at javax.net.ssl.impl.SSLServerSocketImp

    Hi,
    I guess this issue is related to your connectivity and hyperion services. Can you take a restart and try to execute the reports. Also try to check all your Hyperion services are up and running before executing the FR Reports. You can also check the Reporting services by login to Weblogic server (if you have applied it in your environment). In weblogic it should be in Running mode and not in Admin mode. If it is not in Running mode then try to check the Data Source setting from Connection Pool in Weblogic Server.
    Soumya

  • Problem Deploying Composites to Oracle SOA 11.1.1.5 (STUCK Thread Issue)

    Hi All,
    I got stuck thread issues when deploying composites to Oracle SOA 11.1.1.5 running on Windows Server 2008 R2. Even the SimpleApproval composite which ships with the standard installation is having trouble deploying.
    The composites are deploying fine in Oracle SOA 11.1.1.5 running on Windows XP.
    I'm not so good with thread dumps, so I post it below for rescue.
    Please advice,
    Rommel
    <Jul 18, 2011 10:15:25 AM AST> <Error> <WebLogicServer> <BEA-000337> <[STUCK] Ex
    ecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)' has been bus
    y for "605" seconds working on the request "weblogic.servlet.internal.ServletReq
    uestImpl@32d047d4[
    POST /soa-infra/deployer HTTP/1.1
    Connection: TE
    TE: trailers, deflate, gzip, compress
    User-Agent: Oracle HTTPClient Version 10h
    Accept-Encoding: gzip, x-gzip, compress, x-compress
    ECID-Context:
    Authorization: Basic bmFnaGl3ZWJsb2dpY3NvYTp3ZWJsb2dpY3NvYTEwIQ==
    Content-type: application/octet-stream
    Content-Length: 25876
    ]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds
    . Stack trace:
    Thread-55 "[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-
    tuning)'" <alive, in native, suspended, priority=1, DAEMON> {
    jrockit.net.SocketNativeIO.readBytesPinned(SocketNativeIO.java:???)
    jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:24)
    java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)
    java.net.SocketInputStream.read(SocketInputStream.java:107)
    oracle.net.nt.MetricsEnabledInputStream.read(TcpNTAdapter.java:707)
    oracle.net.ns.Packet.receive(Packet.java:243)
    oracle.net.ns.DataPacket.receive(DataPacket.java:106)
    oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:309)
    oracle.net.ns.NetInputStream.read(NetInputStream.java:254)
    oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrap
    per.java:93)
    oracle.jdbc.driver.T4CMAREngine.getNBytes(T4CMAREngine.java:1573)
    oracle.jdbc.driver.T4C8TTILobd.unmarshalLobData(T4C8TTILobd.java:360)
    oracle.jdbc.driver.T4C8TTILob.readLOBD(T4C8TTILob.java:787)
    oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:303)
    oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:181)
    oracle.jdbc.driver.T4C8TTILob.read(T4C8TTILob.java:139)
    oracle.jdbc.driver.T4CConnection.getBytes(T4CConnection.java:2690)
    ^-- Holding lock: oracle.jdbc.driver.T4CConnection@32386377[thin lock]
    oracle.sql.BLOB.getBytes(BLOB.java:557)
    oracle.jdbc.driver.OracleBlobInputStream.needBytes(OracleBlobInputStream.jav
    a:188)
    oracle.jdbc.driver.OracleBufferedStream.readInternal(OracleBufferedStream.ja
    va:171)
    oracle.jdbc.driver.OracleBufferedStream.read(OracleBufferedStream.java:151)
    ^-- Holding lock: oracle.jdbc.driver.OracleBlobInputStream@341d0b87[thin loc
    k]
    java.io.BufferedInputStream.fill(BufferedInputStream.java:189)
    java.io.BufferedInputStream.read(BufferedInputStream.java:236)
    ^-- Holding lock: oracle.mds.internal.persistence.db.DBInputStream@341d0bb2[
    thin lock]
    oracle.xml.parser.v2.XMLReader.pushXMLReader(XMLReader.java:363)
    oracle.xml.parser.v2.XMLReader.pushXMLReader(XMLReader.java:230)
    oracle.xml.parser.v2.XMLParser.parse(XMLParser.java:217)
    oracle.fabric.common.wsdl.SchemaBuilder.parseXMLSrc(SchemaBuilder.java:1016)
    oracle.fabric.common.wsdl.SchemaBuilder.parseXSDSrc(SchemaBuilder.java:984)
    oracle.fabric.common.wsdl.SchemaBuilder.processSchemaImportsAndInclude(Schem
    aBuilder.java:755)
    oracle.fabric.common.wsdl.SchemaBuilder.loadEmbeddedSchemas(SchemaBuilder.ja
    va:474)
    oracle.fabric.common.wsdl.SchemaBuilder.loadSchemasFromWSDL(SchemaBuilder.ja
    va:423)
    oracle.fabric.common.wsdl.SchemaBuilder.loadAllSchemas(SchemaBuilder.java:21
    4)
    oracle.fabric.common.wsdl.SchemaManager.loadAllXSD(SchemaManager.java:209)
    ^-- Holding lock: oracle.fabric.common.wsdl.SchemaManager@36279195[thin lock
    oracle.fabric.common.wsdl.SchemaManager.getXSDEntityResolver(SchemaManager.j
    ava:424)
    oracle.fabric.composite.model.CompositeModel.loadImports(CompositeModel.java
    :361)
    oracle.fabric.composite.model.CompositeModel.getWSDLManager(CompositeModel.j
    ava:196)
    oracle.integration.platform.blocks.adapter.AbstractAdapterBindingComponent.g
    etWSDLDefinition(AbstractAdapterBindingComponent.java:168)
    oracle.integration.platform.blocks.adapter.AdapterReference.loadDefinition(A
    dapterReference.java:543)
    oracle.integration.platform.blocks.adapter.AdapterReference.load(AdapterRefe
    rence.java:377)
    oracle.integration.platform.blocks.adapter.AdapterReference.load(AdapterRefe
    rence.java:73)
    oracle.integration.platform.blocks.deploy.CompositeDeploymentConnection.depl
    oyReferences(CompositeDeploymentConnection.java:177)
    oracle.integration.platform.blocks.deploy.CompositeDeploymentConnection.depl
    oy(CompositeDeploymentConnection.java:81)
    oracle.integration.platform.blocks.deploy.CompositeDeploymentManagerImpl.ini
    tDeployment(CompositeDeploymentManagerImpl.java:143)
    oracle.integration.platform.blocks.deploy.CompositeDeploymentManagerImpl.loa
    d(CompositeDeploymentManagerImpl.java:61)
    sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:?
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:27
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
    .java:25)
    java.lang.reflect.Method.invoke(Method.java:575)
    org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopU
    tils.java:306)
    org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint
    (ReflectiveMethodInvocation.java:182)
    org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
    iveMethodInvocation.java:148)
    oracle.integration.platform.blocks.deploy.DeploymentEventPublisher.invoke(De
    ploymentEventPublisher.java:57)
    org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
    iveMethodInvocation.java:148)
    org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopPro
    xy.java:151)
    $Proxy309.load(Unknown Source)
    oracle.integration.platform.blocks.deploy.StandaloneCompositeDeploymentCoord
    inatorImpl.coordinateCompositeDeployment(StandaloneCompositeDeploymentCoordinato
    rImpl.java:54)
    oracle.integration.platform.blocks.deploy.servlet.BaseDeployProcessor.deploy
    NewComposite(BaseDeployProcessor.java:365)
    oracle.integration.platform.blocks.deploy.servlet.BaseDeployProcessor.deploy
    SARs(BaseDeployProcessor.java:123)
    ^-- Holding lock: java.lang.Object@3621c97c[thin lock]
    oracle.integration.platform.blocks.deploy.servlet.DeployProcessor.doDeployWo
    rk(DeployProcessor.java:122)
    oracle.integration.platform.blocks.deploy.servlet.DeployProcessor.doDeployWo
    rk(DeployProcessor.java:107)
    oracle.integration.platform.blocks.deploy.servlet.DeployProcessor.doDeploy(D
    eployProcessor.java:96)
    oracle.integration.platform.blocks.deploy.servlet.DeployProcessor.process(De
    ployProcessor.java:68)
    oracle.integration.platform.blocks.deploy.servlet.CompositeDeployerServlet.d
    oPostInsideLoggingSession(CompositeDeployerServlet.java:141)
    oracle.integration.platform.blocks.deploy.servlet.CompositeDeployerServlet.d
    oPost(CompositeDeployerServlet.java:119)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:700)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:815)
    weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSe
    curityHelper.java:224)
    weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelpe
    r.java:108)
    weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:206)
    weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:111)
    oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:299)
    oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java
    :405)
    oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:94)
    oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:98)
    oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:70)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:111)
    oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:299)
    oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java
    :405)
    oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:94)
    oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:98)
    oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:70)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:86)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapR
    un(WebAppServletContext.java:3687)
    weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(W
    ebAppServletContext.java:3681)
    weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubjec
    t.java:308)
    weblogic.security.service.SecurityManager.runAs(SecurityManager.java:116)
    weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletC
    ontext.java:2213)
    weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.
    java:2135)
    weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:142
    0)
    weblogic.work.ExecuteThread.execute(ExecuteThread.java:203)
    weblogic.work.ExecuteThread.run(ExecuteThread.java:170)
    >

    I just noted that this is a database connectivity issue because when I point my soa related data sources (mds, soa_infra, etc...)to a local xe database, the deployment is doing fine.

  • ST3B14 : xdo  ReportService.getReportDefinition gives stuck thread

    We use the following code in List LOV source VVO to get the template names associated with the report:
    private static List<String> mTemplates = null;
    private static ReportService mReportService = null;
    private static int mCurrentRow = 0;
    private void initReportService() { 
    try { 
    mReportService = PPBIPUtil.getReportService();
    } catch (Throwable t) {
    t.printStackTrace();
    protected void executeQueryForCollection(Object qc, Object[] params,
    int noUserParams) {
    System.out.println("Nitesh : Entering method executeQueryForCollection");
    String reportPath =
    "/Financials/Receivables/BFBPrintSummary.xdo";
    try {
    ReportDefinition reportDefinition =
    mReportService.getReportDefinition(reportPath);
    System.out.println("Nitesh : Getting the templates");
    mTemplates = reportDefinition.getTemplateIds().getItem();
    System.out.println("Nitesh : mTemplates.size = "+ mTemplates.size());
    I have a local ST3B14 deployment pointing to the st3b14 debug db and I see that when the xdo is not uploaded to the BIP server, the parameterVO UI comes up fine, obviously the template LOV is blank as expected as it cannot find the report. However, when I upload the report the highlighted code :
    ReportDefinition reportDefinition =
    mReportService.getReportDefinition(reportPath);
    gives a stuck thread and execution hangs at this point. Here is what I see in the Adminserver logs:
    oracle.j2ee.ws.client.jaxws.WsClientProxyInvocationHandler.invoke(WsClientProxyInvocationHandler.java:142)
    $Proxy237.getReportDefinition(Unknown Source)
    oracle.xdo.service.client.ReportService.getReportDefinition(ReportService.java:307)
    oracle.apps.financials.receivables.transactions.balanceForwardBills.publicEssModel.view.BFBSummaryReportTemplateVVOImpl.executeQueryForCollection(BFBSummaryReportTemplateVVOImpl.java:55)
    oracle.jbo.server.ViewRowSetImpl.execute(ViewRowSetImpl.java:1036)
    ^-- Holding lock: oracle.jbo.JboSyncLock@75080cf[thin lock]
    oracle.jbo.server.ViewRowSetImpl.execute(ViewRowSetImpl.java:1021)
    oracle.jbo.server.ViewRowSetIteratorImpl.ensureRefreshed(ViewRowSetIteratorImpl.java:2738)
    ^-- Holding lock: oracle.jbo.JboSyncLock@75080cf[thin lock]
    oracle.jbo.server.ViewRowSetIteratorImpl.ensureRefreshed(ViewRowSetIteratorImpl.java:2730)
    oracle.jbo.server.ViewRowSetIteratorImpl.getAllRowsInRangeInternal(ViewRowSetIteratorImpl.java:2107)
    ^-- Holding lock: oracle.jbo.JboSyncLock@75080cf[thin lock]
    oracle.jbo.server.ViewRowSetIteratorImpl.getAllRowsInRange(ViewRowSetIteratorImpl.java:2165)
    Ess server logs
    oracle.jdbc.driver.T4CPreparedStatement.fetch(T4CPreparedStatement.java:995)
    oracle.jdbc.driver.OracleResultSetImpl.close_or_fetch_from_next(OracleResultSetImpl.java:244)
    ^-- Holding lock: oracle.jdbc.driver.T4CConnection@13be4a9[thin lock]
    oracle.jdbc.driver.OracleResultSetImpl.next(OracleResultSetImpl.java:187)
    ^-- Holding lock: oracle.jdbc.driver.T4CConnection@13be4a9[thin lock]
    weblogic.jdbc.wrapper.ResultSet_oracle_jdbc_driver_OracleResultSetImpl.next(Unknown Source)
    Has somebody encountered this? Also, where I can find the XDO source code jar/library that I can import in Jdev to debug further.
    thanks
    Nitesh
    Edited by: user595705 on Nov 2, 2010 3:57 PM

    Hi,
    Any updates on this?How did u solve it?I am encountering a similar issue on my junits.
    Thanks

  • Stuck thread in Oracle Weblogic server

    Dear All,
    We have a stuck thread in Oracle WLS 10.3.3. We have deployed 7 ear files in 5th managed server(psMS5). We have stuck thread in one of the application (psinsurance). We have allocated 1GB heap space for the psMS5 managed server. We are using this in Development environment. Please check the below message from psMS5 managed server log file. Kindly provide reason for stuck thread and solution.
    WatchTime: Sep 23, 2011 10:28:29 AM IST
    WatchDomainName: dev_domain
    WatchServerName: psMS5
    WatchSeverityLevel: Warning
    WatchName: MS5-Watch
    WatchRuleType: Log
    WatchRule: ((MESSAGE LIKE '%has been busy for "%') AND (MSGID = 'BEA-000337') AND (SERVER = 'psMS5'))
    WatchData: DATE = Sep 23, 2011 10:28:28 AM IST SERVER = psMS5 MESSAGE = [STUCK] ExecuteThread: '22' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "629" seconds working on the request "weblogic.servlet.internal.ServletRequestImpl@1cac6525[
    POST */psinsurance/app/lending/insurance/PS00738D.jspx*?_adf.ctrl-state=24f0li74e_4 HTTP/1.1
    Accept: image/jpeg, application/x-ms-application, image/gif, application/xaml+xml, image/pjpeg, application/x-ms-xbap, */*
    Referer: http://sis-prosun11/psinsurance/faces/PS00003?jsessionid=r2BrT8TpqZBsnLhCw2Y8KDpTV7hMpgnLDFW54P87ZpHwn8JMy88s%21168932459%211316753022074&_afrWindowMode=0&_afrLoop=244724092027000&_adf.ctrl-state=24f0li74e_4
    Accept-Language: en-US
    User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; Tablet PC 2.0)
    Content-Type: application/x-www-form-urlencoded
    Accept-Encoding: gzip, deflate
    Content-Length: 554
    Cache-Control: no-cache
    Cookie: JSESSIONID=LkJcT8TfMT64SyQJT4Ls4sXhfDl0pTS7CXJVQ9KcryHGWJ1dhyfz!1373322659
    Connection: Keep-Alive
    Proxy-Client-IP: 192.168.9.179
    X-Forwarded-For: 192.168.9.179
    X-WebLogic-KeepAliveSecs: 30
    X-WebLogic-Force-JVMID: 1373322659
    ]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
    Thread-237 "[STUCK] ExecuteThread: '22' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, suspended, priority=1, DAEMON> {
    java.io.UnixFileSystem.getBooleanAttributes0(UnixFileSystem.java:???)
    java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:228)
    java.io.File.exists(File.java:729)
    weblogic.utils.classloaders.ClasspathClassFinder.getFileSource(ClasspathClassFinder.java:383)
    weblogic.utils.classloaders.ClasspathClassFinder.getSourcesInternal(ClasspathClassFinder.java:333)
    weblogic.utils.classloaders.ClasspathClassFinder.getSource(ClasspathClassFinder.java:316)
    weblogic.utils.classloaders.JarClassFinder.getSource(JarClassFinder.java:58)
    weblogic.utils.classloaders.AbstractClassFinder.getClassSource(AbstractClassFinder.java:22)
    weblogic.utils.classloaders.MultiClassFinder.getClassSource(MultiClassFinder.java:56)
    weblogic.utils.classloaders.CodeGenClassFinder.getClassSource(CodeGenClassFinder.java:26)
    weblogic.utils.classloaders.GenericClassLoader.findLocalClass(GenericClassLoader.java:268)
    *^-- Holding lock: weblogic.utils.classloaders.GenericClassLoader@1e3295a1[thin lock]*
    weblogic.utils.classloaders.GenericClassLoader.findClass(GenericClassLoader.java:237)
    java.lang.ClassLoader.loadClass(ClassLoader.java:292)
    *^-- Holding lock: weblogic.utils.classloaders.GenericClassLoader@1e3295a1[thin lock]*
    java.lang.ClassLoader.loadClass(ClassLoader.java:292)
    java.lang.ClassLoader.loadClass(ClassLoader.java:248)
    weblogic.utils.classloaders.GenericClassLoader.loadClass(GenericClassLoader.java:177)
    weblogic.utils.classloaders.ChangeAwareClassLoader.loadClass(ChangeAwareClassLoader.java:37)
    *^-- Holding lock: weblogic.utils.classloaders.ChangeAwareClassLoader@1e553474[thin lock]*
    java.lang.Class.getDeclaredConstructors0(Class.java:???)
    java.lang.Class.privateGetDeclaredConstructors(Class.java:2370)
    java.lang.Class.getConstructor0(Class.java:2699)
    java.lang.Class.newInstance0(Class.java:318)
    java.lang.Class.newInstance(Class.java:305)
    org.apache.myfaces.trinidadinternal.renderkit.ClassRendererInstantiator.instantiate(ClassRendererInstantiator.java:48)
    org.apache.myfaces.trinidadinternal.renderkit.RenderKitBase.findRenderer(RenderKitBase.java:146)
    org.apache.myfaces.trinidadinternal.renderkit.RenderKitDecorator.findRenderer(RenderKitDecorator.java:84)
    oracle.adfinternal.view.faces.unified.renderkit.UnifiedRenderKit._getRenderer(UnifiedRenderKit.java:218)
    oracle.adfinternal.view.faces.unified.renderkit.UnifiedRenderKit.findRenderer(UnifiedRenderKit.java:156)
    org.apache.myfaces.trinidadinternal.renderkit.RenderKitBase.getRenderer(RenderKitBase.java:129)
    org.apache.myfaces.trinidad.component.UIXComponentBase._getRendererImpl(UIXComponentBase.java:1143)
    org.apache.myfaces.trinidad.component.UIXComponentBase.getRenderer(UIXComponentBase.java:1173)
    org.apache.myfaces.trinidad.component.UIXComponentBase.getClientId(UIXComponentBase.java:316)
    javax.faces.webapp.UIComponentClassicTagBase.doStartTag(UIComponentClassicTagBase.java:1095)
    org.apache.myfaces.trinidad.webapp.UIXComponentELTag.doStartTag(UIXComponentELTag.java:70)
    oracle.adfinternal.view.faces.unified.taglib.input.UnifiedSelectBooleanCheckboxTag.doStartTag(UnifiedSelectBooleanCheckboxTag.java:45)
    oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
    oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
    oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
    oracle.jsp.runtime.tree.OracleJspIterationTagNode.executeHandler(OracleJspIterationTagNode.java:41)
    oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
    oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
    oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
    oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
    oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
    oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
    oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
    oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
    oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
    oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
    oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
    oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
    oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
    oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
    oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
    oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
    oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
    oracle.jsp.runtime.tree.OracleJspIterationTagNode.executeHandler(OracleJspIterationTagNode.java:41)
    oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
    oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
    oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
    oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
    oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
    oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
    oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
    oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
    oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
    oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
    oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
    oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
    oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
    oracle.jsp.runtime.tree.OracleJspNode.execute(OracleJspNode.java:73)
    oracle.jsp.runtimev2.ShortCutServlet._jspService(ShortCutServlet.java:57)
    oracle.jsp.runtime.OracleJspBase.service(OracleJspBase.java:29)
    oracle.jsp.runtimev2.JspPageTable.compileAndServe(JspPageTable.java:650)
    oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.java:330)
    oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.java:680)
    oracle.jsp.runtimev2.JspServlet.service(JspServlet.java:653)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:815)
    weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
    weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
    weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:206)
    weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    oracle.adf.library.webapp.LibraryFilter.doFilter(LibraryFilter.java:147)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    weblogic.servlet.internal.RequestDispatcherImpl.invokeServlet(RequestDispatcherImpl.java:520)
    weblogic.servlet.internal.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:112)
    com.sun.faces.context.ExternalContextImpl.dispatch(ExternalContextImpl.java:402)
    org.apache.myfaces.trinidad.context.ExternalContextDecorator.dispatch(ExternalContextDecorator.java:44)
    org.apache.myfaces.trinidad.context.ExternalContextDecorator.dispatch(ExternalContextDecorator.java:44)
    org.apache.myfaces.trinidad.context.ExternalContextDecorator.dispatch(ExternalContextDecorator.java:44)
    org.apache.myfaces.trinidad.context.ExternalContextDecorator.dispatch(ExternalContextDecorator.java:44)
    org.apache.myfaces.trinidadinternal.context.FacesContextFactoryImpl$OverrideDispatch.dispatch(FacesContextFactoryImpl.java:261)
    com.sun.faces.application.ViewHandlerImpl.executePageToBuildView(ViewHandlerImpl.java:428)
    com.sun.faces.application.ViewHandlerImpl.renderView(ViewHandlerImpl.java:132)
    javax.faces.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:189)
    org.apache.myfaces.trinidadinternal.application.ViewHandlerImpl.renderView(ViewHandlerImpl.java:155)
    oracle.adfinternal.view.faces.lifecycle.LifecycleImpl._renderResponse(LifecycleImpl.java:706)
    oracle.adfinternal.view.faces.lifecycle.LifecycleImpl._executePhase(LifecycleImpl.java:244)
    oracle.adfinternal.view.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:199)
    javax.faces.webapp.FacesServlet.service(FacesServlet.java:246)
    weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
    weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
    weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:206)
    weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    oracle.adf.model.servlet.ADFBindingFilter.doFilter(ADFBindingFilter.java:134)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    oracle.adfinternal.view.faces.webapp.rich.RegistrationFilter.doFilter(RegistrationFilter.java:69)
    org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:411)
    oracle.adfinternal.view.faces.activedata.AdsFilter.doFilter(AdsFilter.java:44)
    org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:411)
    org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl._doFilterImpl(TrinidadFilterImpl.java:210)
    org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl.doFilter(TrinidadFilterImpl.java:126)
    org.apache.myfaces.trinidad.webapp.TrinidadFilter.doFilter(TrinidadFilter.java:91)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:94)
    oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:299)
    oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:406)
    oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:82)
    oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:70)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    oracle.adf.library.webapp.LibraryFilter.doFilter(LibraryFilter.java:147)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    oracle.dms.wls.DMSServletFilter.doFilter(DMSServletFilter.java:114)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:25)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.doIt(WebAppServletContext.java:3656)
    weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3650)
    weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:308)
    weblogic.security.service.SecurityManager.runAs(SecurityManager.java:117)
    weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2204)
    weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2126)
    weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1412)
    weblogic.work.ExecuteThread.execute(ExecuteThread.java:198)
    weblogic.work.ExecuteThread.run(ExecuteThread.java:165)
    SUBSYSTEM = WebLogicServer USERID = <WLS Kernel> SEVERITY = Error THREAD = [ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)' MSGID = BEA-000337 MACHINE = sis-prosun11.sis.ad TXID = CONTEXTID = 11d1def534ea1be0:3bbe257e:13286031cbf:-7ffd-0000000000002fe7 TIMESTAMP = 1316753908672
    WatchAlarmType: AutomaticReset
    WatchAlarmResetPeriod: 60000
    SMTPNotificationName: MS5-Health-Notification

    I found something interesting to the question I posted from the below link:
    http://www.mail-archive.com/[email protected]/msg03217.html

  • Need help with STUCK Threads.  What's causing them?

    In our production environment, we have a soa suite cluster with 3 nodes. All 3 of the servers keep turning up stuck and hogging threads, and eventually they all grind to a halt where I can't even view composite instances through em. There is a bpel process that is using a dbadapter to poll for records every 5 seconds and then performs some service call outs and other dbadapters. This process is generating thousands of instances a day and I think it is the culprit. If I view the logs, I see one of the dbadapters that it calls out to fails occasionally with a PK violation. I looked at the developer's code, and they have no error handling in the process at all. In the logs, there will the a hit for the PK violation, then about 10 minutes later the a STUCK thread gets logged(the configured timeout is 600 seconds, or 10 minutes). Would not handling an error in a BPEL process hog/STUCK a thread? I need to be able to show what code is causing the issue so I can have a resolution path.
    I also see a note in the trace about something holding a lock:
    Holding lock: oracle.jdbc.driver.T4CConnection@1fbc3486[thin lock]....would that be an un-rolled back transaction?
    Here is the dump of the STUCK thread. Any help is greatly appreciated. Thanks!
    <Aug 5, 2011 10:30:02 AM EDT> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '64' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "634" seconds working on the request "oracle.soa.management.internal.ejb.impl.FacadeFinderBean_4vacyo_FacadeFinderBeanImpl", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
    Thread-517 "[STUCK] ExecuteThread: '64' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, priority=1, DAEMON> {
    jrockit.net.SocketNativeIO.readBytesPinned(SocketNativeIO.java:???)
    jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:24)
    java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)
    java.net.SocketInputStream.read(SocketInputStream.java:107)
    oracle.net.nt.MetricsEnabledInputStream.read(TcpNTAdapter.java:553)
    oracle.net.ns.Packet.receive(Packet.java:249)
    oracle.net.ns.DataPacket.receive(DataPacket.java:94)
    oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:168)
    oracle.net.ns.NetInputStream.read(NetInputStream.java:113)
    oracle.net.ns.NetInputStream.read(NetInputStream.java:96)
    oracle.net.ns.NetInputStream.read(NetInputStream.java:81)
    oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:120)
    oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:75)
    oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1081)
    oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1062)
    oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:588)
    oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:70)
    oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:854)
    oracle.jdbc.driver.T4CPreparedStatement.executeMaybeDescribe(T4CPreparedStatement.java:918)
    oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1252)
    oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3541)
    oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3605)
    ^-- Holding lock: oracle.jdbc.driver.T4CConnection@1fbc3486[thin lock]
    oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1667)
    weblogic.jdbc.wrapper.PreparedStatement.executeQuery(PreparedStatement.java:123)
    org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeSelect(DatabaseAccessor.java:887)
    org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:541)
    org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:530)
    org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:521)
    org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:202)
    org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:192)
    org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelectCall(DatasourceCallQueryMechanism.java:254)
    org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelect(DatasourceCallQueryMechanism.java:245)
    org.eclipse.persistence.queries.DataReadQuery.executeNonCursor(DataReadQuery.java:188)
    org.eclipse.persistence.queries.DataReadQuery.executeDatabaseQuery(DataReadQuery.java:141)
    org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:603)
    org.eclipse.persistence.queries.DataReadQuery.execute(DataReadQuery.java:115)
    org.eclipse.persistence.queries.DatabaseQuery.executeInUnitOfWork(DatabaseQuery.java:589)
    org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2898)
    org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1218)
    org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1190)
    org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1176)
    org.eclipse.persistence.internal.jpa.EJBQueryImpl.executeReadQuery(EJBQueryImpl.java:378)
    org.eclipse.persistence.internal.jpa.EJBQueryImpl.getSingleResult(EJBQueryImpl.java:701)
    weblogic.deployment.TransactionalEntityManagerProxyImpl$QueryImpl.getSingleResult(TransactionalEntityManagerProxyImpl.java:232)
    com.collaxa.cube.persistence.dao.impl.jpa.JpaCubeInstanceDaoImpl.getNumberOfInstances(JpaCubeInstanceDaoImpl.java:439)
    oracle.fabric.CubeServiceEngine.getNumberOfComponentInstances(CubeServiceEngine.java:1962)
    oracle.soa.management.internal.ejb.impl.FacadeFinderBeanImpl.getNumberOfComponentInstances(FacadeFinderBeanImpl.java:1618)
    oracle.soa.management.internal.ejb.impl.FacadeFinderBeanImpl.getComponentSnapShot(FacadeFinderBeanImpl.java:1653)
    sun.reflect.GeneratedMethodAccessor5852.invoke(Unknown Source)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    java.lang.reflect.Method.invoke(Method.java:575)
    com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.jee.intercept.MethodInvocationInvocationContext.proceed(MethodInvocationInvocationContext.java:104)
    oracle.security.jps.ee.ejb.JpsAbsInterceptor$1.run(JpsAbsInterceptor.java:88)
    oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:299)
    oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:406)
    oracle.security.jps.wls.JpsWeblogicEjbInterceptor.runJaasMode(JpsWeblogicEjbInterceptor.java:61)
    oracle.security.jps.ee.ejb.JpsAbsInterceptor.intercept(JpsAbsInterceptor.java:81)
    oracle.security.jps.ee.ejb.JpsInterceptor.intercept(JpsInterceptor.java:101)
    sun.reflect.GeneratedMethodAccessor863.invoke(Unknown Source)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    java.lang.reflect.Method.invoke(Method.java:575)
    com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
    com.bea.core.repackaged.springframework.jee.intercept.JeeInterceptorInterceptor.invoke(JeeInterceptorInterceptor.java:66)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
    com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:102)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.jee.spi.MethodInvocationVisitorImpl.visit(MethodInvocationVisitorImpl.java:37)
    weblogic.ejb.container.injection.EnvironmentInterceptorCallbackImpl.callback(EnvironmentInterceptorCallbackImpl.java:22)
    com.bea.core.repackaged.springframework.jee.spi.EnvironmentInterceptor.invoke(EnvironmentInterceptor.java:49)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:86)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
    com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:102)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:151)
    $Proxy309.getComponentSnapShot(Unknown Source)
    oracle.soa.management.internal.ejb.impl.FacadeFinderBean_4vacyo_FacadeFinderBeanImpl.getComponentSnapShot(FacadeFinderBean_4vacyo_FacadeFinderBeanImpl.java:1604)
    oracle.soa.management.internal.ejb.impl.FacadeFinderBean_4vacyo_FacadeFinderBeanImpl_WLSkel.invoke(Unknown Source)
    weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:567)
    weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:477)
    weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:350)
    weblogic.security.service.SecurityManager.runAs(SecurityManager.java:143)
    weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:425)
    weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:117)
    weblogic.work.ExecuteThread.execute(ExecuteThread.java:198)
    weblogic.work.ExecuteThread.run(ExecuteThread.java:165)
    >
    <Aug 5, 2011 10:30:02 AM EDT> <Notice> <Diagnostics> <BEA-320068> <Watch 'StuckThread' with severity 'Notice' on server 'soa_server1' has triggered at Aug 5, 2011 10:30:02 AM EDT. Notification details:
    WatchRuleType: Log
    WatchRule: (SEVERITY = 'Error') AND (MSGID = 'BEA-000337')
    WatchData: DATE = Aug 5, 2011 10:30:02 AM EDT SERVER = soa_server1 MESSAGE = [STUCK] ExecuteThread: '64' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "634" seconds working on the request "oracle.soa.management.internal.ejb.impl.FacadeFinderBean_4vacyo_FacadeFinderBeanImpl", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
    Thread-517 "[STUCK] ExecuteThread: '64' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, priority=1, DAEMON> {
    jrockit.net.SocketNativeIO.readBytesPinned(SocketNativeIO.java:???)
    jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:24)
    java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)
    java.net.SocketInputStream.read(SocketInputStream.java:107)
    oracle.net.nt.MetricsEnabledInputStream.read(TcpNTAdapter.java:553)
    oracle.net.ns.Packet.receive(Packet.java:249)
    oracle.net.ns.DataPacket.receive(DataPacket.java:94)
    oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:168)
    oracle.net.ns.NetInputStream.read(NetInputStream.java:113)
    oracle.net.ns.NetInputStream.read(NetInputStream.java:96)
    oracle.net.ns.NetInputStream.read(NetInputStream.java:81)
    oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:120)
    oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:75)
    oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1081)
    oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1062)
    oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:588)
    oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:70)
    oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:854)
    oracle.jdbc.driver.T4CPreparedStatement.executeMaybeDescribe(T4CPreparedStatement.java:918)
    oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1252)
    oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3541)
    oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3605)
    ^-- Holding lock: oracle.jdbc.driver.T4CConnection@1fbc3486[thin lock]
    oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1667)
    weblogic.jdbc.wrapper.PreparedStatement.executeQuery(PreparedStatement.java:123)
    org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeSelect(DatabaseAccessor.java:887)
    org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:541)
    org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:530)
    org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:521)
    org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:202)
    org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:192)
    org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelectCall(DatasourceCallQueryMechanism.java:254)
    org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelect(DatasourceCallQueryMechanism.java:245)
    org.eclipse.persistence.queries.DataReadQuery.executeNonCursor(DataReadQuery.java:188)
    org.eclipse.persistence.queries.DataReadQuery.executeDatabaseQuery(DataReadQuery.java:141)
    org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:603)
    org.eclipse.persistence.queries.DataReadQuery.execute(DataReadQuery.java:115)
    org.eclipse.persistence.queries.DatabaseQuery.executeInUnitOfWork(DatabaseQuery.java:589)
    org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2898)
    org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1218)
    org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1190)
    org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1176)
    org.eclipse.persistence.internal.jpa.EJBQueryImpl.executeReadQuery(EJBQueryImpl.java:378)
    org.eclipse.persistence.internal.jpa.EJBQueryImpl.getSingleResult(EJBQueryImpl.java:701)
    weblogic.deployment.TransactionalEntityManagerProxyImpl$QueryImpl.getSingleResult(TransactionalEntityManagerProxyImpl.java:232)
    com.collaxa.cube.persistence.dao.impl.jpa.JpaCubeInstanceDaoImpl.getNumberOfInstances(JpaCubeInstanceDaoImpl.java:439)
    oracle.fabric.CubeServiceEngine.getNumberOfComponentInstances(CubeServiceEngine.java:1962)
    oracle.soa.management.internal.ejb.impl.FacadeFinderBeanImpl.getNumberOfComponentInstances(FacadeFinderBeanImpl.java:1618)
    oracle.soa.management.internal.ejb.impl.FacadeFinderBeanImpl.getComponentSnapShot(FacadeFinderBeanImpl.java:1653)
    sun.reflect.GeneratedMethodAccessor5852.invoke(Unknown Source)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    java.lang.reflect.Method.invoke(Method.java:575)
    com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.jee.intercept.MethodInvocationInvocationContext.proceed(MethodInvocationInvocationContext.java:104)
    oracle.security.jps.ee.ejb.JpsAbsInterceptor$1.run(JpsAbsInterceptor.java:88)
    oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:299)
    oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:406)
    oracle.security.jps.wls.JpsWeblogicEjbInterceptor.runJaasMode(JpsWeblogicEjbInterceptor.java:61)
    oracle.security.jps.ee.ejb.JpsAbsInterceptor.intercept(JpsAbsInterceptor.java:81)
    oracle.security.jps.ee.ejb.JpsInterceptor.intercept(JpsInterceptor.java:101)
    sun.reflect.GeneratedMethodAccessor863.invoke(Unknown Source)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    java.lang.reflect.Method.invoke(Method.java:575)
    com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
    com.bea.core.repackaged.springframework.jee.intercept.JeeInterceptorInterceptor.invoke(JeeInterceptorInterceptor.java:66)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
    com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:102)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.jee.spi.MethodInvocationVisitorImpl.visit(MethodInvocationVisitorImpl.java:37)
    weblogic.ejb.container.injection.EnvironmentInterceptorCallbackImpl.callback(EnvironmentInterceptorCallbackImpl.java:22)
    com.bea.core.repackaged.springframework.jee.spi.EnvironmentInterceptor.invoke(EnvironmentInterceptor.java:49)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:86)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
    com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:102)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:151)
    $Proxy309.getComponentSnapShot(Unknown Source)
    oracle.soa.management.internal.ejb.impl.FacadeFinderBean_4vacyo_FacadeFinderBeanImpl.getComponentSnapShot(FacadeFinderBean_4vacyo_FacadeFinderBeanImpl.java:1604)
    oracle.soa.management.internal.ejb.impl.FacadeFinderBean_4vacyo_FacadeFinderBeanImpl_WLSkel.invoke(Unknown Source)
    weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:567)
    weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:477)
    weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:350)
    weblogic.security.service.SecurityManager.runAs(SecurityManager.java:143)
    weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:425)
    weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:117)
    weblogic.work.ExecuteThread.execute(ExecuteThread.java:198)
    weblogic.work.ExecuteThread.run(ExecuteThread.java:165)
    SUBSYSTEM = WebLogicServer USERID = <WLS Kernel> SEVERITY = Error THREAD = [ACTIVE] ExecuteThread: '96' for queue: 'weblogic.kernel.Default (self-tuning)' MSGID = BEA-000337 MACHINE = soaprod TXID = CONTEXTID = f5d0b85a929630a2:6eada0f7:1304e0a2bdc:-7ffb-00000000017e1605 TIMESTAMP = 1312554602698
    WatchAlarmType: AutomaticReset
    WatchAlarmResetPeriod: 600000
    >
    Edited by: Brutus35 on Aug 5, 2011 8:04 AM

    my impression is that you have trouble on the DB,
    monitor the sessions to see if there is some table locked, and increase the number of processes on the DB
    you can use SQLDeveloper for that
    anyway your TX should rollback after some time, check that you have a timeout in place, it's better to rollback a TX than having the entire server crash

  • Page stuck while load and leads to stuck thread

    Hi
    I am facing an issue from a long time.
    When i run the page of my application, when it opens for first time it get stuck and get loading and take very much time and after that it leads to stuck thread issue. All the component of my page opens but there is a table which gets value from OIM APIs also loads for the fiirst time. As in coding i have done it will pass null value to the table and wont fetch data for the first time. it will only fetch when value will be passed in a text box and submit was hit.
    So my page gets hanged or loading in the table section everytime when it loads. for that i have to login in weblogic server, run the workmanager, forcefully shut down the server instance and run the page again. But sometime that doesnt work too.
    Please help its a bug trouble for me and creating a huge problem.
    Thanks
    Sudeep

    I'm not sure I understand your issue - but if you want to delay the fetching of information into the table until you actually have a value to pass to it see:
    http://blogs.oracle.com/shay/2010/08/delay_method_execution_when_us.html

  • Help on stuck thread

    Hi, From my server log I noticed that there are some stuck threads. After looked at the thread dump thread 18 is blocking. MainPortalServlet.java is extends PortalServlet and line 96 is super.service(request,response); Please help me to analyse why it is blocking. Thanks.
    "ExecuteThread: '20' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x0090e6b0 nid=0x3b waiting for monitor entry [0x600fd000..0x600ffc28]
         at com.bea.wlw.netui.script.el.util.ParseUtils.parse(ParseUtils.java:100)
         - waiting to lock <0x8d3a3f18> (a com.bea.wlw.netui.script.el.util.ParseUtils$Cache)
         at com.bea.wlw.netui.script.el.ExpressionEvaluatorImpl.containsExpression(ExpressionEvaluatorImpl.java:239)
         at com.bea.wlw.netui.pageflow.ProcessPopulate.populate(ProcessPopulate.java:295)
         at com.bea.wlw.netui.pageflow.PageFlowRequestProcessor.processPopulate(PageFlowRequestProcessor.java:402)
         at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:254)
         at com.bea.wlw.netui.pageflow.PageFlowRequestProcessor.process(PageFlowRequestProcessor.java:691)
         at com.bea.wlw.netui.pageflow.AutoRegisterActionServlet.process(AutoRegisterActionServlet.java:527)
         at com.bea.wlw.netui.pageflow.PageFlowActionServlet.process(PageFlowActionServlet.java:152)
         at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:507)
         at com.bea.wlw.netui.pageflow.PageFlowUtils.strutsLookup(PageFlowUtils.java:1786)
         at com.bea.portlet.adapter.scopedcontent.ScopedContentCommonSupport.executeAction(ScopedContentCommonSupport.java:561)
         at com.bea.portlet.adapter.scopedcontent.ScopedContentCommonSupport.processActionInternal(ScopedContentCommonSupport.java:121)
         at com.bea.portlet.adapter.scopedcontent.PageFlowStubImpl.processAction(PageFlowStubImpl.java:98)
         at com.bea.netuix.servlets.controls.content.NetuiContent.raiseScopedAction(NetuiContent.java:154)
         at com.bea.netuix.servlets.controls.content.NetuiContent.raiseScopedAction(NetuiContent.java:113)
         at com.bea.netuix.servlets.controls.content.NetuiContent.handlePostbackData(NetuiContent.java:228)
         at com.bea.netuix.nf.ControlLifecycle$3.visit(ControlLifecycle.java:171)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:356)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:126)
         at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:105)
         at com.bea.netuix.nf.Lifecycle.runInbound(Lifecycle.java:173)
         at com.bea.netuix.nf.Lifecycle.run(Lifecycle.java:137)
         at com.bea.netuix.servlets.manager.UIServlet.runLifecycle(UIServlet.java:324)
         at com.bea.netuix.servlets.manager.UIServlet.processControlTree(UIServlet.java:220)
         at com.bea.netuix.servlets.manager.PortalServlet.doPost(PortalServlet.java:820)
         at com.bea.netuix.servlets.manager.UIServlet.service(UIServlet.java:150)
         at com.cd.servlets.portal.MainPortalServlet.service(MainPortalServlet.java:96)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
         at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
         at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:28)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
         at com.bea.p13n.servlets.PortalServletFilter.doFilter(PortalServletFilter.java:293)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
         at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:7053)
         at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
         at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
         at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3902)
         at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2773)
         at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)
    "ExecuteThread: '19' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x0090dab8 nid=0x3a runnable [0x601fb000..0x601ffc28]
         at java.util.HashMap.getEntry(HashMap.java:361)
         at java.util.LinkedHashMap.get(LinkedHashMap.java:266)
         at com.bea.wlw.netui.script.el.util.ParseUtils.parse(ParseUtils.java:84)
         at com.bea.wlw.netui.script.el.util.ParseUtils.evaluate(ParseUtils.java:123)
         at com.bea.wlw.netui.script.el.ExpressionEvaluatorImpl.evaluateStrict(ExpressionEvaluatorImpl.java:87)
         at com.bea.wlw.netui.tags.AbstractBaseTag.evaluateExpressionInternal(AbstractBaseTag.java:573)
         at com.bea.wlw.netui.tags.AbstractBaseTag.evaluateExpression(AbstractBaseTag.java:362)
         at com.bea.wlw.netui.tags.databinding.script.GetData.doEndTag(GetData.java:249)
         at jsp_servlet._portlets.___portlet1._jspService(_portlet1.jsp:27)
         at weblogic.servlet.jsp.JspBase.service(JspBase.java:33)
         at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
         at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:28)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
         at com.bea.p13n.servlets.PortalServletFilter.doFilter(PortalServletFilter.java:293)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
         at com.bea.wlw.netui.pageflow.PageFlowJspFilter.doFilter(PageFlowJspFilter.java:250)
         - locked <0x78ccf6c8> (a portlets.Portlet1Controller)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
         at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:652)
         at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:431)
         at com.bea.wlw.netui.pageflow.scoping.internal.ScopedRequestDispatcher.include(ScopedRequestDispatcher.java:120)
         at com.bea.netuix.servlets.controls.content.JspContent.beginRender(JspContent.java:534)
         at com.bea.netuix.servlets.controls.content.NetuiContent.beginRender(NetuiContent.java:425)
         at com.bea.netuix.nf.ControlLifecycle$1.visit(ControlLifecycle.java:495)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:543)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:247)
         at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:105)
         at com.bea.netuix.nf.Lifecycle.run(Lifecycle.java:356)
         at com.bea.netuix.nf.UIControl.render(UIControl.java:536)
         at com.bea.netuix.servlets.controls.PresentationContext.render(PresentationContext.java:405)
         at com.bea.netuix.servlets.util.RenderToolkit.renderChild(RenderToolkit.java:123)
         at com.bea.netuix.servlets.jsp.taglib.RenderChild.doStartTag(RenderChild.java:58)
         at jsp_servlet._framework._skeletons._cd.__flowlayout._jspService(flowlayout.jsp:30)
         at weblogic.servlet.jsp.JspBase.service(JspBase.java:33)
         at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:348)
         at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:646)
         at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:431)
         at com.bea.netuix.servlets.controls.JspRenderer.renderAlt(JspRenderer.java:194)
         at com.bea.netuix.servlets.controls.JspRenderer.beginRender(JspRenderer.java:96)
         at com.bea.netuix.nf.ControlLifecycle$1.visit(ControlLifecycle.java:491)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:543)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:247)
         at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:105)
         at com.bea.netuix.nf.Lifecycle.run(Lifecycle.java:356)
         at com.bea.netuix.nf.UIControl.render(UIControl.java:536)
         at com.bea.netuix.servlets.controls.PresentationContext.render(PresentationContext.java:405)
         at com.bea.netuix.servlets.util.RenderToolkit.renderChild(RenderToolkit.java:123)
         at com.bea.netuix.servlets.jsp.taglib.RenderChild.doStartTag(RenderChild.java:58)
         at jsp_servlet._framework._skeletons._cd.__flowlayout._jspService(flowlayout.jsp:30)
         at weblogic.servlet.jsp.JspBase.service(JspBase.java:33)
         at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:348)
         at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:646)
         at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:431)
         at com.bea.netuix.servlets.controls.JspRenderer.renderAlt(JspRenderer.java:194)
         at com.bea.netuix.servlets.controls.JspRenderer.beginRender(JspRenderer.java:96)
         at com.bea.netuix.nf.ControlLifecycle$1.visit(ControlLifecycle.java:491)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:543)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:247)
         at com.bea.netuix.nf.Lifecycle.runOutbound(Lifecycle.java:204)
         at com.bea.netuix.nf.Lifecycle.run(Lifecycle.java:153)
         at com.bea.netuix.servlets.manager.UIServlet.runLifecycle(UIServlet.java:324)
         at com.bea.netuix.servlets.manager.UIServlet.processControlTree(UIServlet.java:220)
         at com.bea.netuix.servlets.manager.PortalServlet.doPost(PortalServlet.java:820)
         at com.bea.netuix.servlets.manager.PortalServlet.doGet(PortalServlet.java:671)
         at com.bea.netuix.servlets.manager.UIServlet.service(UIServlet.java:147)
         at com.cd.servlets.portal.MainPortalServlet.service(MainPortalServlet.java:116)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
         at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
         at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:28)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
         at com.bea.p13n.servlets.PortalServletFilter.doFilter(PortalServletFilter.java:293)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
         at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:7053)
         at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
         at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
         at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3902)
         at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2773)
         at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)
    "ExecuteThread: '18' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x0090cec0 nid=0x39 runnable [0x602fd000..0x602ffc28]
         at java.util.LinkedHashMap.transfer(LinkedHashMap.java:224)
         at java.util.HashMap.resize(HashMap.java:452)
         at java.util.LinkedHashMap.addEntry(LinkedHashMap.java:399)
         at java.util.HashMap.put(HashMap.java:392)
         at com.bea.wlw.netui.script.el.util.ParseUtils.parse(ParseUtils.java:100)
         - locked <0x8d3a3f18> (a com.bea.wlw.netui.script.el.util.ParseUtils$Cache)
         at com.bea.wlw.netui.script.el.ExpressionEvaluatorImpl.containsExpression(ExpressionEvaluatorImpl.java:239)
         at com.bea.wlw.netui.pageflow.ProcessPopulate.populate(ProcessPopulate.java:295)
         at com.bea.wlw.netui.pageflow.PageFlowRequestProcessor.processPopulate(PageFlowRequestProcessor.java:402)
         at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:254)
         at com.bea.wlw.netui.pageflow.PageFlowRequestProcessor.process(PageFlowRequestProcessor.java:691)
         at com.bea.wlw.netui.pageflow.AutoRegisterActionServlet.process(AutoRegisterActionServlet.java:527)
         at com.bea.wlw.netui.pageflow.PageFlowActionServlet.process(PageFlowActionServlet.java:152)
         at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:507)
         at com.bea.wlw.netui.pageflow.PageFlowUtils.strutsLookup(PageFlowUtils.java:1786)
         at com.bea.portlet.adapter.scopedcontent.ScopedContentCommonSupport.executeAction(ScopedContentCommonSupport.java:561)
         at com.bea.portlet.adapter.scopedcontent.ScopedContentCommonSupport.processActionInternal(ScopedContentCommonSupport.java:121)
         at com.bea.portlet.adapter.scopedcontent.PageFlowStubImpl.processAction(PageFlowStubImpl.java:98)
         at com.bea.netuix.servlets.controls.content.NetuiContent.raiseScopedAction(NetuiContent.java:154)
         at com.bea.netuix.servlets.controls.content.NetuiContent.raiseScopedAction(NetuiContent.java:113)
         at com.bea.netuix.servlets.controls.content.NetuiContent.handlePostbackData(NetuiContent.java:228)
         at com.bea.netuix.nf.ControlLifecycle$3.visit(ControlLifecycle.java:171)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:356)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:126)
         at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:105)
         at com.bea.netuix.nf.Lifecycle.runInbound(Lifecycle.java:173)
         at com.bea.netuix.nf.Lifecycle.run(Lifecycle.java:137)
         at com.bea.netuix.servlets.manager.UIServlet.runLifecycle(UIServlet.java:324)
         at com.bea.netuix.servlets.manager.UIServlet.processControlTree(UIServlet.java:220)
         at com.bea.netuix.servlets.manager.PortalServlet.doPost(PortalServlet.java:820)
         at com.bea.netuix.servlets.manager.UIServlet.service(UIServlet.java:150)
         at com.cd.servlets.portal.MainPortalServlet.service(MainPortalServlet.java:96)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
         at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
         at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:28)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
         at com.bea.p13n.servlets.PortalServletFilter.doFilter(PortalServletFilter.java:293)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
         at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:7053)
         at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
         at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
         at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3902)
         at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2773)
         at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)

    Hi user9503787,
    Can you please confirm if the Weblogic instance / java process you took the Thread Dump from was showing high CPU utilization?
    Your Thread Dump data is quite interesting and a perfect example of a non Thread safe HashMap combined with a read/write race condition:
    Thread #23 (hanging for 600 seconds+) is showing a potential infinite looping condition in at java.util.HashMap.getEntry(HashMap.java:361)
    Thread #22 (hanging for 600 seconds+) is showing a potential infinite looping condition while performing a resize() following a put() operation
    Weblogic Portal has some known issues which regards to usage of non Thread safe HashHap data structures; which under load can cause infinite looping and sharp surge of CPU (due to corruption of the internal HashMap index preventing exiting the internal loop); ultimately hanging the whole JVM.
    We had the same issue for one of our Weblogic Portal 10.0 production environment and Oracle provided a patch (via Collections.synchronizedMap() ).
    Please open a SR and Oracle, share these 2 Threads and discuss with the engineer about a patch / know issues regarding non Thread safe HashMap and infinite looping.
    Regards,
    P-H
    http://javaeesupportpatterns.blogspot.com/

  • Stuck threads reading blob column from db table

    WLS 10.3.5, JDK 1.6u29, Oracle 11g RAC, ojdbc6 latest driver
    We're having problems with stuck threads trying to read a blob column from a DB table. The query to extract the blob is a simple select, without any locking such as "for update" clauses or whatever. The blob's size is <= 100k.
    The thread dump shows the following stack trace:
    +"[STUCK] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" RUNNABLE native+
    +     java.net.SocketInputStream.socketRead0(Native Method)+
    +     java.net.SocketInputStream.read(SocketInputStream.java:129)+
    +     oracle.net.ns.Packet.receive(Packet.java:300)+
    +     oracle.net.ns.DataPacket.receive(DataPacket.java:106)+
    +     oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:315)+
    +     oracle.net.ns.NetInputStream.read(NetInputStream.java:260)+
    +     oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:105)+
    +     oracle.jdbc.driver.T4CMAREngine.getNBytes(T4CMAREngine.java:1517)+
    +     oracle.jdbc.driver.T4C8TTILobd.unmarshalLobData(T4C8TTILobd.java:476)+
    +     oracle.jdbc.driver.T4C8TTILob.readLOBD(T4C8TTILob.java:770)+
    +     oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:361)+
    +     oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)+
    +     oracle.jdbc.driver.T4C8TTILob.read(T4C8TTILob.java:146)+
    +     oracle.jdbc.driver.T4CConnection.getBytes(T4CConnection.java:2392)+
    +     oracle.sql.BLOB.getBytes(BLOB.java:348)+
    +     oracle.sql.BLOB.getBytes(BLOB.java:222)+
    +     weblogic.jdbc.wrapper.Blob_oracle_sql_BLOB.getBytes(Unknown Source)+
    +     com.ibatis.sqlmap.engine.type.BlobTypeHandlerCallback.getResult(BlobTypeHandlerCallback.java:33)+
    +     com.ibatis.sqlmap.engine.type.CustomTypeHandler.getResult(CustomTypeHandler.java:52)+
    +     com.ibatis.sqlmap.engine.mapping.result.ResultMap.getPrimitiveResultMappingValue(ResultMap.java:619)+
    +     com.ibatis.sqlmap.engine.mapping.result.ResultMap.getResults(ResultMap.java:345)+
    +     com.ibatis.sqlmap.engine.execution.SqlExecutor.handleResults(SqlExecutor.java:384)+
    +     com.ibatis.sqlmap.engine.execution.SqlExecutor.handleMultipleResults(SqlExecutor.java:300)+
    +     com.ibatis.sqlmap.engine.execution.SqlExecutor.executeQuery(SqlExecutor.java:189)+
    +     com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.sqlExecuteQuery(MappedStatement.java:221)+
    +     com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.executeQueryWithCallback(MappedStatement.java:189)+
    +     com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.executeQueryForList(MappedStatement.java:139)+
    +     com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryForList(SqlMapExecutorDelegate.java:567)+
    +     com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryForList(SqlMapExecutorDelegate.java:541)+
    +     com.ibatis.sqlmap.engine.impl.SqlMapSessionImpl.queryForList(SqlMapSessionImpl.java:118)+
    +     com.ibatis.sqlmap.engine.impl.SqlMapClientImpl.queryForList(SqlMapClientImpl.java:94)+
    +     com.ibatis.dao.client.template.SqlMapDaoTemplate.queryForList(SqlMapDaoTemplate.java:282)"[STUCK] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" RUNNABLE native+
    +     java.net.SocketInputStream.socketRead0(Native Method)+
    +     java.net.SocketInputStream.read(SocketInputStream.java:129)+
    +     oracle.net.ns.Packet.receive(Packet.java:300)+
    +     oracle.net.ns.DataPacket.receive(DataPacket.java:106)+
    +     oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:315)+
    +     oracle.net.ns.NetInputStream.read(NetInputStream.java:260)+
    +     oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:105)+
    +     oracle.jdbc.driver.T4CMAREngine.getNBytes(T4CMAREngine.java:1517)+
    +     oracle.jdbc.driver.T4C8TTILobd.unmarshalLobData(T4C8TTILobd.java:476)+
    +     oracle.jdbc.driver.T4C8TTILob.readLOBD(T4C8TTILob.java:770)+
    +     oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:361)+
    +     oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)+
    +     oracle.jdbc.driver.T4C8TTILob.read(T4C8TTILob.java:146)+
    +     oracle.jdbc.driver.T4CConnection.getBytes(T4CConnection.java:2392)+
    +     oracle.sql.BLOB.getBytes(BLOB.java:348)+
    +     oracle.sql.BLOB.getBytes(BLOB.java:222)+
    +     weblogic.jdbc.wrapper.Blob_oracle_sql_BLOB.getBytes(Unknown Source)+
    +     com.ibatis.sqlmap.engine.type.BlobTypeHandlerCallback.getResult(BlobTypeHandlerCallback.java:33)+
    +     com.ibatis.sqlmap.engine.type.CustomTypeHandler.getResult(CustomTypeHandler.java:52)+
    +     com.ibatis.sqlmap.engine.mapping.result.ResultMap.getPrimitiveResultMappingValue(ResultMap.java:619)+
    +     com.ibatis.sqlmap.engine.mapping.result.ResultMap.getResults(ResultMap.java:345)+
    +     com.ibatis.sqlmap.engine.execution.SqlExecutor.handleResults(SqlExecutor.java:384)+
    +     com.ibatis.sqlmap.engine.execution.SqlExecutor.handleMultipleResults(SqlExecutor.java:300)+
    +     com.ibatis.sqlmap.engine.execution.SqlExecutor.executeQuery(SqlExecutor.java:189)+
    +     com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.sqlExecuteQuery(MappedStatement.java:221)+
         com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.executeQueryWithCallback(MappedStatement.java:189)
    +     com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.executeQueryForList(MappedStatement.java:139)+
    +     com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryForList(SqlMapExecutorDelegate.java:567)+
    +     com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryForList(SqlMapExecutorDelegate.java:541)+
    +     com.ibatis.sqlmap.engine.impl.SqlMapSessionImpl.queryForList(SqlMapSessionImpl.java:118)+
    +     com.ibatis.sqlmap.engine.impl.SqlMapClientImpl.queryForList(SqlMapClientImpl.java:94)+
    +     com.ibatis.dao.client.template.SqlMapDaoTemplate.queryForList(SqlMapDaoTemplate.java:282)+
    Some threads eventually end (after 1-2 hours), most of them remain there for days.
    Any hint would be quite useful, thanks.

    Threads are executing the actual allocated request from the Weblogic Kernel. Most of the problems happen when the Thread execution is reaching the application or business layer.
    At this point your application Java code module is sending or receiving data from external sources such as a an Oracle database for example. Any problem with such external system will cause the Thread to hang and wait for data to come back.
    Other situations can occur such as internal deadlock, infinite looping, heavy IO contention on your server etc.
    Doesn't loo like a driver issue.
    http://docs.oracle.com/cd/E21764_01/doc.1111/e14770/weblogic_server_issues.htm#autoId2
    Check at the Database end
    Cheers ...

  • Stuck Thread causing heap to grow steeply and resulting in crash

    Hi,
    We have an ADF application deployed in weblogic 10.3.2.
    We have few PL/SQL's and SQL's which are complex and take around 5 minutes of time to complete at times. We also have the script running in database to kill long running queries so that it helps us to get rid of weblogic stuck threads to some extent.
    Now when ever there is an SQL running for long time , we see a stuck thread casuing heap to grow very fast and casuing server to crash.
    I know after a particular point of idle time thread is marked as stuck , but why does it filling the heap space - does stuck threads always causes heap to grow or is it just my specific case
    any pointers on how to debug this issue to find the root cause.
    Thanks,
    Naresh

    I think this setup needs profiling. To start-off, i would recommend collecting heap dump or jrf (if jrockit), instructions for them are available on net, as it varies with jvm versions and vendors.
    Post that you should look for the thread which you believe is stuck and a possible cause using OQL in memory analyzer tool over the heap dumps.
    Conceptually, till the time the objects (or data if you say) is being referenced, JVM cannot garbage collect it. So, I would say we need to figure out what are the contents of the heap, before coming to a conclusion that the threads which are stuck is causing the issue. Why I say so it that, I have seen in past that many-a-times, we suspect stuck threads to be an issue but it turns out to be that we are "knocking the wrong door".
    Long running threads (aka STUCK thread in WLS), will normally cause memory issues, if the references/task that its trying to perform has a large local data (i.e. data with the thread scope), so it the time the thread is not released the data is alive (irrespective of the fact that the data is used or not).
    Now, my first thought of this situation is that stuck thread is not an issue, considering that you are not able to replicate this in UAT post simulating stuck threads. Thus please collect some heap dumps (may be when you start observing memory increase) and then review them comparatively, I think that should be helpful.
    AJ

  • Oracle Service Bus "ThreadPool has stuck threads"

    Hi all,
    I have JMS Proxy service that listen to the queue, with small quantity of message every second, it works great.
    But after put message with bigger quantity and faster insertion on queue, the proxy service cannot
    pull the message anymore, the error message is "STUCK THREAD", it makes the weblogic console give warning status.
    This is the Dump Thread Stacks :
    ===== FULL THREAD DUMP ===============
    Tue Jun 8 11:30:24 2010
    BEA JRockit(R) R27.6.1-20-104573-1.6.0_05-20081016-1236-linux-ia32
    "Main Thread" id=1 idx=0x4 tid=7168 prio=5 alive, in native, waiting
    -- Waiting for notification on: weblogic/t3/srvr/T3Srvr@0x1cd20688[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at java/lang/Object.wait(Object.java:485)
    at weblogic/t3/srvr/T3Srvr.waitForDeath(T3Srvr.java:811)
    ^-- Lock released while waiting: weblogic/t3/srvr/T3Srvr@0x1cd20688[fat lock]
    at weblogic/t3/srvr/T3Srvr.run(T3Srvr.java:459)
    at weblogic/Server.main(Server.java:67)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
    "(Signal Handler)" id=2 idx=0x8 tid=7169 prio=5 alive, in native, daemon
    "(GC Main Thread)" id=3 idx=0xc tid=7170 prio=5 alive, in native, native_waiting, daemon
    "(GC Worker Thread 1)" id=? idx=0x10 tid=7171 prio=5 alive, in native, daemon
    "(GC Worker Thread 2)" id=? idx=0x14 tid=7172 prio=5 alive, in native, daemon
    "(GC Worker Thread 3)" id=? idx=0x18 tid=7173 prio=5 alive, in native, daemon
    "(GC Worker Thread 4)" id=? idx=0x1c tid=7174 prio=5 alive, in native, daemon
    "(GC Worker Thread 5)" id=? idx=0x20 tid=7175 prio=5 alive, in native, daemon
    "(GC Worker Thread 6)" id=? idx=0x24 tid=7176 prio=5 alive, in native, daemon
    "(GC Worker Thread 7)" id=? idx=0x28 tid=7177 prio=5 alive, in native, daemon
    "(GC Worker Thread 8)" id=? idx=0x2c tid=7178 prio=5 alive, in native, daemon
    "(Code Generation Thread 1)" id=4 idx=0x30 tid=7179 prio=5 alive, in native, native_waiting, daemon
    "(Code Optimization Thread 1)" id=5 idx=0x34 tid=7180 prio=5 alive, in native, native_waiting, daemon
    "(VM Periodic Task)" id=6 idx=0x38 tid=7181 prio=10 alive, in native, daemon
    Is there any tuning that I need to do on Jrockit?
    Really thanks in advance.
    Regards,
    Mr.K

    Hi Eric,
    Thank you for the link, I'll try for increasing it.
    Hi Manoj,
    hope it'll give you some additional information for helping me.
    ####<Jun 8, 2010 10:43:44 AM WIT> <Error> <WebLogicServer> <cspproduction> <AdminServerOSB> <[ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1275968624958> <BEA-000337> <[STUCK] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "653" seconds working on the request "weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl@39c79ac", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
    Thread-50 "[STUCK] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, waiting, priority=1, DAEMON> {
    -- Waiting for notification on: weblogic.rjvm.ResponseImpl@39cf5b1[fat lock]
    java.lang.Object.wait(Object.java:???)
    weblogic.rjvm.ResponseImpl.waitForData(ResponseImpl.java:80)
    weblogic.rjvm.ResponseImpl.getTxContext(ResponseImpl.java:127)
    weblogic.messaging.dispatcher.DispatcherProxy.unmarshalResponse(DispatcherProxy.java:236)
    weblogic.messaging.dispatcher.DispatcherProxy.dispatchSyncTranFuture(DispatcherProxy.java:133)
    weblogic.messaging.dispatcher.DispatcherWrapperState.dispatchSyncTran(DispatcherWrapperState.java:333)
    weblogic.messaging.dispatcher.DispatcherWrapperState.dispatchSyncNoTran(DispatcherWrapperState.java:384)
    weblogic.jms.dispatcher.DispatcherAdapter.dispatchSyncNoTran(DispatcherAdapter.java:59)
    weblogic.jms.client.JMSProducer.toFEProducer(JMSProducer.java:1151)
    weblogic.jms.client.JMSProducer.deliveryInternal(JMSProducer.java:679)
    weblogic.jms.client.JMSProducer.sendInternal(JMSProducer.java:541)
    weblogic.jms.client.JMSProducer.sendWithListener(JMSProducer.java:366)
    weblogic.jms.client.JMSProducer.send(JMSProducer.java:353)
    weblogic.jms.client.WLProducerImpl.send(WLProducerImpl.java:1050)
    weblogic.jms.client.WLProducerImpl.send(WLProducerImpl.java:107)
    weblogic.deployment.jms.WrappedMessageProducer.send(WrappedMessageProducer.java:235)
    com.bea.wli.sb.transports.jms.JmsOutboundMessageContext$SendAction.sendMessage(JmsOutboundMessageContext.java:823)
    com.bea.wli.sb.transports.jms.JmsOutboundMessageContext$SendAction.run(JmsOutboundMessageContext.java:745)
    weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:350)
    weblogic.security.service.SecurityManager.runAs(Unknown Source)
    weblogic.security.Security.runAs(Security.java:60)
    com.bea.wli.sb.transports.jms.JmsOutboundMessageContext.send(JmsOutboundMessageContext.java:502)
    com.bea.wli.sb.transports.jms.JmsTransportProvider.sendMessageAsync(JmsTransportProvider.java:680)
    sun.reflect.GeneratedMethodAccessor302.invoke(Unknown Source)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    java.lang.reflect.Method.invoke(Method.java:575)
    com.bea.wli.sb.transports.Util$1.invoke(Util.java:56)
    $Proxy61.sendMessageAsync(Unknown Source)
    com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageAsync(LoadBalanceFailoverListener.java:136)
    com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageToServiceAsync(LoadBalanceFailoverListener.java:510)
    com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageToService(LoadBalanceFailoverListener.java:477)
    com.bea.wli.sb.transports.TransportManagerImpl.sendMessageToService(TransportManagerImpl.java:491)
    com.bea.wli.sb.transports.TransportManagerImpl.sendMessageAsync(TransportManagerImpl.java:405)
    com.bea.wli.sb.pipeline.PipelineContextImpl.doDispatch(PipelineContextImpl.java:508)
    com.bea.wli.sb.pipeline.PipelineContextImpl.dispatch(PipelineContextImpl.java:486)
    stages.publish.runtime.PublishRuntimeStep.processMessage(PublishRuntimeStep.java:83)
    com.bea.wli.sb.stages.StageMetadataImpl$WrapperRuntimeStep.processMessage(StageMetadataImpl.java:343)
    com.bea.wli.sb.stages.impl.SequenceRuntimeStep.processMessage(SequenceRuntimeStep.java:31)
    com.bea.wli.sb.pipeline.PipelineStage.processMessage(PipelineStage.java:84)
    com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:840)
    com.bea.wli.sb.pipeline.Pipeline.processMessage(Pipeline.java:140)
    com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:840)
    com.bea.wli.sb.pipeline.PipelineNode.doRequest(PipelineNode.java:51)
    com.bea.wli.sb.pipeline.Node.processMessage(Node.java:65)
    com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:840)
    com.bea.wli.sb.pipeline.Router.processMessage(Router.java:178)
    com.bea.wli.sb.pipeline.MessageProcessor.processRequest(MessageProcessor.java:39)
    com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:508)
    com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:506)
    weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:350)
    weblogic.security.service.SecurityManager.runAs(Unknown Source)
    com.bea.wli.sb.security.WLSSecurityContextService.runAs(WLSSecurityContextService.java:55)
    com.bea.wli.sb.pipeline.RouterManager.processMessage(RouterManager.java:487)
    com.bea.wli.sb.transports.TransportManagerImpl.receiveMessage(TransportManagerImpl.java:248)
    com.bea.wli.sb.transports.jms.JmsInboundMDB.onMessage(JmsInboundMDB.java:79)
    sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:???)
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:27)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    java.lang.reflect.Method.invoke(Method.java:575)
    com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:277)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:187)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:153)
    com.bea.core.repackaged.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:86)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:153)
    com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:126)
    com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:100)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:153)
    com.bea.core.repackaged.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:148)
    $Proxy79.onMessage(Unknown Source)
    weblogic.ejb.container.internal.MDListener.execute(MDListener.java:403)
    weblogic.ejb.container.internal.MDListener.transactionalOnMessage(MDListener.java:352)
    weblogic.ejb.container.internal.MDListener.onMessage(MDListener.java:264)
    weblogic.jms.client.JMSSession.onMessage(JMSSession.java:4526)
    weblogic.jms.client.JMSSession.execute(JMSSession.java:4022)
    weblogic.jms.client.JMSSession.executeMessage(JMSSession.java:3708)
    weblogic.jms.client.JMSSession.access$000(JMSSession.java:114)
    weblogic.jms.client.JMSSession$UseForRunnable.run(JMSSession.java:5058)
    weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:516)
    weblogic.work.ExecuteThread.execute(ExecuteThread.java:198)
    weblogic.work.ExecuteThread.run(ExecuteThread.java:165)
    That is error message from Oracle service bus console, it stated there is JMS Proxy service that got stuck.
    On previous post, I already attached the dump trace, it explained something about jrockit. But I still don't get the way for tuning it properly.
    Regards,
    Mr.K

  • Stuck Thread - JMS Business proxy Service (ALSB 2.5)

    Hi All,
    We are having a jms proxy service which uses a JMS Business Service to send messages to a jms queue on weblogic 8.1 sp6. The alsb domain runs on weblogic 9.2.The jms proxy service is made single threaded by configuring a suitable work manager.
    Now and then, we get a stuck thread issue in production and the proxy service stops consuming the messaging off the jms queue in wls 9.2. These are the errors in the logs:
    <[ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1235437298516> <BEA-000337> <[STUCK] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "695" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@5321caf", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
    Thread-70 "[STUCK] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, waiting, priority=1, DAEMON> {
    -- Waiting for notification on: weblogic.rjvm.ResponseImpl@51961a5[fat lock]
    jrockit.vm.Threads.waitForSignal(Native Method)
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Native Method)
    weblogic.rjvm.ResponseImpl.waitForData(ResponseImpl.java:73)
    ^-- Lock released while waiting: weblogic.rjvm.ResponseImpl@51961a5[fat lock]
    weblogic.rjvm.ResponseImpl.getTxContext(ResponseImpl.java:100)
    weblogic.rjvm.BasicOutboundRequest.sendReceive(BasicOutboundRequest.java:109)
    weblogic.rmi.internal.BasicRemoteRef.invoke(BasicRemoteRef.java:223)
    weblogic.jms.dispatcher.DispatcherImpl_816_WLStub.dispatchSyncTranFuture(Unknown Source)
    weblogic.jms.dispatcher.DispatcherInteropAdapter.dispatchSyncTranFuture(DispatcherInteropAdapter.java:80)
    weblogic.messaging.dispatcher.DispatcherWrapperState.dispatchSyncTran(DispatcherWrapperState.java:230)
    weblogic.jms.dispatcher.DispatcherAdapter.dispatchSyncTran(DispatcherAdapter.java:51)
    weblogic.jms.client.JMSProducer.toFEProducer(JMSProducer.java:1047)
    weblogic.jms.client.JMSProducer.deliveryInternal(JMSProducer.java:670)
    weblogic.jms.client.JMSProducer.sendInternal(JMSProducer.java:455)
    weblogic.jms.client.JMSProducer.send(JMSProducer.java:352)
    weblogic.jms.client.WLProducerImpl.send(WLProducerImpl.java:1009)
    weblogic.deployment.jms.WrappedMessageProducer.send(WrappedMessageProducer.java:157)
    com.bea.wli.sb.transports.jms.JmsOutboundMessageContext$SendAction.run(JmsOutboundMessageContext.java:795)
    com.bea.wli.sb.transports.jms.JmsOutboundMessageContext.send(JmsOutboundMessageContext.java:500)
    com.bea.wli.sb.transports.jms.JmsTransportProvider.sendMessageAsync(JmsTransportProvider.java:612)
    jrockit.reflect.CompiledMethodInvoker.invoke0(Unknown Source)
    jrockit.reflect.CompiledMethodInvoker.invoke(Unknown Source)
    java.lang.reflect.Method.invoke(Unknown Source)
    com.bea.wli.sb.transports.Util$1.invoke(Util.java:241)
    $Proxy39.sendMessageAsync(Unknown Source)
    com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageAsync(LoadBalanceFailoverListener.java:137)
    com.bea.wli.sb.transports.TransportManagerImpl.sendMessageToService(TransportManagerImpl.java:437)
    com.bea.wli.sb.transports.TransportManagerImpl.sendMessageAsync(TransportManagerImpl.java:328)
    com.bea.wli.sb.pipeline.PipelineContextImpl.doDispatch(PipelineContextImpl.java:503)
    com.bea.wli.sb.pipeline.PipelineContextImpl.dispatch(PipelineContextImpl.java:415)
    stages.publish.runtime.PublishRuntimeStep.processMessage(PublishRuntimeStep.java:106)
    com.bea.wli.sb.stages.StageMetadataImpl$WrapperRuntimeStep.processMessage(StageMetadataImpl.java:327)
    com.bea.wli.sb.stages.impl.SequenceRuntimeStep.processMessage(SequenceRuntimeStep.java:33)
    com.bea.wli.sb.pipeline.PipelineStage.processMessage(PipelineStage.java:100)
    com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:771)
    com.bea.wli.sb.pipeline.Pipeline.processMessage(Pipeline.java:73)
    com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:771)
    com.bea.wli.sb.pipeline.PipelineNode.doRequest(PipelineNode.java:47)
    com.bea.wli.sb.pipeline.Node.processMessage(Node.java:57)
    com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:771)
    com.bea.wli.sb.pipeline.Router.processMessage(Router.java:141)
    com.bea.wli.sb.pipeline.MessageProcessor.processRequest(MessageProcessor.java:75)
    com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:646)
    com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:645)
    weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
    weblogic.security.service.SecurityManager.runAs(SecurityManager.java:147)
    com.bea.wli.sb.pipeline.RouterManager.processMessage(RouterManager.java:627)
    com.bea.wli.sb.transports.TransportManagerImpl.receiveMessage(TransportManagerImpl.java:264)
    com.bea.wli.sb.transports.jms.JmsInboundMDB.onMessage(JmsInboundMDB.java:108)
    weblogic.ejb.container.internal.MDListener.execute(MDListener.java:429)
    weblogic.ejb.container.internal.MDListener.transactionalOnMessage(MDListener.java:335)
    weblogic.ejb.container.internal.MDListener.onMessage(MDListener.java:292)
    weblogic.jms.client.JMSSession.onMessage(JMSSession.java:4060)
    weblogic.jms.client.JMSSession.execute(JMSSession.java:4012)
    weblogic.jms.client.JMSSession$UseForRunnable.run(JMSSession.java:4467)
    weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
    weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
    weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
    jrockit.vm.RNI.c2java(Native Method)
    Any one know the root cause of this issue or ways to prevent this?
    Also Is there a way to make the proxy service to reconsume messages off the queue without a server restart?
    Thanks,
    Atheek

    Hi atheek, did you find the solution??
    I already got the same/identic issue, do you mind giving me an aid?
    Really thanks
    Regards,
    Mr.K

  • Stuck thread in adf application with weblogic

    hi every body
    I am working in JDev 11.1.1.4 and weblogic 10.3.4
    when I deploy my Application in my production weblogic
    after some time (it does not constant it is variable) I got warnning on the server
    when I check the server I got some stuck thread
    any idea or notes how can I know what is the reason of these stuck threads?
    is there any objects have thread unsafe issue, resource contention or race condition in ADF?
    BR,
    Alaa

    actully I do that guys
    but what I get does not have any related to my application
    even those http request does not request the same page
    this is the stuck thread when I dump
    "[STUCK] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" id=61 idx=0xe8 tid=13369 prio=1 alive, waiting, native_blocked, daemon
    -- Waiting for notification on: oracle/adfinternal/controller/util/RequestLock@0x88e06b58[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
    at java/lang/Object.wait(Object.java:485)[inlined]
    at oracle/adfinternal/controller/util/RequestLock.lock(RequestLock.java:42)[inlined]
    at oracle/adfinternal/controller/state/RootViewPortContextImpl.lockViewPortRequestLock(RootViewPortContextImpl.java:604)[optimized]
    ^-- Lock released while waiting: oracle/adfinternal/controller/util/RequestLock@0x88e06b58[fat lock]
    at oracle/adfinternal/controller/state/ControllerState.initializeRequest(ControllerState.java:833)[inlined]
    at oracle/adfinternal/controller/state/ControllerState.initializeRequest(ControllerState.java:754)[inlined]
    at oracle/adfinternal/controller/application/AdfcConfigurator.beginRequest(AdfcConfigurator.java:50)[optimized]
    at org/apache/myfaces/trinidadinternal/config/GlobalConfiguratorImpl._startConfiguratorServiceRequest(GlobalConfiguratorImpl.java:562)[inlined]
    at org/apache/myfaces/trinidadinternal/config/GlobalConfiguratorImpl.beginRequest(GlobalConfiguratorImpl.java:212)[optimized]
    at org/apache/myfaces/trinidadinternal/webapp/TrinidadFilterImpl.doFilter(TrinidadFilterImpl.java:155)[optimized]
    at org/apache/myfaces/trinidad/webapp/TrinidadFilter.doFilter(TrinidadFilter.java:92)[optimized]
    at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[optimized]
    at oracle/adf/library/webapp/LibraryFilter.doFilter(LibraryFilter.java:175)[optimized]
    at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[optimized]
    at oracle/security/jps/ee/http/JpsAbsFilter$1.run(JpsAbsFilter.java:111)[optimized]
    at jrockit/vm/AccessController.doPrivileged(AccessController.java:254)[inlined]
    at oracle/security/jps/util/JpsSubject.doAsPrivileged(JpsSubject.java:313)[inlined]
    at oracle/security/jps/ee/util/JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:413)[inlined]
    at oracle/security/jps/ee/http/JpsAbsFilter.runJaasMode(JpsAbsFilter.java:94)[inlined]
    at oracle/security/jps/ee/http/JpsAbsFilter.doFilter(JpsAbsFilter.java:161)[optimized]
    at oracle/security/jps/ee/http/JpsFilter.doFilter(JpsFilter.java:71)[optimized]
    at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[optimized]
    at oracle/dms/servlet/DMSServletFilter.doFilter(DMSServletFilter.java:136)[optimized]
    at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[optimized]
    at weblogic/servlet/internal/RequestEventsFilter.doFilter(RequestEventsFilter.java:27)[optimized]
    at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[inlined]
    at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3715)[inlined]
    at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3681)[optimized]
    at weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)[optimized]
    at weblogic/security/service/SecurityManager.runAs(SecurityManager.java:120)[inlined]
    at weblogic/servlet/internal/WebAppServletContext.securedExecute(WebAppServletContext.java:2277)[inlined]
    at weblogic/servlet/internal/WebAppServletContext.execute(WebAppServletContext.java:2183)[optimized]
    at weblogic/servlet/internal/ServletRequestImpl.run(ServletRequestImpl.java:1454)[optimized]
    at weblogic/work/ExecuteThread.execute(ExecuteThread.java:207)[optimized]
    at weblogic/work/ExecuteThread.run(ExecuteThread.java:176)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
    any another ideas?
    BR,
    Alaa

  • "ThreadPool has stuck threads" - Warning while starting server soa 11.1.1.3

    Hi,
    When we are trying to restart the server its taking lot of time (nearly 2 hrs) and admin server status is showing as "Warning". when we check Monitoring -> Health for admin server we can see as "Thread Pool as stuck threads".
    When clicked on "threadpool", can see "ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" and current request is for "weblogic.kernel.WorkManagerWrapper$1@6ce46ce4" thread.
    Any solution for what to do.
    Thanks,
    RR

    Further Updates on this :
    I further debugged and found out few more things. This might help.
    In Weblogic Admin console Home > Summary of Deployments >oracle-bam(11.1.1)
    MDB application oracle-bam is NOT connected to messaging system with below error.
    EJBs
    MessageDispatcherBean - Error java.lang.IllegalArgumentException: Getting Deployment configuration...
    connectionFactoryJNDIName - Red Cross with warning
    destinationJNDIName - Red Cross with warning
    resourceAdapterJNDIName - Red Cross with warning
    Modules
    Also please check out another similar thread and please update.
    BAMAdapter Issue : java.rmi.ConnectException: Destination unreachable;
    Thanks,
    Sagar

  • WorkManager and ignore-stuck-thread setting

    In WL 10.3, using an application scoped work manager with the following .war deployment descriptors:
    web.xml:
    <resource-ref>
    <res-ref-name>wm/WorkManagerABC</res-ref-name>
    <res-type>commonj.work.WorkManager</res-type>
    <res-auth>Container</res-auth>
    <res-sharing-scope>Shareable</res-sharing-scope>
    </resource-ref>
    weblogic.xml:
    <work-manager>
    <name>WorkManagerABC</name>
    <fair-share-request-class>
    <name>abcClass</name>
    <fair-share>1000</fair-share>
    </fair-share-request-class>
    <ignore-stuck-threads>true</ignore-stuck-threads>
    <min-threads-constraint>
    <name>MinThreadsCountTen</name>
    <count>10</count>
    </min-threads-constraint>
    <max-threads-constraint>
    <name>MaxThreadsCountTwenty</name>
    <count>20</count>
    </max-threads-constraint>
    </work-manager>
    <wl-dispatch-policy>WorkManagerABC</wl-dispatch-policy>
    ( This particular app is meant to be a socket listener using javax.nio ). This results in WL seeing "stuck threads" after the default 600 seconds. In our clusters, our clusters are configured that 3 stuck threads cause a server restart by nm.
    So I created the same objects at the domain level ( workmanager, min/max/fair-share ), with the workmanager set to ignore stuck threads.
    web.xml
    <resource-ref>
    <res-ref-name>WorkManagerABC</res-ref-name>
    <res-type>commonj.work.WorkManager</res-type>
    <res-auth>Container</res-auth>
    <res-sharing-scope>Shareable</res-sharing-scope>
    </resource-ref>
    weblogic.xml
    <weblogic-version>10.3</weblogic-version>
    <!-- no work manager defined -->
    <wl-dispatch-policy>WorkManagerEni</wl-dispatch-policy>
    The app deploys and simple web pages are processed within the WorkManagerABC wm. Although WL still marks 4 threads as hoggers, none of them are stuck. So no restart, and the server Health stays OK.
    So a workManager at the server level obeys the "Ignore Stuck Threads" while an application scoped workmanager does not.

    I wasn't too disappointed that the application-scoped workmanager didn't obey the stuck threads; I'd rather have the work manager configurable at the server console level without having to involve customized deployment plans to override hard-coded thread restraint levels in the app's deployment descriptors.
    I think having it defined within the server console will make it simpler to manage the different performances in production versus qa, etc., than having it defined within the app and then adjusting it in each environment.

Maybe you are looking for

  • M. DisplayPort to DVI + Generic DVI to VGA Adapter???

    Does anyone know if hooking up the Apple Mini DisplayPort adapter up to a generic DVI to VGA will produce usable output? I primarily use VGA monitors, but there are a few scenarios where I do need the DVI output. Could I save $30 with this setup?

  • Hello, i want install mac osx leopard 10.5 but error in installing

    i need guaid help for error install mac os x v10.5 leopard i dont install it because send error

  • Email settings won't open

    I'm unable to access emailsettings. When trying it rewerts to home screen. I have tried restart and I have tried setting/removing restrictions, still no luck. Any ideas?

  • Software Inventory Configuration

    We have Zenworks for Desktops 7.0 SP1 installed on Netware 6.5 SP6 server. Some weeks ago hot patch 6 had been installed. Probably after that we can't access Software inventory configuration: --> Manage Unidentifed Software --> Edit button from Prope

  • Can't see incoming messages

    A funny thing happened on the way to a system reinstall. iChat quit working. At first, I would get a buddy list, but double-clicking on a name brought up a window with no name in the title, and sending a message didn't. Ah-ha, I thought, let's re-ins