Detecting and Pinning a stuck thread
While using WLS 7.0 there exists a problem where some execute threads get stuck (i.e don't respond) and do not release the associated resources such as connections, files etc. How does one detect such a stuck thread ? Also, how can we go about pinning stuck threads?
"Haziq Pathan" <[email protected]> wrote in message news:3ffbfad4$[email protected]..
While using WLS 7.0 there exists a problem where some execute threads
get stuck (i.e don't respond) and do not release the associated resources
such as connections, files etc. How does one detect such a stuck thread ?Basically there is a server health monitor thread that periodically checks
threads and reports those who started execution and didn't finish it within
configured time.
Also, how can we go about pinning stuck threads?The best thing to do is to get a thread dump and see where the thread stuck,
and deliver a fix for the problem.
Hope this helps.
Regards,
Slava Imeshev
Similar Messages
-
Page stuck while load and leads to stuck thread
Hi
I am facing an issue from a long time.
When i run the page of my application, when it opens for first time it get stuck and get loading and take very much time and after that it leads to stuck thread issue. All the component of my page opens but there is a table which gets value from OIM APIs also loads for the fiirst time. As in coding i have done it will pass null value to the table and wont fetch data for the first time. it will only fetch when value will be passed in a text box and submit was hit.
So my page gets hanged or loading in the table section everytime when it loads. for that i have to login in weblogic server, run the workmanager, forcefully shut down the server instance and run the page again. But sometime that doesnt work too.
Please help its a bug trouble for me and creating a huge problem.
Thanks
SudeepI'm not sure I understand your issue - but if you want to delay the fetching of information into the table until you actually have a value to pass to it see:
http://blogs.oracle.com/shay/2010/08/delay_method_execution_when_us.html -
OSB STUCK THREAD at LoadBalanceFailoverListener
Hi,
We get the following stuck threads on our OSB servers.
Can someone help?
Due to this, our servers are experiencing a huge resource crunch.
"[STUCK] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=3 tid=0x0000000110704800 nid=0x94 runnable [0xfffffffa1d0fc000]
java.lang.Thread.State: RUNNABLE
at java.util.HashMap.get(HashMap.java:303)
at com.bea.wli.sb.service.ServiceStats.getOperationStats(ServiceStats.java:104)
at com.bea.wli.sb.pipeline.MessageProcessor.startOperationalTimer(MessageProcessor.java:431)
at com.bea.wli.sb.pipeline.MessageProcessor.processRequest(MessageProcessor.java:93)
at com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:593)
at com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:591)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:146)
at com.bea.wli.sb.security.WLSSecurityContextService.runAs(WLSSecurityContextService.java:55)
at com.bea.wli.sb.pipeline.RouterManager.processMessage(RouterManager.java:590)
at com.bea.wli.sb.transports.TransportManagerImpl.receiveMessage(TransportManagerImpl.java:375)
at com.bea.wli.sb.transports.CoLocatedMessageContext$1.run(CoLocatedMessageContext.java:162)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:146)
at weblogic.security.Security.runAs(Security.java:61)
at com.bea.wli.sb.util.security.SecurityUtils.runAs(SecurityUtils.java:58)
at com.bea.wli.sb.transports.CoLocatedMessageContext.send(CoLocatedMessageContext.java:157)
at com.bea.wli.sb.transports.http.wls.HttpTransportProvider.sendMessageAsync(HttpTransportProvider.java:215)
at sun.reflect.GeneratedMethodAccessor914.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.bea.wli.sb.transports.Util$1.invoke(Util.java:83)
at $Proxy123.sendMessageAsync(Unknown Source)
at com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageAsync(LoadBalanceFailoverListener.java:148)
at com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageToServiceAsync(LoadBalanceFailoverListener.java:603)
at com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageToService(LoadBalanceFailoverListener.java:538)
at com.bea.wli.sb.transports.TransportManagerImpl.sendMessageToService(TransportManagerImpl.java:558)
at com.bea.wli.sb.transports.TransportManagerImpl.sendMessageAsync(TransportManagerImpl.java:426)
at com.bea.wli.sb.pipeline.PipelineContextImpl.doDispatch(PipelineContextImpl.java:670)
at com.bea.wli.sb.pipeline.PipelineContextImpl.dispatch(PipelineContextImpl.java:585)
at stages.routing.runtime.DynamicRouteRuntimeStep.processMessage(DynamicRouteRuntimeStep.java:161)
at com.bea.wli.sb.stages.StageMetadataImpl$WrapperRuntimeStep.processMessage(StageMetadataImpl.java:346)
at com.bea.wli.sb.pipeline.RouteNode.doRequest(RouteNode.java:106)
at com.bea.wli.sb.pipeline.Node.processMessage(Node.java:67)
at com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:1055)
at com.bea.wli.sb.pipeline.Router.processMessage(Router.java:214)
at com.bea.wli.sb.pipeline.MessageProcessor.processRequest(MessageProcessor.java:96)
at com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:593)
at com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:591)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:146)
at com.bea.wli.sb.security.WLSSecurityContextService.runAs(WLSSecurityContextService.java:55)
at com.bea.wli.sb.pipeline.RouterManager.processMessage(RouterManager.java:590)
at com.bea.wli.sb.transports.TransportManagerImpl.receiveMessage(TransportManagerImpl.java:375)
at com.bea.wli.sb.transports.jms.JmsInboundMDB.onMessage(JmsInboundMDB.java:132)
at sun.reflect.GeneratedMethodAccessor915.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310)
at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
at com.bea.core.repackaged.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)What i have not understood is how disabling log tracing will prevent the STUCK thread?The stuck threads which you are seeing are because of a known bug in the tracing functionality. So if you will disable the tracing then you will not hit that bug and hence no stuck threads will appear. To use tracing, get a fix for this problem from Oracle Support by raising a SR with them.
Regards,
Anuj -
I have tried to update my phone and it is stuck on "connect to iTunes" I have plugged it up to the wall charger because i have read that the new update can cause a battery drain and i have tried to reset it. None of this is working. Please help.
Device not detected in iTunes for Windows
-
Hi,
I am need of help! I have had this issue for a while now and it's becoming frustrating.
My iTunes installed on my HP Windows 8 Netbook is unable to recognize my iPod, the error message along the lines of "iTunes has detected an iPod but unable to recognise"
I have followed all the trouble shooting tips and am still stuck? I am now being told by a friend I need a link sent to me to fix this??
Someone please help me fix this problem.
Many thanks.
Matt.Hi
So just close iTunes and re-open iTunes thats it and that worked for you?
When you say reset your settings do you mean to factory settings? As my iPod was swapped at the store for a new one and the problem is still there...
Thanks. -
I have tried to update IOS 5.1 in my Iphone 4 and unfortunately I got an error message and phone is stucked at restore point. It is not coming back to normal mode and is not able to detect my phone in iTunes. Please help me how to recover my Iphone IOS, unfortunately I don't have my iphone backup too.
we have a same problem
-
Stuck Thread causing heap to grow steeply and resulting in crash
Hi,
We have an ADF application deployed in weblogic 10.3.2.
We have few PL/SQL's and SQL's which are complex and take around 5 minutes of time to complete at times. We also have the script running in database to kill long running queries so that it helps us to get rid of weblogic stuck threads to some extent.
Now when ever there is an SQL running for long time , we see a stuck thread casuing heap to grow very fast and casuing server to crash.
I know after a particular point of idle time thread is marked as stuck , but why does it filling the heap space - does stuck threads always causes heap to grow or is it just my specific case
any pointers on how to debug this issue to find the root cause.
Thanks,
NareshI think this setup needs profiling. To start-off, i would recommend collecting heap dump or jrf (if jrockit), instructions for them are available on net, as it varies with jvm versions and vendors.
Post that you should look for the thread which you believe is stuck and a possible cause using OQL in memory analyzer tool over the heap dumps.
Conceptually, till the time the objects (or data if you say) is being referenced, JVM cannot garbage collect it. So, I would say we need to figure out what are the contents of the heap, before coming to a conclusion that the threads which are stuck is causing the issue. Why I say so it that, I have seen in past that many-a-times, we suspect stuck threads to be an issue but it turns out to be that we are "knocking the wrong door".
Long running threads (aka STUCK thread in WLS), will normally cause memory issues, if the references/task that its trying to perform has a large local data (i.e. data with the thread scope), so it the time the thread is not released the data is alive (irrespective of the fact that the data is used or not).
Now, my first thought of this situation is that stuck thread is not an issue, considering that you are not able to replicate this in UAT post simulating stuck threads. Thus please collect some heap dumps (may be when you start observing memory increase) and then review them comparatively, I think that should be helpful.
AJ -
WorkManager and ignore-stuck-thread setting
In WL 10.3, using an application scoped work manager with the following .war deployment descriptors:
web.xml:
<resource-ref>
<res-ref-name>wm/WorkManagerABC</res-ref-name>
<res-type>commonj.work.WorkManager</res-type>
<res-auth>Container</res-auth>
<res-sharing-scope>Shareable</res-sharing-scope>
</resource-ref>
weblogic.xml:
<work-manager>
<name>WorkManagerABC</name>
<fair-share-request-class>
<name>abcClass</name>
<fair-share>1000</fair-share>
</fair-share-request-class>
<ignore-stuck-threads>true</ignore-stuck-threads>
<min-threads-constraint>
<name>MinThreadsCountTen</name>
<count>10</count>
</min-threads-constraint>
<max-threads-constraint>
<name>MaxThreadsCountTwenty</name>
<count>20</count>
</max-threads-constraint>
</work-manager>
<wl-dispatch-policy>WorkManagerABC</wl-dispatch-policy>
( This particular app is meant to be a socket listener using javax.nio ). This results in WL seeing "stuck threads" after the default 600 seconds. In our clusters, our clusters are configured that 3 stuck threads cause a server restart by nm.
So I created the same objects at the domain level ( workmanager, min/max/fair-share ), with the workmanager set to ignore stuck threads.
web.xml
<resource-ref>
<res-ref-name>WorkManagerABC</res-ref-name>
<res-type>commonj.work.WorkManager</res-type>
<res-auth>Container</res-auth>
<res-sharing-scope>Shareable</res-sharing-scope>
</resource-ref>
weblogic.xml
<weblogic-version>10.3</weblogic-version>
<!-- no work manager defined -->
<wl-dispatch-policy>WorkManagerEni</wl-dispatch-policy>
The app deploys and simple web pages are processed within the WorkManagerABC wm. Although WL still marks 4 threads as hoggers, none of them are stuck. So no restart, and the server Health stays OK.
So a workManager at the server level obeys the "Ignore Stuck Threads" while an application scoped workmanager does not.I wasn't too disappointed that the application-scoped workmanager didn't obey the stuck threads; I'd rather have the work manager configurable at the server console level without having to involve customized deployment plans to override hard-coded thread restraint levels in the app's deployment descriptors.
I think having it defined within the server console will make it simpler to manage the different performances in production versus qa, etc., than having it defined within the app and then adjusting it in each environment. -
How Stuck Thread Timeout and JTA timeout values are related.
Hi All,
Can some one help me in understanding the the relation between Stuck Thread Time out and JTA timeout values.
Thanks,
VardhanHi,
Both are totally different in this case.
Stuck thread timeout is specific to weblogic monitoring guide line which inform you how long every thread is in process continuously by default it would 600 sec if any thread is taken more than 600 sec to finish off its work then weblogic will throw an error message that specific thread took more than 600 seconds.
Note: Mostly of the time these stuck thread are not really an issue
Where are JTA timeout on global side is specific to your every transaction either from EJB or JMS or JDBC transaction.
This timeout will rule on every individual.
Hope you got address your query.
Regards,
Kal -
Stuck Threads Leading to server hang
Hi ,
We have been facing a problem with Stuck Threads in our production environment. We are currently running version 8.1. Though the stuck Threads do point to JDBC issues, we suspect that these problems may be related to some Weblogic/OS configuration issue as well. These probelms eventually lead to a server hang.
The general pattern of the problem is that we see messages like :
Dec 24, 2006 8:56:09 AM EST> <Debug> <Common> <BEA-000610> < RP(rtqdbConnectionPool):timeoutInactiveRes (20)>
####<Dec 25, 2006 8:54:51 AM EST> <Error> <WebLogicServer> <hygeia> <hygeia-rtq-vm7> <weblogic.health.CoreHealthMonitor> <<WLS Kernel>> <> <BEA-000337> <UNKNOWN_SESSION has been busy for "896" seconds working on the request "Http Request: /quote.do", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.> ==================== This email/fax message is for the sole use of the intended
Sometimes a deadlock Exception would be thrown from the Database,
nested exception is java.sql.SQLException: ORA-00060: deadlock detected while waiting for resource
But this is not as often as we get the Stuck Thread messages.
This is a typical Thread dump from one of our servers when it complained of a stuck thread.
///*************Thread Dump Start**********************///
Full thread dump Java HotSpot(TM) Server VM (1.4.2_08-b03 mixed mode):
"idle thread" daemon prio=5 tid=0x00d2c150 nid=0x7f in Object.wait() [94e7f000..94e7fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xb2010000> (a org.xbill.Task.WorkerThread)
at org.xbill.Task.WorkerThread.run(WorkerThread.java:130)
- locked <0xb2010000> (a org.xbill.Task.WorkerThread)
"Thread-28" daemon prio=5 tid=0x00c6a7c0 nid=0x7d in Object.wait() [92c7f000..92c7fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xbce667f8> (a com.geico.util.LockManager)
at com.geico.util.LockManager.waitForWork(LockManager.java:85)
- locked <0xbce667f8> (a com.geico.util.LockManager)
at com.geico.mail.Mail.run(Mail.java:259)
at java.lang.Thread.run(Thread.java:534)
"CacheManager-idle" daemon prio=5 tid=0x00c7d950 nid=0x7c in Object.wait() [92d7f000..92d7fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xbce66878> (a com.geico.util.LockManager)
at com.geico.util.LockManager.waitForWork(LockManager.java:85)
- locked <0xbce66878> (a com.geico.util.LockManager)
at com.geico.cache.CacheManager.run(CacheManager.java:353)
at java.lang.Thread.run(Thread.java:534)
"ExecuteThread: '2' for queue: 'weblogic.kernel.Non-Blocking'" daemon prio=5 tid=0x006a6c90 nid=0x7a in Object.wait() [92e7f000..92e7fc28]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:429)
at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
- locked <0xbc2a5db8> (a weblogic.kernel.ExecuteThread)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '1' for queue: 'weblogic.kernel.Non-Blocking'" daemon prio=5 tid=0x0120c568 nid=0x79 in Object.wait() [92f7f000..92f7fc28]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:429)
at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
- locked <0xbc2a5e38> (a weblogic.kernel.ExecuteThread)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '0' for queue: 'weblogic.kernel.Non-Blocking'" daemon prio=5 tid=0x006a6af0 nid=0x78 in Object.wait() [9307f000..9307fc28]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:429)
at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
- locked <0xbc2a5eb8> (a weblogic.kernel.ExecuteThread)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '0' for queue: 'JMSStore<AptSoftFileStore-morpheus-rtq-vm7>.ioThreadPool'" daemon prio=5 tid=0x0052a198 nid=0x77 in Object.wait() [9317f000..9317fc28]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:429)
at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
- locked <0xbc2a5f38> (a weblogic.kernel.ExecuteThread)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"Thread-26" daemon prio=5 tid=0x00635d58 nid=0x75 in Object.wait() [9327f000..9327fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xbb97ad20> (a com.geico.util.LockManager)
at java.lang.Object.wait(Object.java:429)
at com.geico.util.LockManager.waitForWork(LockManager.java:63)
- locked <0xbb97ad20> (a com.geico.util.LockManager)
at com.geico.util.providers.StorageProvider.run(StorageProvider.java:158)
at java.lang.Thread.run(Thread.java:534)
"Thread-25" daemon prio=5 tid=0x00fac430 nid=0x72 waiting on condition [9337f000..9337fc28]
at java.lang.Thread.sleep(Native Method)
at org.apache.commons.pool.impl.GenericObjectPool$Evictor.run(GenericObjectPool.java:1080)
at java.lang.Thread.run(Thread.java:534)
"ListenThread.Default" prio=5 tid=0x014ec6e8 nid=0x71 runnable [9347f000..9347fc28]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
- locked <0xbb97ae60> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at java.net.ServerSocket.accept(ServerSocket.java:419)
at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:26)
at weblogic.t3.srvr.ListenThread.accept(ListenThread.java:735)
at weblogic.t3.srvr.ListenThread.run(ListenThread.java:301)
"Thread-24" daemon prio=5 tid=0x011e9668 nid=0x70 waiting on condition [9357f000..9357fc28]
at java.lang.Thread.sleep(Native Method)
at com.geico.log.HeartbeatLogger$1.run(HeartbeatLogger.java:133)
"Thread-22" daemon prio=5 tid=0x011e8610 nid=0x6f waiting on condition [9367f000..9367fc28]
at java.lang.Thread.sleep(Native Method)
at com.geico.log.StreamLogger.pause(StreamLogger.java:263)
at com.geico.log.StreamLogger.access$300(StreamLogger.java:15)
at com.geico.log.StreamLogger$WorkerThread.run(StreamLogger.java:217)
at java.lang.Thread.run(Thread.java:534)
"filewatcher" daemon prio=5 tid=0x0111ca50 nid=0x6e waiting on condition [9377f000..9377fc28]
at java.lang.Thread.sleep(Native Method)
at com.geico.watcher.FileWatcher.sleep(FileWatcher.java:135)
at com.geico.watcher.FileWatcher.run(FileWatcher.java:118)
at java.lang.Thread.run(Thread.java:534)
"Thread-18" prio=5 tid=0x00de2388 nid=0x6d in Object.wait() [9437e000..9437fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xa20f1a60> (a weblogic.jms.backend.BEConsumerReceiveRequest)
at java.lang.Object.wait(Object.java:429)
at weblogic.jms.dispatcher.Request.sleepTillNotified(Request.java:240)
at weblogic.jms.dispatcher.Request.wrappedFiniteStateMachine(Request.java:685)
- locked <0xa20f1a60> (a weblogic.jms.backend.BEConsumerReceiveRequest)
at weblogic.jms.dispatcher.DispatcherImpl.dispatchAsyncInternal(DispatcherImpl.java:132)
at weblogic.jms.dispatcher.DispatcherImpl.dispatchAsync(DispatcherImpl.java:123)
at weblogic.jms.dispatcher.Request.dispatchAsync(Request.java:878)
at weblogic.jms.frontend.FEConsumer.receive(FEConsumer.java:452)
at weblogic.jms.frontend.FEConsumer.invoke(FEConsumer.java:643)
at weblogic.jms.dispatcher.Request.wrappedFiniteStateMachine(Request.java:643)
at weblogic.jms.dispatcher.DispatcherImpl.dispatchSyncTran(DispatcherImpl.java:192)
at weblogic.jms.client.JMSSession.receiveMessage(JMSSession.java:456)
at weblogic.jms.client.JMSConsumer.receive(JMSConsumer.java:421)
at org.springframework.jms.core.JmsTemplate.doReceive(JmsTemplate.java:800)
at org.springframework.jms.core.JmsTemplate.doReceive(JmsTemplate.java:783)
at org.springframework.jms.core.JmsTemplate$8.doInJms(JmsTemplate.java:741)
at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:582)
at org.springframework.jms.core.JmsTemplate.receive(JmsTemplate.java:739)
at org.springframework.jms.core.JmsTemplate.receive(JmsTemplate.java:731)
at com.geico.uierrorlog.UiErrorUtil.getMessage(UiErrorUtil.java:89)
at com.geico.uierrorlog.UiErrorDaemon.getMessage(UiErrorDaemon.java:108)
at com.geico.uierrorlog.UiErrorDaemon.run(UiErrorDaemon.java:166)
at java.lang.Thread.run(Thread.java:534)
"StatisticsThread" daemon prio=5 tid=0x00d0c798 nid=0x6c in Object.wait() [9447f000..9447fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xb8ee7b18> (a com.geico.tangosol.stats.StatsDaemon)
at com.geico.tangosol.stats.StatsDaemon.run(StatsDaemon.java:144)
- locked <0xb8ee7b18> (a com.geico.tangosol.stats.StatsDaemon)
at com.tangosol.util.Daemon$1.run(Daemon.java:63)
"StatisticsThread" daemon prio=5 tid=0x01202c18 nid=0x6b in Object.wait() [9457f000..9457fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xb8ee7bd8> (a com.geico.tangosol.stats.StatsDaemon)
at com.geico.tangosol.stats.StatsDaemon.run(StatsDaemon.java:144)
- locked <0xb8ee7bd8> (a com.geico.tangosol.stats.StatsDaemon)
at com.tangosol.util.Daemon$1.run(Daemon.java:63)
"StatisticsThread" daemon prio=5 tid=0x00b33f18 nid=0x6a in Object.wait() [9467f000..9467fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xb8ee7c90> (a com.geico.tangosol.stats.StatsDaemon)
at com.geico.tangosol.stats.StatsDaemon.run(StatsDaemon.java:144)
- locked <0xb8ee7c90> (a com.geico.tangosol.stats.StatsDaemon)
at com.tangosol.util.Daemon$1.run(Daemon.java:63)
"DistributedCache:DistributedRTQOverflow" daemon prio=5 tid=0x009895e8 nid=0x69 in Object.wait() [9477f000..9477fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xb8ee7d40> (a com.tangosol.coherence.component.util.daemon.QueueProcessor$Queue)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
- locked <0xb8ee7d40> (a com.tangosol.coherence.component.util.daemon.QueueProcessor$Queue)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"DistributedCache:PoolStatsDistributedCache:EventDispatcher" daemon prio=5 tid=0x00dff060 nid=0x68 in Object.wait() [9487f000..9487fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xb8ee7f18> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
- locked <0xb8ee7f18> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"DistributedCache:PoolStatsDistributedCache" daemon prio=5 tid=0x006f3e98 nid=0x67 in Object.wait() [9497f000..9497fc28]
at java.lang.Object.wait(Native Method)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
- locked <0xb8ee7fe8> (a com.tangosol.coherence.component.util.daemon.QueueProcessor$Queue)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"StatisticsThread" daemon prio=5 tid=0x0090d8d0 nid=0x66 in Object.wait() [94a7f000..94a7fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xb8ee81c8> (a com.geico.tangosol.stats.StatsDaemon)
at com.geico.tangosol.stats.StatsDaemon.run(StatsDaemon.java:144)
- locked <0xb8ee81c8> (a com.geico.tangosol.stats.StatsDaemon)
at com.tangosol.util.Daemon$1.run(Daemon.java:63)
"DistributedCache:dist-RTQSession:EventDispatcher" daemon prio=5 tid=0x00a05110 nid=0x65 in Object.wait() [94b7f000..94b7fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xb8ee8278> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
- locked <0xb8ee8278> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"DistributedCache:dist-RTQSession" daemon prio=5 tid=0x0036e600 nid=0x64 in Object.wait() [94c7f000..94c7fc28]
at java.lang.Object.wait(Native Method)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
- locked <0xb8ee8348> (a com.tangosol.coherence.component.util.daemon.QueueProcessor$Queue)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"ReplicatedCache" daemon prio=5 tid=0x00e9f9f8 nid=0x63 in Object.wait() [94d7f000..94d7fc28]
at java.lang.Object.wait(Native Method)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
- locked <0xb8ee8520> (a com.tangosol.coherence.component.util.daemon.QueueProcessor$Queue)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"Scheduler_Worker-9" prio=5 tid=0x00339820 nid=0x61 in Object.wait() [94f7f000..94f7fc28]
at java.lang.Object.wait(Native Method)
at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:423)
- locked <0xb8ee8788> (a java.lang.Object)
at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:53)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:514)
"Scheduler_Worker-8" prio=5 tid=0x0033af58 nid=0x60 in Object.wait() [9507f000..9507fc28]
at java.lang.Object.wait(Native Method)
at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:423)
- locked <0xb8ee8788> (a java.lang.Object)
at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:53)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:514)
"Scheduler_Worker-7" prio=5 tid=0x0033a878 nid=0x5f in Object.wait() [9517f000..9517fc28]
at java.lang.Object.wait(Native Method)
at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:423)
- locked <0xb8ee8788> (a java.lang.Object)
at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:53)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:514)
"Scheduler_Worker-6" prio=5 tid=0x00935488 nid=0x5e in Object.wait() [9527f000..9527fc28]
at java.lang.Object.wait(Native Method)
at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:423)
- locked <0xb8ee8788> (a java.lang.Object)
at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:53)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:514)
"Scheduler_Worker-5" prio=5 tid=0x00bb2190 nid=0x5d in Object.wait() [9537f000..9537fc28]
at java.lang.Object.wait(Native Method)
at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:423)
- locked <0xb8ee8788> (a java.lang.Object)
at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:53)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:514)
"Scheduler_Worker-4" prio=5 tid=0x010b9038 nid=0x5c in Object.wait() [9547f000..9547fc28]
at java.lang.Object.wait(Native Method)
at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:423)
- locked <0xb8ee8788> (a java.lang.Object)
at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:53)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:514)
"Scheduler_Worker-3" prio=5 tid=0x00a34678 nid=0x5b in Object.wait() [9557f000..9557fc28]
at java.lang.Object.wait(Native Method)
at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:423)
- locked <0xb8ee8788> (a java.lang.Object)
at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:53)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:514)
"Scheduler_Worker-2" prio=5 tid=0x014f9860 nid=0x5a in Object.wait() [9567f000..9567fc28]
at java.lang.Object.wait(Native Method)
at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:423)
- locked <0xb8ee8788> (a java.lang.Object)
at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:53)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:514)
"Scheduler_Worker-1" prio=5 tid=0x0044e530 nid=0x59 in Object.wait() [9577f000..9577fc28]
at java.lang.Object.wait(Native Method)
at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:423)
- locked <0xb8ee8788> (a java.lang.Object)
at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:53)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:514)
"Scheduler_Worker-0" prio=5 tid=0x0110b8e8 nid=0x58 in Object.wait() [9587f000..9587fc28]
at java.lang.Object.wait(Native Method)
at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:423)
- locked <0xb8ee8788> (a java.lang.Object)
at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:53)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:514)
"StatisticsThread" daemon prio=5 tid=0x011352b8 nid=0x57 in Object.wait() [9597f000..9597fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xb8ee8b20> (a com.geico.tangosol.stats.StatsDaemon)
at com.geico.tangosol.stats.StatsDaemon.run(StatsDaemon.java:144)
- locked <0xb8ee8b20> (a com.geico.tangosol.stats.StatsDaemon)
at com.tangosol.util.Daemon$1.run(Daemon.java:63)
"DistributedCache:dist-rtq-sessionheader:EventDispatcher" daemon prio=5 tid=0x004222a0 nid=0x56 in Object.wait() [95a7f000..95a7fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xb8ee8bd8> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
- locked <0xb8ee8bd8> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"DistributedCache:dist-rtq-sessionheader" daemon prio=5 tid=0x01278fa0 nid=0x55 in Object.wait() [95b7f000..95b7fc28]
at java.lang.Object.wait(Native Method)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
- locked <0xb8ee8ca8> (a com.tangosol.coherence.component.util.daemon.QueueProcessor$Queue)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"DistributedCache:dist-hibernate-default:EventDispatcher" daemon prio=5 tid=0x015273f0 nid=0x54 in Object.wait() [95c7f000..95c7fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xb8ee8e80> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
- locked <0xb8ee8e80> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"DistributedCache:dist-hibernate-default" daemon prio=5 tid=0x00ea1328 nid=0x53 in Object.wait() [9607f000..9607fc28]
at java.lang.Object.wait(Native Method)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
- locked <0xb8ee8f50> (a com.tangosol.coherence.component.util.daemon.QueueProcessor$Queue)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"Invocation:Management" daemon prio=5 tid=0x003d8968 nid=0x52 in Object.wait() [95d7f000..95d7fc28]
at java.lang.Object.wait(Native Method)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
- locked <0xb8448698> (a com.tangosol.coherence.component.util.daemon.QueueProcessor$Queue)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"Invocation:Management:EventDispatcher" daemon prio=5 tid=0x007e6040 nid=0x51 in Object.wait() [95e7f000..95e7fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xb84487f0> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
- locked <0xb84487f0> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"TcpRingListener" daemon prio=6 tid=0x007e6f40 nid=0x50 runnable [95f7f000..95f7fc28]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
- locked <0xb84488d0> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at java.net.ServerSocket.accept(ServerSocket.java:419)
at com.tangosol.coherence.component.net.socket.TcpSocketAccepter.accept(TcpSocketAccepter.CDB:17)
at com.tangosol.coherence.component.util.daemon.TcpRingListener.acceptConnection(TcpRingListener.CDB:9)
at com.tangosol.coherence.component.util.daemon.TcpRingListener.onNotify(TcpRingListener.CDB:1)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:34)
at java.lang.Thread.run(Thread.java:534)
"Cluster" daemon prio=5 tid=0x00c94738 nid=0x4e in Object.wait() [9617f000..9617fc28]
at java.lang.Object.wait(Native Method)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
- locked <0xb8448ac0> (a com.tangosol.coherence.component.net.Cluster$ClusterService$Queue)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"PacketPublisher" daemon prio=6 tid=0x00d83418 nid=0x4d runnable [9627f000..9627fc28]
at java.lang.Object.wait(Native Method)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
- locked <0xb8448c78> (a com.tangosol.coherence.component.net.Cluster$PacketPublisher$Queue)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"PacketReceiver" daemon prio=7 tid=0x00b3cd70 nid=0x4c in Object.wait() [9637f000..9637fc28]
at java.lang.Object.wait(Native Method)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
- locked <0xb8448dc0> (a com.tangosol.coherence.component.net.Cluster$PacketReceiver$Queue)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"PacketListenerN" daemon prio=8 tid=0x003d7cd0 nid=0x4b runnable [9647f000..9647fc28]
at java.net.PlainDatagramSocketImpl.receive(Native Method)
at java.net.DatagramSocket.receive(DatagramSocket.java:711)
- locked <0xb84494c8> (a java.net.DatagramPacket)
- locked <0xb84494e8> (a java.net.MulticastSocket)
at com.tangosol.coherence.component.net.socket.UdpSocket.receive(UdpSocket.CDB:18)
at com.tangosol.coherence.component.net.UdpPacket.receive(UdpPacket.CDB:17)
at com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onWait(PacketListener.CDB:5)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"PacketListener1" daemon prio=8 tid=0x00df5538 nid=0x4a runnable [9657f000..9657fc28]
at java.net.PlainDatagramSocketImpl.receive(Native Method)
at java.net.DatagramSocket.receive(DatagramSocket.java:711)
- locked <0xb8449cb8> (a java.net.DatagramPacket)
- locked <0xb8449cd8> (a java.net.DatagramSocket)
at com.tangosol.coherence.component.net.socket.UdpSocket.receive(UdpSocket.CDB:18)
at com.tangosol.coherence.component.net.UdpPacket.receive(UdpPacket.CDB:17)
at com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onWait(PacketListener.CDB:5)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"Logger@13785534 3.1.2/343" daemon prio=3 tid=0x006b7358 nid=0x49 in Object.wait() [9667f000..9667fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xb8449e90> (a com.tangosol.coherence.component.application.console.Coherence$Logger$Queue)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
- locked <0xb8449e90> (a com.tangosol.coherence.component.application.console.Coherence$Logger$Queue)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"ExecuteThread: '5' for queue: 'JMS.TimerClientPool'" daemon prio=5 tid=0x011384b0 nid=0x48 in Object.wait() [9677f000..9677fc28]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:429)
at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
- locked <0xb82c91f8> (a weblogic.kernel.ExecuteThread)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '4' for queue: 'JMS.TimerClientPool'" daemon prio=5 tid=0x01137bc8 nid=0x47 in Object.wait() [9687f000..9687fc28]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:429)
at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
- locked <0xb82c9278> (a weblogic.kernel.ExecuteThread)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '3' for queue: 'JMS.TimerClientPool'" daemon prio=5 tid=0x0111d4e0 nid=0x46 in Object.wait() [9697f000..9697fc28]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:429)
at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
- locked <0xb82c92f8> (a weblogic.kernel.ExecuteThread)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '2' for queue: 'JMS.TimerClientPool'" daemon prio=5 tid=0x010dd218 nid=0x45 in Object.wait() [96a7f000..96a7fc28]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:429)
at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
- locked <0xb82c9378> (a weblogic.kernel.ExecuteThread)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '1' for queue: 'JMS.TimerClientPool'" daemon prio=5 tid=0x00bb1c98 nid=0x44 in Object.wait() [96b7f000..96b7fc28]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:429)
at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
- locked <0xb82c93f8> (a weblogic.kernel.ExecuteThread)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '0' for queue: 'JMS.TimerClientPool'" daemon prio=5 tid=0x00979ac8 nid=0x43 in Object.wait() [96c7f000..96c7fc28]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:429)
at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
- locked <0xb82c9478> (a weblogic.kernel.ExecuteThread)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"Thread-12" daemon prio=5 tid=0x00200610 nid=0x42 waiting on condition [96d7f000..96d7fc28]
at java.lang.Thread.sleep(Native Method)
at com.geico.log.HeartbeatLogger$1.run(HeartbeatLogger.java:133)
"Thread-10" daemon prio=5 tid=0x00200470 nid=0x41 waiting on condition [96e7f000..96e7fc28]
at java.lang.Thread.sleep(Native Method)
at com.geico.log.StreamLogger.pause(StreamLogger.java:267)
at com.geico.log.StreamLogger.access$300(StreamLogger.java:19)
at com.geico.log.StreamLogger$WorkerThread.run(StreamLogger.java:221)
at java.lang.Thread.run(Thread.java:534)
"ClockDaemonThread-0" daemon prio=5 tid=0x00ab6a48 nid=0x40 in Object.wait() [96f7f000..96f7fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xb8051b20> (a com.geico.util.concurrent.ClockDaemon)
at com.geico.util.concurrent.ClockDaemon.nextTask(ClockDaemon.java:328)
- locked <0xb8051b20> (a com.geico.util.concurrent.ClockDaemon)
at com.geico.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:362)
at java.lang.Thread.run(Thread.java:534)
"Thread-9" prio=5 tid=0x008912e0 nid=0x3f in Object.wait() [9ad7f000..9ad7fc28]
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:429)
- locked <0xb8051bc0> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:382)
"ExecuteThread: '0' for queue: 'JMS.TimerTreePool'" daemon prio=5 tid=0x001d7fa8 nid=0x3e in Object.wait() [9ae7f000..9ae7fc28]
at java.lang.Object.wait(Native Method)
at weblogic.jms.backend.BETimerTree.execute(BETimerTree.java:146)
- locked <0xb8051c90> (a weblogic.jms.backend.BETimerTree)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)
"Thread-8" prio=5 tid=0x00247418 nid=0x3d in Object.wait() [9af7f000..9af7fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xb8051d20> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:429)
- locked <0xb8051d20> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:382)
"Thread-7" prio=5 tid=0x001d8a28 nid=0x3c in Object.wait() [9b07f000..9b07fc28]
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:429)
- locked <0xb8051dc0> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:382)
"ExecuteThread: '0' for queue: 'weblogic.cluster.MulticastManager'" daemon prio=5 tid=0x00da0a60 nid=0x3b runnable [9b17f000..9b17fc28]
at java.net.PlainDatagramSocketImpl.receive(Native Method)
at java.net.DatagramSocket.receive(DatagramSocket.java:711)
- locked <0xa22dc3e8> (a java.net.DatagramPacket)
- locked <0xb7558130> (a java.net.MulticastSocket)
at weblogic.cluster.FragmentSocket.receive(FragmentSocket.java:169)
at weblogic.cluster.MulticastManager.execute(MulticastManager.java:395)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)
"SSLListenThread.Administrator" prio=5 tid=0x00c7db08 nid=0x3a runnable [9b27f000..9b27fc28]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
- locked <0xb75582a0> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at javax.net.ssl.impl.SSLServerSocketImpHi,
I guess this issue is related to your connectivity and hyperion services. Can you take a restart and try to execute the reports. Also try to check all your Hyperion services are up and running before executing the FR Reports. You can also check the Reporting services by login to Weblogic server (if you have applied it in your environment). In weblogic it should be in Running mode and not in Admin mode. If it is not in Running mode then try to check the Data Source setting from Connection Pool in Weblogic Server.
Soumya -
PAPI client calls - stuck threads with albpm57 engine
PAPI client running on weblogic portal 10.3, connecting to ALBPM 5.7 engine (on weblogic 8.1 cluster).
threads get stuck in WLP servers (where PAPI client runs), and the call is for getActiveInstances from default view. We have around 15000 active instances, and the papi cache size is configured for 25000.
"[STUCK] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" id=17 idx=0x7c tid=6615 prio=1 alive, in native, waiting, daemon
-- Waiting for notification on: fuego/papi/impl/lock/SimpleLock@0x1f443568[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)[optimized]
at java/lang/Object.wait(Object.java:485)
at fuego/papi/impl/lock/SimpleLock.readLock(SimpleLock.java:50)
^-- Lock released while waiting: fuego/papi/impl/lock/SimpleLock@0x1f443568[fat lock]
at fuego/papi/impl/InstanceCache$Entry.readLock(InstanceCache.java:708)
at fuego/papi/impl/InstanceCache.getInstancesByFilter(InstanceCache.java:505)
at fuego/papi/impl/InstanceCache.getInstancesByFilter(InstanceCache.java:231)
at fuego/papi/impl/ProcessServiceImpl.getInstancesByFilter(ProcessServiceImpl.java:1084)
at fuego/papi/impl/ProcessServiceNESessionImpl.getInstancesByFilter(ProcessServiceNESessionImpl.java:754)
Please help how to resolve this.
thanks in advance.
Edited by: 811778 on May 17, 2013 5:05 PMsupport team suggested to use,
-Dfuego.papi.lockManagerClass=fuego.papi.impl.lock.ParanoidLockManager for each java process, acting as PAPI client. This class available in fuegopapi-client.jar
additional comments from support team,
Implementing this property does not generate side-effects. This property detects and kills/recovers deadlocks in the PAPI instance cache -
Where we can know about the stuck threads without taking thread dump.
Hi Everyone,
Where we can View/know about the stuck threads without taking thread dump.
Thank YouHi!
WebLogic Server diagnoses a thread as stuck if it is continually working (not idle) for a set period of time. You can tune stuck threads in the Weblogic Server.
You can tune a server’s thread detection behavior by changing the length of time before a thread is diagnosed as stuck, and by changing the frequency with which the server checks for stuck threads.
It can be configured using below mentioned 2 parameters. These can be configured from Admin console -> Servers ->Configuration > Tuning tab.
1)Stuck Thread Max Time
2)Stuck Thread Timer Interval
Although you can change the criteria WebLogic Server uses to determine whether a thread is stuck, you cannot change the default behavior of setting the “warning” and “critical” health states when all threads in a particular execute queue become stuck.
There is a parameter "Stuck Thread Count" which can be configured from Console
(Servers -> Configuration ->Overload -> Stuck Thread count) which helps to transition the Server to FAILED state once the stuck threads reaches the value.
You can also use "OverloadProtectionMBean" for tuning. In Weblogic Server 9.x and later, it is recommended that you use the "ServerFailureTriggerMBean" in the "OverloadProtectionMBean".The ServerFailureTriggerMBean transitions the server to a FAILED state after the specified number of stuck threads are detected.The OverloadProtectionMBean has options to suspend or shutdown a failed server.
WebLogic Server checks for stuck threads periodically. If all application threads are stuck, a server instance marks itself failed, if configured to do so, exits. You can configure Node Manager or a third-party high-availability solution to restart the server instance for automatic failure recovery.
You can configure these actions to occur when not all threads are stuck, but the number of stuck threads have exceeded a configured threshold:
- Shut down the Work Manager if it has stuck threads. A Work Manager that is shut down will refuse new work and reject existing work in the queue by sending a rejection message. In a cluster, clustered clients will fail over to another cluster member.
- Shut down the application if there are stuck threads in the application. The application is shutdown by bringing it into ADMIN mode. All Work Managers belonging to the application are shut down, and behave as described above.
- Mark the server instance as failed and shut it down it down if there are stuck threads in the server. In a cluster, clustered clients that are connected or attempting to connect will fail over to another cluster member.
You can configure the "ServerFailureTriggerMBean" in the "OverloadProtectionMBean".
Below is documentation link for "ServerFailureTriggerMBean" methods.
http://download.oracle.com/docs/cd/E12839_01/apirefs.1111/e13945/weblogic/management/configuration/ServerFailureTriggerMBean.html
You can use getStuckThreadCount() method to check the number of stuck threads and transition the server to Failed State once itreaches the limit.
getStuckThreadCount:
int getStuckThreadCount() - The number of stuck threads after which the server is transitioned into FAILED state. There are options inOverloadProtectionMBean to suspend and shutdown a FAILED server. By default, the server continues to run in FAILED state. If the StuckThreadCount value is set to zero then the server never transitions into FAILED server irrespective of the number of stuck threads.
Returns:
The StuckThreadCount value
Default Value:
0
Maximum Value:
java.lang.Integer.MAX_VALUE
Minimum Value:
0
Below is documentation link for "OverloadProtectionMBean" methods.
http://download.oracle.com/docs/cd/E11035_01/wls100/javadocs_mhome/weblogic/management/configuration/OverloadProtectionMBean.html
In the Admin console, you can set the "FailureAction" under Servers->Configuration->Overload to force shutdown the managed server once the server is in Failed state.
The OverloadProtectionMBean has a method getFailureAction to achieve the same.
getFailureAction:
String getFailureAction() - Enable automatic forceshutdown of the server on failed state. The server self-health monitoring detects fatal failures and mark the server as failed. The server can be restarted using NodeManager or a HA agent.
Valid Values:
OverloadProtectionMBean.NO_ACTION, OverloadProtectionMBean.FORCE_SHUTDOWN, OverloadProtectionMBean.ADMIN_STATE
If you start the managed servers using node manager, you can enable "Auto Kill if Failed" and "Auto Restart" in the Admin console, under Servers-> configuration->Health Monitoring. Node Manager will take care of restarting the managed server if you enable "Auto Restart".
You can also configure the "Stuck Thread Count" and "Failure Action" to "Force Immediate shutdown of the Server" from Admin console under servers-> configuration-> Overload. This will help you to shutdown the server when the stuck thread count is reached. But there is no way to release the threads once they are stuck from the configuration.
Suppose if you set the value of the Stuck Thread Count to 20, The server will be transitioned to failed state once the count reaches 20 and if you enable the Failure Action, the server self-health monitoring detects fatal failures and mark the server as failed.
Depending on how you are starting the servers (custom scripts or Node Manager or startup scripts,...), you can restart the servers.
For more details on this please refer to below link:
http://www.oracle.com/technetwork/articles/entarch/workload-management-088692.html
Hope this helps.
Thanks,
Cris -
Stuck Threads on weblogic 10.3.2 Holding lock HttpURLConnection
We are having issue with STUCK THREADS on weblogic 10.3.2. Basically my application makes a lot of web service calls over HTTPS and once in a while I see few STUCK THREADS and most of the time they clear but sometimes I have to bounce my server to clean.
I have a thread dump, according to this its holding lock at weblogic.net.http.HttpURLConnection.getInputStream
Any help to resolve this problem is highly appreciated.
ThreadDump:
<May 24, 2011 9:59:08 AM EDT> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "604" seconds working on the request "weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl@b607d75", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-75 "[STUCK] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, priority=1, DAEMON> {
jrockit.net.SocketNativeIO.readBytesPinned(SocketNativeIO.java:???)
jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:31)
java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)
java.net.SocketInputStream.read(SocketInputStream.java:107)
weblogic.utils.io.ChunkedInputStream.read(ChunkedInputStream.java:149)
java.io.InputStream.read(InputStream.java:85)
com.certicom.tls.record.ReadHandler.readFragment(Unknown Source)
com.certicom.tls.record.ReadHandler.readRecord(Unknown Source)
com.certicom.tls.record.ReadHandler.read(Unknown Source)
^-- Holding lock: com.certicom.tls.record.ReadHandler@bec563c[thin lock]
com.certicom.io.InputSSLIOStreamWrapper.read(Unknown Source)
java.io.BufferedInputStream.fill(BufferedInputStream.java:189)
java.io.BufferedInputStream.read(BufferedInputStream.java:236)
^-- Holding lock: java.io.BufferedInputStream@bec5d9f[thin lock]
weblogic.net.http.MessageHeader.isHTTP(MessageHeader.java:214)
weblogic.net.http.MessageHeader.parseHeader(MessageHeader.java:141)
weblogic.net.http.HttpClient.parseHTTP(HttpClient.java:452)
^-- Holding lock: weblogic.net.http.HttpsClient@bec5202[thin lock]
weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:328)
weblogic.net.http.SOAPHttpsURLConnection.getInputStream(SOAPHttpsURLConnection.java:37)
^-- Holding lock: weblogic.net.http.SOAPHttpsURLConnection@bd3b209[thin lock]
weblogic.net.http.HttpURLConnection.getResponseCode(HttpURLConnection.java:939)
com.sun.xml.ws.transport.http.client.HttpClientTransport.readResponseCodeAndMessage(HttpClientTransport.java:213)
com.sun.xml.ws.transport.http.client.HttpTransportPipe.process(HttpTransportPipe.java:116)
com.sun.xml.ws.transport.http.client.HttpTransportPipe.processRequest(HttpTransportPipe.java:101)
com.sun.xml.ws.api.pipe.Fiber.__doRun(Fiber.java:580)
com.sun.xml.ws.api.pipe.Fiber._doRun(Fiber.java:553)
com.sun.xml.ws.api.pipe.Fiber.doRun(Fiber.java:535)
com.sun.xml.ws.api.pipe.Fiber.runSync(Fiber.java:433)
^-- Holding lock: com.sun.xml.ws.api.pipe.Fiber@bd3afa8[thin lock]
com.sun.xml.ws.client.Stub.process(Stub.java:238)
com.sun.xml.ws.client.sei.SEIStub.doProcess(SEIStub.java:135)
com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:102)
com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:89)
com.sun.xml.ws.client.sei.SEIStub.invoke(SEIStub.java:116)
$Proxy107.jobSubmit(Unknown Source)
com.somejob.invokeService(JobOperation.java:53)
com.somejobws.operations.ticket.execute(ticket.java:30)
com.somejobutil.Somehelper.process(Somehelper.java:174)
com.somejobutil.Somehelper.processJob(Somehelper.java:147)
com.somejobutil.Somehelper.processMDB(Somehelper.java:91)
com.somejobejb.ProcessMDB.onMessage(ProcessMDB.java:28)
sun.reflect.GeneratedMethodAccessor109.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:102)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:86)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:102)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:151)
$Proxy67.onMessage(Unknown Source)
weblogic.ejb.container.internal.MDListener.execute(MDListener.java:403)
weblogic.ejb.container.internal.MDListener.transactionalOnMessage(MDListener.java:352)
weblogic.ejb.container.internal.MDListener.onMessage(MDListener.java:264)
weblogic.jms.client.JMSSession.onMessage(JMSSession.java:4564)
weblogic.jms.client.JMSSession.execute(JMSSession.java:4060)
weblogic.jms.client.JMSSession.executeMessage(JMSSession.java:3746)
weblogic.jms.client.JMSSession.access$000(JMSSession.java:114)
weblogic.jms.client.JMSSession$UseForRunnable.run(JMSSession.java:5096)
weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:516)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:198)
weblogic.work.ExecuteThread.run(ExecuteThread.java:165)
}Yes, we have the same problem, stuck thread on weblogic.net.http.HttpsURLConnection.getResponseCode() (actually implemented by weblogic.net.http.SOAPHttpsURLConnection which also appears to be deprecated ???)
When we use the class javax.net.ssl.HttpsURLConnection the getResponseCode() method works perfectly well, so it looks like the weblogic implementation is to blame.
This is becoming rather urgent now because we're looking to migrate from OC4J to Weblogic and this issue has therefore become a blocker. -
Stuck thread in Oracle Weblogic server
Dear All,
We have a stuck thread in Oracle WLS 10.3.3. We have deployed 7 ear files in 5th managed server(psMS5). We have stuck thread in one of the application (psinsurance). We have allocated 1GB heap space for the psMS5 managed server. We are using this in Development environment. Please check the below message from psMS5 managed server log file. Kindly provide reason for stuck thread and solution.
WatchTime: Sep 23, 2011 10:28:29 AM IST
WatchDomainName: dev_domain
WatchServerName: psMS5
WatchSeverityLevel: Warning
WatchName: MS5-Watch
WatchRuleType: Log
WatchRule: ((MESSAGE LIKE '%has been busy for "%') AND (MSGID = 'BEA-000337') AND (SERVER = 'psMS5'))
WatchData: DATE = Sep 23, 2011 10:28:28 AM IST SERVER = psMS5 MESSAGE = [STUCK] ExecuteThread: '22' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "629" seconds working on the request "weblogic.servlet.internal.ServletRequestImpl@1cac6525[
POST */psinsurance/app/lending/insurance/PS00738D.jspx*?_adf.ctrl-state=24f0li74e_4 HTTP/1.1
Accept: image/jpeg, application/x-ms-application, image/gif, application/xaml+xml, image/pjpeg, application/x-ms-xbap, */*
Referer: http://sis-prosun11/psinsurance/faces/PS00003?jsessionid=r2BrT8TpqZBsnLhCw2Y8KDpTV7hMpgnLDFW54P87ZpHwn8JMy88s%21168932459%211316753022074&_afrWindowMode=0&_afrLoop=244724092027000&_adf.ctrl-state=24f0li74e_4
Accept-Language: en-US
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; Tablet PC 2.0)
Content-Type: application/x-www-form-urlencoded
Accept-Encoding: gzip, deflate
Content-Length: 554
Cache-Control: no-cache
Cookie: JSESSIONID=LkJcT8TfMT64SyQJT4Ls4sXhfDl0pTS7CXJVQ9KcryHGWJ1dhyfz!1373322659
Connection: Keep-Alive
Proxy-Client-IP: 192.168.9.179
X-Forwarded-For: 192.168.9.179
X-WebLogic-KeepAliveSecs: 30
X-WebLogic-Force-JVMID: 1373322659
]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-237 "[STUCK] ExecuteThread: '22' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, suspended, priority=1, DAEMON> {
java.io.UnixFileSystem.getBooleanAttributes0(UnixFileSystem.java:???)
java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:228)
java.io.File.exists(File.java:729)
weblogic.utils.classloaders.ClasspathClassFinder.getFileSource(ClasspathClassFinder.java:383)
weblogic.utils.classloaders.ClasspathClassFinder.getSourcesInternal(ClasspathClassFinder.java:333)
weblogic.utils.classloaders.ClasspathClassFinder.getSource(ClasspathClassFinder.java:316)
weblogic.utils.classloaders.JarClassFinder.getSource(JarClassFinder.java:58)
weblogic.utils.classloaders.AbstractClassFinder.getClassSource(AbstractClassFinder.java:22)
weblogic.utils.classloaders.MultiClassFinder.getClassSource(MultiClassFinder.java:56)
weblogic.utils.classloaders.CodeGenClassFinder.getClassSource(CodeGenClassFinder.java:26)
weblogic.utils.classloaders.GenericClassLoader.findLocalClass(GenericClassLoader.java:268)
*^-- Holding lock: weblogic.utils.classloaders.GenericClassLoader@1e3295a1[thin lock]*
weblogic.utils.classloaders.GenericClassLoader.findClass(GenericClassLoader.java:237)
java.lang.ClassLoader.loadClass(ClassLoader.java:292)
*^-- Holding lock: weblogic.utils.classloaders.GenericClassLoader@1e3295a1[thin lock]*
java.lang.ClassLoader.loadClass(ClassLoader.java:292)
java.lang.ClassLoader.loadClass(ClassLoader.java:248)
weblogic.utils.classloaders.GenericClassLoader.loadClass(GenericClassLoader.java:177)
weblogic.utils.classloaders.ChangeAwareClassLoader.loadClass(ChangeAwareClassLoader.java:37)
*^-- Holding lock: weblogic.utils.classloaders.ChangeAwareClassLoader@1e553474[thin lock]*
java.lang.Class.getDeclaredConstructors0(Class.java:???)
java.lang.Class.privateGetDeclaredConstructors(Class.java:2370)
java.lang.Class.getConstructor0(Class.java:2699)
java.lang.Class.newInstance0(Class.java:318)
java.lang.Class.newInstance(Class.java:305)
org.apache.myfaces.trinidadinternal.renderkit.ClassRendererInstantiator.instantiate(ClassRendererInstantiator.java:48)
org.apache.myfaces.trinidadinternal.renderkit.RenderKitBase.findRenderer(RenderKitBase.java:146)
org.apache.myfaces.trinidadinternal.renderkit.RenderKitDecorator.findRenderer(RenderKitDecorator.java:84)
oracle.adfinternal.view.faces.unified.renderkit.UnifiedRenderKit._getRenderer(UnifiedRenderKit.java:218)
oracle.adfinternal.view.faces.unified.renderkit.UnifiedRenderKit.findRenderer(UnifiedRenderKit.java:156)
org.apache.myfaces.trinidadinternal.renderkit.RenderKitBase.getRenderer(RenderKitBase.java:129)
org.apache.myfaces.trinidad.component.UIXComponentBase._getRendererImpl(UIXComponentBase.java:1143)
org.apache.myfaces.trinidad.component.UIXComponentBase.getRenderer(UIXComponentBase.java:1173)
org.apache.myfaces.trinidad.component.UIXComponentBase.getClientId(UIXComponentBase.java:316)
javax.faces.webapp.UIComponentClassicTagBase.doStartTag(UIComponentClassicTagBase.java:1095)
org.apache.myfaces.trinidad.webapp.UIXComponentELTag.doStartTag(UIXComponentELTag.java:70)
oracle.adfinternal.view.faces.unified.taglib.input.UnifiedSelectBooleanCheckboxTag.doStartTag(UnifiedSelectBooleanCheckboxTag.java:45)
oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
oracle.jsp.runtime.tree.OracleJspIterationTagNode.executeHandler(OracleJspIterationTagNode.java:41)
oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
oracle.jsp.runtime.tree.OracleJspIterationTagNode.executeHandler(OracleJspIterationTagNode.java:41)
oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
oracle.jsp.runtime.tree.OracleJspNode.execute(OracleJspNode.java:73)
oracle.jsp.runtimev2.ShortCutServlet._jspService(ShortCutServlet.java:57)
oracle.jsp.runtime.OracleJspBase.service(OracleJspBase.java:29)
oracle.jsp.runtimev2.JspPageTable.compileAndServe(JspPageTable.java:650)
oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.java:330)
oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.java:680)
oracle.jsp.runtimev2.JspServlet.service(JspServlet.java:653)
javax.servlet.http.HttpServlet.service(HttpServlet.java:815)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:206)
weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.adf.library.webapp.LibraryFilter.doFilter(LibraryFilter.java:147)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
weblogic.servlet.internal.RequestDispatcherImpl.invokeServlet(RequestDispatcherImpl.java:520)
weblogic.servlet.internal.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:112)
com.sun.faces.context.ExternalContextImpl.dispatch(ExternalContextImpl.java:402)
org.apache.myfaces.trinidad.context.ExternalContextDecorator.dispatch(ExternalContextDecorator.java:44)
org.apache.myfaces.trinidad.context.ExternalContextDecorator.dispatch(ExternalContextDecorator.java:44)
org.apache.myfaces.trinidad.context.ExternalContextDecorator.dispatch(ExternalContextDecorator.java:44)
org.apache.myfaces.trinidad.context.ExternalContextDecorator.dispatch(ExternalContextDecorator.java:44)
org.apache.myfaces.trinidadinternal.context.FacesContextFactoryImpl$OverrideDispatch.dispatch(FacesContextFactoryImpl.java:261)
com.sun.faces.application.ViewHandlerImpl.executePageToBuildView(ViewHandlerImpl.java:428)
com.sun.faces.application.ViewHandlerImpl.renderView(ViewHandlerImpl.java:132)
javax.faces.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:189)
org.apache.myfaces.trinidadinternal.application.ViewHandlerImpl.renderView(ViewHandlerImpl.java:155)
oracle.adfinternal.view.faces.lifecycle.LifecycleImpl._renderResponse(LifecycleImpl.java:706)
oracle.adfinternal.view.faces.lifecycle.LifecycleImpl._executePhase(LifecycleImpl.java:244)
oracle.adfinternal.view.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:199)
javax.faces.webapp.FacesServlet.service(FacesServlet.java:246)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:206)
weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.adf.model.servlet.ADFBindingFilter.doFilter(ADFBindingFilter.java:134)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.adfinternal.view.faces.webapp.rich.RegistrationFilter.doFilter(RegistrationFilter.java:69)
org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:411)
oracle.adfinternal.view.faces.activedata.AdsFilter.doFilter(AdsFilter.java:44)
org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:411)
org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl._doFilterImpl(TrinidadFilterImpl.java:210)
org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl.doFilter(TrinidadFilterImpl.java:126)
org.apache.myfaces.trinidad.webapp.TrinidadFilter.doFilter(TrinidadFilter.java:91)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:94)
oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:299)
oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:406)
oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:82)
oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:70)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.adf.library.webapp.LibraryFilter.doFilter(LibraryFilter.java:147)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.dms.wls.DMSServletFilter.doFilter(DMSServletFilter.java:114)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:25)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.doIt(WebAppServletContext.java:3656)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3650)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:308)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:117)
weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2204)
weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2126)
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1412)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:198)
weblogic.work.ExecuteThread.run(ExecuteThread.java:165)
SUBSYSTEM = WebLogicServer USERID = <WLS Kernel> SEVERITY = Error THREAD = [ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)' MSGID = BEA-000337 MACHINE = sis-prosun11.sis.ad TXID = CONTEXTID = 11d1def534ea1be0:3bbe257e:13286031cbf:-7ffd-0000000000002fe7 TIMESTAMP = 1316753908672
WatchAlarmType: AutomaticReset
WatchAlarmResetPeriod: 60000
SMTPNotificationName: MS5-Health-NotificationI found something interesting to the question I posted from the below link:
http://www.mail-archive.com/[email protected]/msg03217.html -
Need help with STUCK Threads. What's causing them?
In our production environment, we have a soa suite cluster with 3 nodes. All 3 of the servers keep turning up stuck and hogging threads, and eventually they all grind to a halt where I can't even view composite instances through em. There is a bpel process that is using a dbadapter to poll for records every 5 seconds and then performs some service call outs and other dbadapters. This process is generating thousands of instances a day and I think it is the culprit. If I view the logs, I see one of the dbadapters that it calls out to fails occasionally with a PK violation. I looked at the developer's code, and they have no error handling in the process at all. In the logs, there will the a hit for the PK violation, then about 10 minutes later the a STUCK thread gets logged(the configured timeout is 600 seconds, or 10 minutes). Would not handling an error in a BPEL process hog/STUCK a thread? I need to be able to show what code is causing the issue so I can have a resolution path.
I also see a note in the trace about something holding a lock:
Holding lock: oracle.jdbc.driver.T4CConnection@1fbc3486[thin lock]....would that be an un-rolled back transaction?
Here is the dump of the STUCK thread. Any help is greatly appreciated. Thanks!
<Aug 5, 2011 10:30:02 AM EDT> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '64' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "634" seconds working on the request "oracle.soa.management.internal.ejb.impl.FacadeFinderBean_4vacyo_FacadeFinderBeanImpl", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-517 "[STUCK] ExecuteThread: '64' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, priority=1, DAEMON> {
jrockit.net.SocketNativeIO.readBytesPinned(SocketNativeIO.java:???)
jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:24)
java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)
java.net.SocketInputStream.read(SocketInputStream.java:107)
oracle.net.nt.MetricsEnabledInputStream.read(TcpNTAdapter.java:553)
oracle.net.ns.Packet.receive(Packet.java:249)
oracle.net.ns.DataPacket.receive(DataPacket.java:94)
oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:168)
oracle.net.ns.NetInputStream.read(NetInputStream.java:113)
oracle.net.ns.NetInputStream.read(NetInputStream.java:96)
oracle.net.ns.NetInputStream.read(NetInputStream.java:81)
oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:120)
oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:75)
oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1081)
oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1062)
oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:588)
oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:70)
oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:854)
oracle.jdbc.driver.T4CPreparedStatement.executeMaybeDescribe(T4CPreparedStatement.java:918)
oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1252)
oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3541)
oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3605)
^-- Holding lock: oracle.jdbc.driver.T4CConnection@1fbc3486[thin lock]
oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1667)
weblogic.jdbc.wrapper.PreparedStatement.executeQuery(PreparedStatement.java:123)
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeSelect(DatabaseAccessor.java:887)
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:541)
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:530)
org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:521)
org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:202)
org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:192)
org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelectCall(DatasourceCallQueryMechanism.java:254)
org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelect(DatasourceCallQueryMechanism.java:245)
org.eclipse.persistence.queries.DataReadQuery.executeNonCursor(DataReadQuery.java:188)
org.eclipse.persistence.queries.DataReadQuery.executeDatabaseQuery(DataReadQuery.java:141)
org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:603)
org.eclipse.persistence.queries.DataReadQuery.execute(DataReadQuery.java:115)
org.eclipse.persistence.queries.DatabaseQuery.executeInUnitOfWork(DatabaseQuery.java:589)
org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2898)
org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1218)
org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1190)
org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1176)
org.eclipse.persistence.internal.jpa.EJBQueryImpl.executeReadQuery(EJBQueryImpl.java:378)
org.eclipse.persistence.internal.jpa.EJBQueryImpl.getSingleResult(EJBQueryImpl.java:701)
weblogic.deployment.TransactionalEntityManagerProxyImpl$QueryImpl.getSingleResult(TransactionalEntityManagerProxyImpl.java:232)
com.collaxa.cube.persistence.dao.impl.jpa.JpaCubeInstanceDaoImpl.getNumberOfInstances(JpaCubeInstanceDaoImpl.java:439)
oracle.fabric.CubeServiceEngine.getNumberOfComponentInstances(CubeServiceEngine.java:1962)
oracle.soa.management.internal.ejb.impl.FacadeFinderBeanImpl.getNumberOfComponentInstances(FacadeFinderBeanImpl.java:1618)
oracle.soa.management.internal.ejb.impl.FacadeFinderBeanImpl.getComponentSnapShot(FacadeFinderBeanImpl.java:1653)
sun.reflect.GeneratedMethodAccessor5852.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.jee.intercept.MethodInvocationInvocationContext.proceed(MethodInvocationInvocationContext.java:104)
oracle.security.jps.ee.ejb.JpsAbsInterceptor$1.run(JpsAbsInterceptor.java:88)
oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:299)
oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:406)
oracle.security.jps.wls.JpsWeblogicEjbInterceptor.runJaasMode(JpsWeblogicEjbInterceptor.java:61)
oracle.security.jps.ee.ejb.JpsAbsInterceptor.intercept(JpsAbsInterceptor.java:81)
oracle.security.jps.ee.ejb.JpsInterceptor.intercept(JpsInterceptor.java:101)
sun.reflect.GeneratedMethodAccessor863.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
com.bea.core.repackaged.springframework.jee.intercept.JeeInterceptorInterceptor.invoke(JeeInterceptorInterceptor.java:66)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:102)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.jee.spi.MethodInvocationVisitorImpl.visit(MethodInvocationVisitorImpl.java:37)
weblogic.ejb.container.injection.EnvironmentInterceptorCallbackImpl.callback(EnvironmentInterceptorCallbackImpl.java:22)
com.bea.core.repackaged.springframework.jee.spi.EnvironmentInterceptor.invoke(EnvironmentInterceptor.java:49)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:86)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:102)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:151)
$Proxy309.getComponentSnapShot(Unknown Source)
oracle.soa.management.internal.ejb.impl.FacadeFinderBean_4vacyo_FacadeFinderBeanImpl.getComponentSnapShot(FacadeFinderBean_4vacyo_FacadeFinderBeanImpl.java:1604)
oracle.soa.management.internal.ejb.impl.FacadeFinderBean_4vacyo_FacadeFinderBeanImpl_WLSkel.invoke(Unknown Source)
weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:567)
weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:477)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:350)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:143)
weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:425)
weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:117)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:198)
weblogic.work.ExecuteThread.run(ExecuteThread.java:165)
>
<Aug 5, 2011 10:30:02 AM EDT> <Notice> <Diagnostics> <BEA-320068> <Watch 'StuckThread' with severity 'Notice' on server 'soa_server1' has triggered at Aug 5, 2011 10:30:02 AM EDT. Notification details:
WatchRuleType: Log
WatchRule: (SEVERITY = 'Error') AND (MSGID = 'BEA-000337')
WatchData: DATE = Aug 5, 2011 10:30:02 AM EDT SERVER = soa_server1 MESSAGE = [STUCK] ExecuteThread: '64' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "634" seconds working on the request "oracle.soa.management.internal.ejb.impl.FacadeFinderBean_4vacyo_FacadeFinderBeanImpl", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-517 "[STUCK] ExecuteThread: '64' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, priority=1, DAEMON> {
jrockit.net.SocketNativeIO.readBytesPinned(SocketNativeIO.java:???)
jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:24)
java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)
java.net.SocketInputStream.read(SocketInputStream.java:107)
oracle.net.nt.MetricsEnabledInputStream.read(TcpNTAdapter.java:553)
oracle.net.ns.Packet.receive(Packet.java:249)
oracle.net.ns.DataPacket.receive(DataPacket.java:94)
oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:168)
oracle.net.ns.NetInputStream.read(NetInputStream.java:113)
oracle.net.ns.NetInputStream.read(NetInputStream.java:96)
oracle.net.ns.NetInputStream.read(NetInputStream.java:81)
oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:120)
oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:75)
oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1081)
oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1062)
oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:588)
oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:70)
oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:854)
oracle.jdbc.driver.T4CPreparedStatement.executeMaybeDescribe(T4CPreparedStatement.java:918)
oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1252)
oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3541)
oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3605)
^-- Holding lock: oracle.jdbc.driver.T4CConnection@1fbc3486[thin lock]
oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1667)
weblogic.jdbc.wrapper.PreparedStatement.executeQuery(PreparedStatement.java:123)
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeSelect(DatabaseAccessor.java:887)
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:541)
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:530)
org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:521)
org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:202)
org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:192)
org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelectCall(DatasourceCallQueryMechanism.java:254)
org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelect(DatasourceCallQueryMechanism.java:245)
org.eclipse.persistence.queries.DataReadQuery.executeNonCursor(DataReadQuery.java:188)
org.eclipse.persistence.queries.DataReadQuery.executeDatabaseQuery(DataReadQuery.java:141)
org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:603)
org.eclipse.persistence.queries.DataReadQuery.execute(DataReadQuery.java:115)
org.eclipse.persistence.queries.DatabaseQuery.executeInUnitOfWork(DatabaseQuery.java:589)
org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2898)
org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1218)
org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1190)
org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1176)
org.eclipse.persistence.internal.jpa.EJBQueryImpl.executeReadQuery(EJBQueryImpl.java:378)
org.eclipse.persistence.internal.jpa.EJBQueryImpl.getSingleResult(EJBQueryImpl.java:701)
weblogic.deployment.TransactionalEntityManagerProxyImpl$QueryImpl.getSingleResult(TransactionalEntityManagerProxyImpl.java:232)
com.collaxa.cube.persistence.dao.impl.jpa.JpaCubeInstanceDaoImpl.getNumberOfInstances(JpaCubeInstanceDaoImpl.java:439)
oracle.fabric.CubeServiceEngine.getNumberOfComponentInstances(CubeServiceEngine.java:1962)
oracle.soa.management.internal.ejb.impl.FacadeFinderBeanImpl.getNumberOfComponentInstances(FacadeFinderBeanImpl.java:1618)
oracle.soa.management.internal.ejb.impl.FacadeFinderBeanImpl.getComponentSnapShot(FacadeFinderBeanImpl.java:1653)
sun.reflect.GeneratedMethodAccessor5852.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.jee.intercept.MethodInvocationInvocationContext.proceed(MethodInvocationInvocationContext.java:104)
oracle.security.jps.ee.ejb.JpsAbsInterceptor$1.run(JpsAbsInterceptor.java:88)
oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:299)
oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:406)
oracle.security.jps.wls.JpsWeblogicEjbInterceptor.runJaasMode(JpsWeblogicEjbInterceptor.java:61)
oracle.security.jps.ee.ejb.JpsAbsInterceptor.intercept(JpsAbsInterceptor.java:81)
oracle.security.jps.ee.ejb.JpsInterceptor.intercept(JpsInterceptor.java:101)
sun.reflect.GeneratedMethodAccessor863.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
com.bea.core.repackaged.springframework.jee.intercept.JeeInterceptorInterceptor.invoke(JeeInterceptorInterceptor.java:66)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:102)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.jee.spi.MethodInvocationVisitorImpl.visit(MethodInvocationVisitorImpl.java:37)
weblogic.ejb.container.injection.EnvironmentInterceptorCallbackImpl.callback(EnvironmentInterceptorCallbackImpl.java:22)
com.bea.core.repackaged.springframework.jee.spi.EnvironmentInterceptor.invoke(EnvironmentInterceptor.java:49)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:86)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:102)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
com.bea.core.repackaged.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:151)
$Proxy309.getComponentSnapShot(Unknown Source)
oracle.soa.management.internal.ejb.impl.FacadeFinderBean_4vacyo_FacadeFinderBeanImpl.getComponentSnapShot(FacadeFinderBean_4vacyo_FacadeFinderBeanImpl.java:1604)
oracle.soa.management.internal.ejb.impl.FacadeFinderBean_4vacyo_FacadeFinderBeanImpl_WLSkel.invoke(Unknown Source)
weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:567)
weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:477)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:350)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:143)
weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:425)
weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:117)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:198)
weblogic.work.ExecuteThread.run(ExecuteThread.java:165)
SUBSYSTEM = WebLogicServer USERID = <WLS Kernel> SEVERITY = Error THREAD = [ACTIVE] ExecuteThread: '96' for queue: 'weblogic.kernel.Default (self-tuning)' MSGID = BEA-000337 MACHINE = soaprod TXID = CONTEXTID = f5d0b85a929630a2:6eada0f7:1304e0a2bdc:-7ffb-00000000017e1605 TIMESTAMP = 1312554602698
WatchAlarmType: AutomaticReset
WatchAlarmResetPeriod: 600000
>
Edited by: Brutus35 on Aug 5, 2011 8:04 AMmy impression is that you have trouble on the DB,
monitor the sessions to see if there is some table locked, and increase the number of processes on the DB
you can use SQLDeveloper for that
anyway your TX should rollback after some time, check that you have a timeout in place, it's better to rollback a TX than having the entire server crash
Maybe you are looking for
-
Is there a PRINT SCREEN option on a MacBook Pro running OS LION
Is there a PRINT SCREEN option on a Mac book Pro 17" running on OS LION/ Sometimes I want to print what I see on the screern in a PDF file that does not have page numbers. Guesws work at best...wasted ink and paper if not the correct page
-
SCOM 2007 R2 CU 7 SQL Database script update
Hi..I'm in the process of installing CU 7 for SCOM 2007 R2 (SQL 2008 R2). I completed the Server update successfully. While running the SQL operations manager DB upgrade I get the message '(0 row(s) affected' Auto-attach to process '[144] [SQL] Serve
-
VERY new here. I am working on cenverting a VBScript/MySQL script to oracle. I plan to use the vbscript to place data into an oracle express 10g database. I used migration workbench to migrate the schema from MySQL to Oracle Express. This went well.
-
Whats up and Happy weekend. I often use this simple keyboard shortcut while using Safari, (holding)⌘ and clicking link to open in a new tab. Quite often, I need that link opened in a separate window. My first machine was an appleII after 37 years on
-
Error message on launch of iMovie
I get following message on launch of iMovie. I have re-installed iMovie but problem persists: "The QuickTime components necessary to view, edit, import, and export several kinds of movies are not installed. These components are included with the iMov