Stuck Thread causing heap to grow steeply and resulting in crash

Hi,
We have an ADF application deployed in weblogic 10.3.2.
We have few PL/SQL's and SQL's which are complex and take around 5 minutes of time to complete at times. We also have the script running in database to kill long running queries so that it helps us to get rid of weblogic stuck threads to some extent.
Now when ever there is an SQL running for long time , we see a stuck thread casuing heap to grow very fast and casuing server to crash.
I know after a particular point of idle time thread is marked as stuck , but why does it filling the heap space - does stuck threads always causes heap to grow or is it just my specific case
any pointers on how to debug this issue to find the root cause.
Thanks,
Naresh

I think this setup needs profiling. To start-off, i would recommend collecting heap dump or jrf (if jrockit), instructions for them are available on net, as it varies with jvm versions and vendors.
Post that you should look for the thread which you believe is stuck and a possible cause using OQL in memory analyzer tool over the heap dumps.
Conceptually, till the time the objects (or data if you say) is being referenced, JVM cannot garbage collect it. So, I would say we need to figure out what are the contents of the heap, before coming to a conclusion that the threads which are stuck is causing the issue. Why I say so it that, I have seen in past that many-a-times, we suspect stuck threads to be an issue but it turns out to be that we are "knocking the wrong door".
Long running threads (aka STUCK thread in WLS), will normally cause memory issues, if the references/task that its trying to perform has a large local data (i.e. data with the thread scope), so it the time the thread is not released the data is alive (irrespective of the fact that the data is used or not).
Now, my first thought of this situation is that stuck thread is not an issue, considering that you are not able to replicate this in UAT post simulating stuck threads. Thus please collect some heap dumps (may be when you start observing memory increase) and then review them comparatively, I think that should be helpful.
AJ

Similar Messages

  • WorkManager and ignore-stuck-thread setting

    In WL 10.3, using an application scoped work manager with the following .war deployment descriptors:
    web.xml:
    <resource-ref>
    <res-ref-name>wm/WorkManagerABC</res-ref-name>
    <res-type>commonj.work.WorkManager</res-type>
    <res-auth>Container</res-auth>
    <res-sharing-scope>Shareable</res-sharing-scope>
    </resource-ref>
    weblogic.xml:
    <work-manager>
    <name>WorkManagerABC</name>
    <fair-share-request-class>
    <name>abcClass</name>
    <fair-share>1000</fair-share>
    </fair-share-request-class>
    <ignore-stuck-threads>true</ignore-stuck-threads>
    <min-threads-constraint>
    <name>MinThreadsCountTen</name>
    <count>10</count>
    </min-threads-constraint>
    <max-threads-constraint>
    <name>MaxThreadsCountTwenty</name>
    <count>20</count>
    </max-threads-constraint>
    </work-manager>
    <wl-dispatch-policy>WorkManagerABC</wl-dispatch-policy>
    ( This particular app is meant to be a socket listener using javax.nio ). This results in WL seeing "stuck threads" after the default 600 seconds. In our clusters, our clusters are configured that 3 stuck threads cause a server restart by nm.
    So I created the same objects at the domain level ( workmanager, min/max/fair-share ), with the workmanager set to ignore stuck threads.
    web.xml
    <resource-ref>
    <res-ref-name>WorkManagerABC</res-ref-name>
    <res-type>commonj.work.WorkManager</res-type>
    <res-auth>Container</res-auth>
    <res-sharing-scope>Shareable</res-sharing-scope>
    </resource-ref>
    weblogic.xml
    <weblogic-version>10.3</weblogic-version>
    <!-- no work manager defined -->
    <wl-dispatch-policy>WorkManagerEni</wl-dispatch-policy>
    The app deploys and simple web pages are processed within the WorkManagerABC wm. Although WL still marks 4 threads as hoggers, none of them are stuck. So no restart, and the server Health stays OK.
    So a workManager at the server level obeys the "Ignore Stuck Threads" while an application scoped workmanager does not.

    I wasn't too disappointed that the application-scoped workmanager didn't obey the stuck threads; I'd rather have the work manager configurable at the server console level without having to involve customized deployment plans to override hard-coded thread restraint levels in the app's deployment descriptors.
    I think having it defined within the server console will make it simpler to manage the different performances in production versus qa, etc., than having it defined within the app and then adjusting it in each environment.

  • Need help with STUCK Threads.  What's causing them?

    In our production environment, we have a soa suite cluster with 3 nodes. All 3 of the servers keep turning up stuck and hogging threads, and eventually they all grind to a halt where I can't even view composite instances through em. There is a bpel process that is using a dbadapter to poll for records every 5 seconds and then performs some service call outs and other dbadapters. This process is generating thousands of instances a day and I think it is the culprit. If I view the logs, I see one of the dbadapters that it calls out to fails occasionally with a PK violation. I looked at the developer's code, and they have no error handling in the process at all. In the logs, there will the a hit for the PK violation, then about 10 minutes later the a STUCK thread gets logged(the configured timeout is 600 seconds, or 10 minutes). Would not handling an error in a BPEL process hog/STUCK a thread? I need to be able to show what code is causing the issue so I can have a resolution path.
    I also see a note in the trace about something holding a lock:
    Holding lock: oracle.jdbc.driver.T4CConnection@1fbc3486[thin lock]....would that be an un-rolled back transaction?
    Here is the dump of the STUCK thread. Any help is greatly appreciated. Thanks!
    <Aug 5, 2011 10:30:02 AM EDT> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '64' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "634" seconds working on the request "oracle.soa.management.internal.ejb.impl.FacadeFinderBean_4vacyo_FacadeFinderBeanImpl", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
    Thread-517 "[STUCK] ExecuteThread: '64' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, priority=1, DAEMON> {
    jrockit.net.SocketNativeIO.readBytesPinned(SocketNativeIO.java:???)
    jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:24)
    java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)
    java.net.SocketInputStream.read(SocketInputStream.java:107)
    oracle.net.nt.MetricsEnabledInputStream.read(TcpNTAdapter.java:553)
    oracle.net.ns.Packet.receive(Packet.java:249)
    oracle.net.ns.DataPacket.receive(DataPacket.java:94)
    oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:168)
    oracle.net.ns.NetInputStream.read(NetInputStream.java:113)
    oracle.net.ns.NetInputStream.read(NetInputStream.java:96)
    oracle.net.ns.NetInputStream.read(NetInputStream.java:81)
    oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:120)
    oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:75)
    oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1081)
    oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1062)
    oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:588)
    oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:70)
    oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:854)
    oracle.jdbc.driver.T4CPreparedStatement.executeMaybeDescribe(T4CPreparedStatement.java:918)
    oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1252)
    oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3541)
    oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3605)
    ^-- Holding lock: oracle.jdbc.driver.T4CConnection@1fbc3486[thin lock]
    oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1667)
    weblogic.jdbc.wrapper.PreparedStatement.executeQuery(PreparedStatement.java:123)
    org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeSelect(DatabaseAccessor.java:887)
    org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:541)
    org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:530)
    org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:521)
    org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:202)
    org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:192)
    org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelectCall(DatasourceCallQueryMechanism.java:254)
    org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelect(DatasourceCallQueryMechanism.java:245)
    org.eclipse.persistence.queries.DataReadQuery.executeNonCursor(DataReadQuery.java:188)
    org.eclipse.persistence.queries.DataReadQuery.executeDatabaseQuery(DataReadQuery.java:141)
    org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:603)
    org.eclipse.persistence.queries.DataReadQuery.execute(DataReadQuery.java:115)
    org.eclipse.persistence.queries.DatabaseQuery.executeInUnitOfWork(DatabaseQuery.java:589)
    org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2898)
    org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1218)
    org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1190)
    org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1176)
    org.eclipse.persistence.internal.jpa.EJBQueryImpl.executeReadQuery(EJBQueryImpl.java:378)
    org.eclipse.persistence.internal.jpa.EJBQueryImpl.getSingleResult(EJBQueryImpl.java:701)
    weblogic.deployment.TransactionalEntityManagerProxyImpl$QueryImpl.getSingleResult(TransactionalEntityManagerProxyImpl.java:232)
    com.collaxa.cube.persistence.dao.impl.jpa.JpaCubeInstanceDaoImpl.getNumberOfInstances(JpaCubeInstanceDaoImpl.java:439)
    oracle.fabric.CubeServiceEngine.getNumberOfComponentInstances(CubeServiceEngine.java:1962)
    oracle.soa.management.internal.ejb.impl.FacadeFinderBeanImpl.getNumberOfComponentInstances(FacadeFinderBeanImpl.java:1618)
    oracle.soa.management.internal.ejb.impl.FacadeFinderBeanImpl.getComponentSnapShot(FacadeFinderBeanImpl.java:1653)
    sun.reflect.GeneratedMethodAccessor5852.invoke(Unknown Source)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    java.lang.reflect.Method.invoke(Method.java:575)
    com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.jee.intercept.MethodInvocationInvocationContext.proceed(MethodInvocationInvocationContext.java:104)
    oracle.security.jps.ee.ejb.JpsAbsInterceptor$1.run(JpsAbsInterceptor.java:88)
    oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:299)
    oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:406)
    oracle.security.jps.wls.JpsWeblogicEjbInterceptor.runJaasMode(JpsWeblogicEjbInterceptor.java:61)
    oracle.security.jps.ee.ejb.JpsAbsInterceptor.intercept(JpsAbsInterceptor.java:81)
    oracle.security.jps.ee.ejb.JpsInterceptor.intercept(JpsInterceptor.java:101)
    sun.reflect.GeneratedMethodAccessor863.invoke(Unknown Source)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    java.lang.reflect.Method.invoke(Method.java:575)
    com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
    com.bea.core.repackaged.springframework.jee.intercept.JeeInterceptorInterceptor.invoke(JeeInterceptorInterceptor.java:66)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
    com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:102)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.jee.spi.MethodInvocationVisitorImpl.visit(MethodInvocationVisitorImpl.java:37)
    weblogic.ejb.container.injection.EnvironmentInterceptorCallbackImpl.callback(EnvironmentInterceptorCallbackImpl.java:22)
    com.bea.core.repackaged.springframework.jee.spi.EnvironmentInterceptor.invoke(EnvironmentInterceptor.java:49)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:86)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
    com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:102)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:151)
    $Proxy309.getComponentSnapShot(Unknown Source)
    oracle.soa.management.internal.ejb.impl.FacadeFinderBean_4vacyo_FacadeFinderBeanImpl.getComponentSnapShot(FacadeFinderBean_4vacyo_FacadeFinderBeanImpl.java:1604)
    oracle.soa.management.internal.ejb.impl.FacadeFinderBean_4vacyo_FacadeFinderBeanImpl_WLSkel.invoke(Unknown Source)
    weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:567)
    weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:477)
    weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:350)
    weblogic.security.service.SecurityManager.runAs(SecurityManager.java:143)
    weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:425)
    weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:117)
    weblogic.work.ExecuteThread.execute(ExecuteThread.java:198)
    weblogic.work.ExecuteThread.run(ExecuteThread.java:165)
    >
    <Aug 5, 2011 10:30:02 AM EDT> <Notice> <Diagnostics> <BEA-320068> <Watch 'StuckThread' with severity 'Notice' on server 'soa_server1' has triggered at Aug 5, 2011 10:30:02 AM EDT. Notification details:
    WatchRuleType: Log
    WatchRule: (SEVERITY = 'Error') AND (MSGID = 'BEA-000337')
    WatchData: DATE = Aug 5, 2011 10:30:02 AM EDT SERVER = soa_server1 MESSAGE = [STUCK] ExecuteThread: '64' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "634" seconds working on the request "oracle.soa.management.internal.ejb.impl.FacadeFinderBean_4vacyo_FacadeFinderBeanImpl", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
    Thread-517 "[STUCK] ExecuteThread: '64' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, priority=1, DAEMON> {
    jrockit.net.SocketNativeIO.readBytesPinned(SocketNativeIO.java:???)
    jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:24)
    java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)
    java.net.SocketInputStream.read(SocketInputStream.java:107)
    oracle.net.nt.MetricsEnabledInputStream.read(TcpNTAdapter.java:553)
    oracle.net.ns.Packet.receive(Packet.java:249)
    oracle.net.ns.DataPacket.receive(DataPacket.java:94)
    oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:168)
    oracle.net.ns.NetInputStream.read(NetInputStream.java:113)
    oracle.net.ns.NetInputStream.read(NetInputStream.java:96)
    oracle.net.ns.NetInputStream.read(NetInputStream.java:81)
    oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:120)
    oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:75)
    oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1081)
    oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1062)
    oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:588)
    oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:70)
    oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:854)
    oracle.jdbc.driver.T4CPreparedStatement.executeMaybeDescribe(T4CPreparedStatement.java:918)
    oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1252)
    oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3541)
    oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3605)
    ^-- Holding lock: oracle.jdbc.driver.T4CConnection@1fbc3486[thin lock]
    oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1667)
    weblogic.jdbc.wrapper.PreparedStatement.executeQuery(PreparedStatement.java:123)
    org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeSelect(DatabaseAccessor.java:887)
    org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:541)
    org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:530)
    org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:521)
    org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:202)
    org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:192)
    org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelectCall(DatasourceCallQueryMechanism.java:254)
    org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelect(DatasourceCallQueryMechanism.java:245)
    org.eclipse.persistence.queries.DataReadQuery.executeNonCursor(DataReadQuery.java:188)
    org.eclipse.persistence.queries.DataReadQuery.executeDatabaseQuery(DataReadQuery.java:141)
    org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:603)
    org.eclipse.persistence.queries.DataReadQuery.execute(DataReadQuery.java:115)
    org.eclipse.persistence.queries.DatabaseQuery.executeInUnitOfWork(DatabaseQuery.java:589)
    org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2898)
    org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1218)
    org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1190)
    org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1176)
    org.eclipse.persistence.internal.jpa.EJBQueryImpl.executeReadQuery(EJBQueryImpl.java:378)
    org.eclipse.persistence.internal.jpa.EJBQueryImpl.getSingleResult(EJBQueryImpl.java:701)
    weblogic.deployment.TransactionalEntityManagerProxyImpl$QueryImpl.getSingleResult(TransactionalEntityManagerProxyImpl.java:232)
    com.collaxa.cube.persistence.dao.impl.jpa.JpaCubeInstanceDaoImpl.getNumberOfInstances(JpaCubeInstanceDaoImpl.java:439)
    oracle.fabric.CubeServiceEngine.getNumberOfComponentInstances(CubeServiceEngine.java:1962)
    oracle.soa.management.internal.ejb.impl.FacadeFinderBeanImpl.getNumberOfComponentInstances(FacadeFinderBeanImpl.java:1618)
    oracle.soa.management.internal.ejb.impl.FacadeFinderBeanImpl.getComponentSnapShot(FacadeFinderBeanImpl.java:1653)
    sun.reflect.GeneratedMethodAccessor5852.invoke(Unknown Source)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    java.lang.reflect.Method.invoke(Method.java:575)
    com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.jee.intercept.MethodInvocationInvocationContext.proceed(MethodInvocationInvocationContext.java:104)
    oracle.security.jps.ee.ejb.JpsAbsInterceptor$1.run(JpsAbsInterceptor.java:88)
    oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:299)
    oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:406)
    oracle.security.jps.wls.JpsWeblogicEjbInterceptor.runJaasMode(JpsWeblogicEjbInterceptor.java:61)
    oracle.security.jps.ee.ejb.JpsAbsInterceptor.intercept(JpsAbsInterceptor.java:81)
    oracle.security.jps.ee.ejb.JpsInterceptor.intercept(JpsInterceptor.java:101)
    sun.reflect.GeneratedMethodAccessor863.invoke(Unknown Source)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    java.lang.reflect.Method.invoke(Method.java:575)
    com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
    com.bea.core.repackaged.springframework.jee.intercept.JeeInterceptorInterceptor.invoke(JeeInterceptorInterceptor.java:66)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
    com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:102)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.jee.spi.MethodInvocationVisitorImpl.visit(MethodInvocationVisitorImpl.java:37)
    weblogic.ejb.container.injection.EnvironmentInterceptorCallbackImpl.callback(EnvironmentInterceptorCallbackImpl.java:22)
    com.bea.core.repackaged.springframework.jee.spi.EnvironmentInterceptor.invoke(EnvironmentInterceptor.java:49)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:86)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
    com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:102)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:151)
    $Proxy309.getComponentSnapShot(Unknown Source)
    oracle.soa.management.internal.ejb.impl.FacadeFinderBean_4vacyo_FacadeFinderBeanImpl.getComponentSnapShot(FacadeFinderBean_4vacyo_FacadeFinderBeanImpl.java:1604)
    oracle.soa.management.internal.ejb.impl.FacadeFinderBean_4vacyo_FacadeFinderBeanImpl_WLSkel.invoke(Unknown Source)
    weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:567)
    weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:477)
    weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:350)
    weblogic.security.service.SecurityManager.runAs(SecurityManager.java:143)
    weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:425)
    weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:117)
    weblogic.work.ExecuteThread.execute(ExecuteThread.java:198)
    weblogic.work.ExecuteThread.run(ExecuteThread.java:165)
    SUBSYSTEM = WebLogicServer USERID = <WLS Kernel> SEVERITY = Error THREAD = [ACTIVE] ExecuteThread: '96' for queue: 'weblogic.kernel.Default (self-tuning)' MSGID = BEA-000337 MACHINE = soaprod TXID = CONTEXTID = f5d0b85a929630a2:6eada0f7:1304e0a2bdc:-7ffb-00000000017e1605 TIMESTAMP = 1312554602698
    WatchAlarmType: AutomaticReset
    WatchAlarmResetPeriod: 600000
    >
    Edited by: Brutus35 on Aug 5, 2011 8:04 AM

    my impression is that you have trouble on the DB,
    monitor the sessions to see if there is some table locked, and increase the number of processes on the DB
    you can use SQLDeveloper for that
    anyway your TX should rollback after some time, check that you have a timeout in place, it's better to rollback a TX than having the entire server crash

  • Page stuck while load and leads to stuck thread

    Hi
    I am facing an issue from a long time.
    When i run the page of my application, when it opens for first time it get stuck and get loading and take very much time and after that it leads to stuck thread issue. All the component of my page opens but there is a table which gets value from OIM APIs also loads for the fiirst time. As in coding i have done it will pass null value to the table and wont fetch data for the first time. it will only fetch when value will be passed in a text box and submit was hit.
    So my page gets hanged or loading in the table section everytime when it loads. for that i have to login in weblogic server, run the workmanager, forcefully shut down the server instance and run the page again. But sometime that doesnt work too.
    Please help its a bug trouble for me and creating a huge problem.
    Thanks
    Sudeep

    I'm not sure I understand your issue - but if you want to delay the fetching of information into the table until you actually have a value to pass to it see:
    http://blogs.oracle.com/shay/2010/08/delay_method_execution_when_us.html

  • Detecting and Pinning a stuck thread

    While using WLS 7.0 there exists a problem where some execute threads get stuck (i.e don't respond) and do not release the associated resources such as connections, files etc. How does one detect such a stuck thread ? Also, how can we go about pinning stuck threads?

    "Haziq Pathan" <[email protected]> wrote in message news:3ffbfad4$[email protected]..
    While using WLS 7.0 there exists a problem where some execute threads
    get stuck (i.e don't respond) and do not release the associated resources
    such as connections, files etc. How does one detect such a stuck thread ?Basically there is a server health monitor thread that periodically checks
    threads and reports those who started execution and didn't finish it within
    configured time.
    Also, how can we go about pinning stuck threads?The best thing to do is to get a thread dump and see where the thread stuck,
    and deliver a fix for the problem.
    Hope this helps.
    Regards,
    Slava Imeshev

  • How Stuck Thread Timeout and JTA timeout values are related.

    Hi All,
    Can some one help me in understanding the the relation between Stuck Thread Time out and JTA timeout values.
    Thanks,
    Vardhan

    Hi,
    Both are totally different in this case.
    Stuck thread timeout is specific to weblogic monitoring guide line which inform you how long every thread is in process continuously by default it would 600 sec if any thread is taken more than 600 sec to finish off its work then weblogic will throw an error message that specific thread took more than 600 seconds.
    Note: Mostly of the time these stuck thread are not really an issue
    Where are JTA timeout on global side is specific to your every transaction either from EJB or JMS or JDBC transaction.
    This timeout will rule on every individual.
    Hope you got address your query.
    Regards,
    Kal

  • Need some help with stucked threads

    Hi, I am running an ADF application on Weblogic 10.3.4. I get a lot of stucked threads and can't figure out the reason.
    Here is my stack dump:
    <31.mar.2011 kl 08.36 CEST> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '21' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "625" seconds working on the request "weblogic.servlet.internal.ServletRequestImpl@237aaa4c[
    POST /faces/hovedside?_adf.ctrl-state=13zq3miluz_4 HTTP/1.1
    Connection: keep-alive
    Referer: https://please.help.me/?_afrLoop=8410499084123&_afrWindowMode=0&_afrWindowId=null
    Content-Length: 356
    Adf-Ads-Page-Id: 3
    Origin: https://please.help.me
    User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.204 Safari/534.16
    Content-Type: application/x-www-form-urlencoded; charset=UTF-8
    Adf-Rich-Message: true
    Accept: */*
    Accept-Encoding: gzip,deflate,sdch
    Accept-Language: nb-NO,nb;q=0.8,no;q=0.6,nn;q=0.4,en-US;q=0.2,en;q=0.2
    Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
    Cookie: JSESSIONID=1dQ7NJpXZhYGM11Sf2ppDz4LZwQM5FpL6yxDrj0cSlMTsTdWvPPN!-222523808
    ]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
    Thread-133 "[STUCK] ExecuteThread: '21' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, suspended, priority=1, DAEMON> {
    java.util.ArrayList.clone(ArrayList.java:250)
    oracle.jbo.uicli.binding.JUCtrlHierNodeBinding.getChildren(JUCtrlHierNodeBinding.java:719)
    oracle.jbo.uicli.binding.JUCtrlHierNodeBinding.findChildNode(JUCtrlHierNodeBinding.java:844)
    oracle.jbo.uicli.binding.JUCtrlHierBinding.bringNodeToRangeKeyPath(JUCtrlHierBinding.java:772)
    oracle.adfinternal.view.faces.model.binding.FacesCtrlHierBinding.bringNodeToRangeKeyPath(FacesCtrlHierBinding.java:101)
    oracle.adfinternal.view.faces.model.binding.RowDataManager.setRowKey(RowDataManager.java:119)
    oracle.adfinternal.view.faces.model.binding.FacesCtrlHierBinding$FacesModel.setRowKey(FacesCtrlHierBinding.java:633)
    org.apache.myfaces.trinidad.model.RowKeySetTreeImpl$PathIterator.nextModelKey(RowKeySetTreeImpl.java:745)
    org.apache.myfaces.trinidad.model.RowKeySetTreeImpl$PathIterator.nextItem(RowKeySetTreeImpl.java:740)
    org.apache.myfaces.trinidad.model.RowKeySetTreeImpl$PathIterator.next(RowKeySetTreeImpl.java:721)
    oracle.adfinternal.view.faces.renderkit.rich.TreeRendererUtils.initModel(TreeRendererUtils.java:288)
    oracle.adfinternal.view.faces.renderkit.rich.TreeRendererUtils.wrapModel(TreeRendererUtils.java:527)
    oracle.adfinternal.view.faces.renderkit.rich.TreeRendererUtils.getCollectionObject(TreeRendererUtils.java:340)
    oracle.adfinternal.view.faces.renderkit.rich.TreeTableRenderer.getCollectionObject(TreeTableRenderer.java:762)
    oracle.adfinternal.view.faces.renderkit.rich.TableRenderer.storeHeightAutosized(TableRenderer.java:2365)
    oracle.adfinternal.view.faces.renderkit.rich.TableRenderer.encodeAll(TableRenderer.java:482)
    oracle.adfinternal.view.faces.renderkit.rich.TreeTableRenderer.encodeAll(TreeTableRenderer.java:585)
    oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1252)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:326)
    org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:758)
    org.apache.myfaces.trinidad.component.UIXCollection.encodeEnd(UIXCollection.java:527)
    javax.faces.component.UIComponent.encodeAll(UIComponent.java:920)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:398)
    oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2608)
    oracle.adfinternal.view.faces.renderkit.rich.PanelGroupLayoutRenderer._encodeChild(PanelGroupLayoutRenderer.java:398)
    oracle.adfinternal.view.faces.renderkit.rich.PanelGroupLayoutRenderer.access$300(PanelGroupLayoutRenderer.java:30)
    oracle.adfinternal.view.faces.renderkit.rich.PanelGroupLayoutRenderer$EncoderCallback.processComponent(PanelGroupLayoutRenderer.java:582)
    oracle.adfinternal.view.faces.renderkit.rich.PanelGroupLayoutRenderer$EncoderCallback.processComponent(PanelGroupLayoutRenderer.java:573)
    org.apache.myfaces.trinidad.component.UIXComponent.processFlattenedChildren(UIXComponent.java:154)
    org.apache.myfaces.trinidad.component.UIXComponent.processFlattenedChildren(UIXComponent.java:285)
    org.apache.myfaces.trinidad.component.UIXComponent.encodeFlattenedChildren(UIXComponent.java:252)
    oracle.adfinternal.view.faces.renderkit.rich.PanelGroupLayoutRenderer.encodeAll(PanelGroupLayoutRenderer.java:236)
    oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1252)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:326)
    org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:758)
    javax.faces.component.UIComponent.encodeAll(UIComponent.java:920)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:398)
    oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2608)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeAllChildren(CoreRenderer.java:407)
    oracle.adfinternal.view.faces.renderkit.rich.PanelHeaderRenderer.renderChildrenAfterHelpAndInfo(PanelHeaderRenderer.java:471)
    oracle.adfinternal.view.faces.renderkit.rich.PanelHeaderRenderer._renderContentCell(PanelHeaderRenderer.java:939)
    oracle.adfinternal.view.faces.renderkit.rich.PanelHeaderRenderer.renderContentRow(PanelHeaderRenderer.java:425)
    oracle.adfinternal.view.faces.renderkit.rich.PanelHeaderRenderer.encodeAll(PanelHeaderRenderer.java:160)
    oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1252)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:326)
    org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:758)
    javax.faces.component.UIComponent.encodeAll(UIComponent.java:920)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:398)
    javax.faces.component.UIComponent.encodeAll(UIComponent.java:920)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:398)
    oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2608)
    oracle.adfinternal.view.faces.renderkit.rich.PanelGroupLayoutRenderer._encodeChild(PanelGroupLayoutRenderer.java:398)
    oracle.adfinternal.view.faces.renderkit.rich.PanelGroupLayoutRenderer.access$300(PanelGroupLayoutRenderer.java:30)
    oracle.adfinternal.view.faces.renderkit.rich.PanelGroupLayoutRenderer$EncoderCallback.processComponent(PanelGroupLayoutRenderer.java:582)
    oracle.adfinternal.view.faces.renderkit.rich.PanelGroupLayoutRenderer$EncoderCallback.processComponent(PanelGroupLayoutRenderer.java:573)
    org.apache.myfaces.trinidad.component.UIXComponent.processFlattenedChildren(UIXComponent.java:154)
    org.apache.myfaces.trinidad.component.UIXComponent.processFlattenedChildren(UIXComponent.java:285)
    org.apache.myfaces.trinidad.component.UIXComponent.encodeFlattenedChildren(UIXComponent.java:252)
    oracle.adfinternal.view.faces.renderkit.rich.PanelGroupLayoutRenderer.encodeAll(PanelGroupLayoutRenderer.java:236)
    oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1252)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:326)
    org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:758)
    javax.faces.component.UIComponent.encodeAll(UIComponent.java:920)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:398)
    oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2608)
    oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer.encodeCenterFacet(PanelStretchLayoutRenderer.java:769)
    oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer._encodeCenterPane(PanelStretchLayoutRenderer.java:1055)
    oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer._encodeMiddlePanes(PanelStretchLayoutRenderer.java:345)
    oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer.encodeAll(PanelStretchLayoutRenderer.java:122)
    oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1252)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:326)
    org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:758)
    javax.faces.component.UIComponent.encodeAll(UIComponent.java:920)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:398)
    oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2608)
    oracle.adf.view.rich.render.RichRenderer.encodeStretchedChild(RichRenderer.java:1998)
    oracle.adfinternal.view.faces.renderkit.rich.RegionRenderer.access$400(RegionRenderer.java:46)
    oracle.adfinternal.view.faces.renderkit.rich.RegionRenderer$ChildEncoderCallback.processComponent(RegionRenderer.java:562)
    oracle.adfinternal.view.faces.renderkit.rich.RegionRenderer$ChildEncoderCallback.processComponent(RegionRenderer.java:553)
    org.apache.myfaces.trinidad.component.UIXComponent.processFlattenedChildren(UIXComponent.java:154)
    org.apache.myfaces.trinidad.component.UIXComponent.processFlattenedChildren(UIXComponent.java:285)
    org.apache.myfaces.trinidad.component.UIXComponent.encodeFlattenedChildren(UIXComponent.java:252)
    oracle.adfinternal.view.faces.renderkit.rich.RegionRenderer._encodeChildren(RegionRenderer.java:229)
    oracle.adfinternal.view.faces.renderkit.rich.RegionRenderer.encodeAll(RegionRenderer.java:144)
    oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1252)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:326)
    org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:758)
    oracle.adf.view.rich.component.fragment.UIXRegion.encodeEnd(UIXRegion.java:292)
    javax.faces.component.UIComponent.encodeAll(UIComponent.java:920)
    oracle.adfinternal.view.faces.taglib.region.IncludeTag$FacetWrapper.encodeAll(IncludeTag.java:511)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:398)
    oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2608)
    oracle.adfinternal.view.faces.renderkit.rich.DecorativeBoxRenderer.encodeFacet(DecorativeBoxRenderer.java:361)
    oracle.adfinternal.view.faces.renderkit.rich.DecorativeBoxRenderer._encodeCenterPane(DecorativeBoxRenderer.java:529)
    oracle.adfinternal.view.faces.renderkit.rich.DecorativeBoxRenderer.encodeAll(DecorativeBoxRenderer.java:167)
    oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1252)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:326)
    org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:758)
    javax.faces.component.UIComponent.encodeAll(UIComponent.java:920)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:398)
    oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2608)
    oracle.adf.view.rich.render.RichRenderer.encodeStretchedChild(RichRenderer.java:1998)
    oracle.adfinternal.view.faces.renderkit.rich.PanelSplitterRenderer._renderPane(PanelSplitterRenderer.java:1267)
    oracle.adfinternal.view.faces.renderkit.rich.PanelSplitterRenderer.encodeAll(PanelSplitterRenderer.java:88)
    oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1252)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:326)
    org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:758)
    javax.faces.component.UIComponent.encodeAll(UIComponent.java:920)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:398)
    oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2608)
    oracle.adf.view.rich.render.RichRenderer.encodeStretchedChild(RichRenderer.java:1998)
    oracle.adfinternal.view.faces.renderkit.rich.PanelSplitterRenderer._renderPane(PanelSplitterRenderer.java:1267)
    oracle.adfinternal.view.faces.renderkit.rich.PanelSplitterRenderer.encodeAll(PanelSplitterRenderer.java:88)
    oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1252)
    oracle.adfinternal.view.faces.renderkit.rich.PanelSplitterRenderer.encodeAll(PanelSplitterRenderer.java:88)
    oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1252)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:326)
    org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:758)
    javax.faces.component.UIComponent.encodeAll(UIComponent.java:920)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:398)
    oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2608)
    oracle.adfinternal.view.faces.renderkit.rich.DecorativeBoxRenderer.encodeFacet(DecorativeBoxRenderer.java:361)
    oracle.adfinternal.view.faces.renderkit.rich.DecorativeBoxRenderer._encodeCenterPane(DecorativeBoxRenderer.java:529)
    oracle.adfinternal.view.faces.renderkit.rich.DecorativeBoxRenderer.encodeAll(DecorativeBoxRenderer.java:167)
    oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1252)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:326)
    org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:758)
    javax.faces.component.UIComponent.encodeAll(UIComponent.java:920)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:398)
    oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2608)
    oracle.adf.view.rich.render.RichRenderer.encodeStretchedChild(RichRenderer.java:1998)
    oracle.adfinternal.view.faces.renderkit.rich.PanelSplitterRenderer._renderPane(PanelSplitterRenderer.java:1267)
    oracle.adfinternal.view.faces.renderkit.rich.PanelSplitterRenderer.encodeAll(PanelSplitterRenderer.java:88)
    oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1252)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:326)
    org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:758)
    javax.faces.component.UIComponent.encodeAll(UIComponent.java:920)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:398)
    oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2608)
    oracle.adf.view.rich.render.RichRenderer.encodeStretchedChild(RichRenderer.java:1998)
    oracle.adfinternal.view.faces.renderkit.rich.PanelSplitterRenderer._renderPane(PanelSplitterRenderer.java:1267)
    oracle.adfinternal.view.faces.renderkit.rich.PanelSplitterRenderer.encodeAll(PanelSplitterRenderer.java:88)
    oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1252)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:326)
    org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:758)
    javax.faces.component.UIComponent.encodeAll(UIComponent.java:920)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:398)
    oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2608)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeAllChildren(CoreRenderer.java:407)
    oracle.adfinternal.view.faces.renderkit.rich.PageTemplateRenderer.encodeAll(PageTemplateRenderer.java:69)
    oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1252)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:326)
    org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:758)
    oracle.adf.view.rich.component.fragment.ContextSwitchingComponent.encodeEnd(ContextSwitchingComponent.java:155)
    javax.faces.component.UIComponent.encodeAll(UIComponent.java:920)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:398)
    oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2608)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeAllChildren(CoreRenderer.java:407)
    oracle.adfinternal.view.faces.renderkit.rich.FormRenderer.encodeAll(FormRenderer.java:158)
    oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1252)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:326)
    org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:758)
    javax.faces.component.UIComponent.encodeAll(UIComponent.java:920)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:398)
    oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2608)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeAllChildren(CoreRenderer.java:407)
    oracle.adfinternal.view.faces.renderkit.rich.DocumentRenderer.encodeAll(DocumentRenderer.java:725)
    oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1252)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:326)
    org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:758)
    javax.faces.component.UIComponent.encodeAll(UIComponent.java:920)
    javax.faces.component.UIComponent.encodeAll(UIComponent.java:920)
    com.sun.faces.application.ViewHandlerImpl.doRenderView(ViewHandlerImpl.java:256)
    com.sun.faces.application.ViewHandlerImpl.renderView(ViewHandlerImpl.java:131)
    javax.faces.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:189)
    org.apache.myfaces.trinidadinternal.application.ViewHandlerImpl.renderView(ViewHandlerImpl.java:155)
    oracle.adfinternal.view.faces.lifecycle.LifecycleImpl._renderResponse(LifecycleImpl.java:773)
    oracle.adfinternal.view.faces.lifecycle.LifecycleImpl._executePhase(LifecycleImpl.java:252)
    oracle.adfinternal.view.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:207)
    javax.faces.webapp.FacesServlet.service(FacesServlet.java:246)
    oracle.adfinternal.view.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:207)
    javax.faces.webapp.FacesServlet.service(FacesServlet.java:246)
    weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
    weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
    weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:206)
    weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    oracle.adf.model.servlet.ADFBindingFilter.doFilter(ADFBindingFilter.java:147)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    oracle.adfinternal.view.faces.webapp.rich.RegistrationFilter.doFilter(RegistrationFilter.java:69)
    org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:437)
    oracle.adfinternal.view.faces.activedata.AdsFilter.doFilter(AdsFilter.java:44)
    org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:437)
    org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl._doFilterImpl(TrinidadFilterImpl.java:231)
    org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl.doFilter(TrinidadFilterImpl.java:133)
    org.apache.myfaces.trinidad.webapp.TrinidadFilter.doFilter(TrinidadFilter.java:91)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    oracle.adf.library.webapp.LibraryFilter.doFilter(LibraryFilter.java:163)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:111)
    oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:299)
    oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:405)
    oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:94)
    oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:98)
    oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:70)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:86)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:25)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3687)
    weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3681)
    weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:308)
    weblogic.security.service.SecurityManager.runAs(SecurityManager.java:116)
    weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2213)
    weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2135)
    weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1420)
    weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
    weblogic.work.ExecuteThread.run(ExecuteThread.java:168)
    Please help.
    Thomas

    Hi Thomas,
    Your Thread is showing a processor computing operation ArrayList.clone() triggered from the Oracle JBO framework / API.
    Few questions for you:
    -     Are you using Solaris OS? If yes can you please capture the PRSTAT at the same time as your JVM Thread Dump using command:
    prstat -L 0 1 | grep <java pid>;kill -3 <java pid>
    -     How many threads are showing this condition?
    -     Was CPU utilization of the affected Java process very high on the server at that time?
    -     What is the utilization of your Java Heap at the time of the Thread Dump capture?
    This Thread could be stuck in an infinite loop since it is stuck for at least 600 seconds (10 minutes). It would probably help to generate a few more snapshots earlier and understand if Thread is hanging in the same code portion (possibility Oracle JBO) for every snapshot or not.
    This could also be data related (too many data rows) returned and causing high processing time for the clone operation from the Myfaces / JSF table that you are trying to render.
    Regards,
    P-H
    http://javaeesupportpatterns.blogspot.com/

  • Stuck thread in Oracle Weblogic server

    Dear All,
    We have a stuck thread in Oracle WLS 10.3.3. We have deployed 7 ear files in 5th managed server(psMS5). We have stuck thread in one of the application (psinsurance). We have allocated 1GB heap space for the psMS5 managed server. We are using this in Development environment. Please check the below message from psMS5 managed server log file. Kindly provide reason for stuck thread and solution.
    WatchTime: Sep 23, 2011 10:28:29 AM IST
    WatchDomainName: dev_domain
    WatchServerName: psMS5
    WatchSeverityLevel: Warning
    WatchName: MS5-Watch
    WatchRuleType: Log
    WatchRule: ((MESSAGE LIKE '%has been busy for "%') AND (MSGID = 'BEA-000337') AND (SERVER = 'psMS5'))
    WatchData: DATE = Sep 23, 2011 10:28:28 AM IST SERVER = psMS5 MESSAGE = [STUCK] ExecuteThread: '22' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "629" seconds working on the request "weblogic.servlet.internal.ServletRequestImpl@1cac6525[
    POST */psinsurance/app/lending/insurance/PS00738D.jspx*?_adf.ctrl-state=24f0li74e_4 HTTP/1.1
    Accept: image/jpeg, application/x-ms-application, image/gif, application/xaml+xml, image/pjpeg, application/x-ms-xbap, */*
    Referer: http://sis-prosun11/psinsurance/faces/PS00003?jsessionid=r2BrT8TpqZBsnLhCw2Y8KDpTV7hMpgnLDFW54P87ZpHwn8JMy88s%21168932459%211316753022074&_afrWindowMode=0&_afrLoop=244724092027000&_adf.ctrl-state=24f0li74e_4
    Accept-Language: en-US
    User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; Tablet PC 2.0)
    Content-Type: application/x-www-form-urlencoded
    Accept-Encoding: gzip, deflate
    Content-Length: 554
    Cache-Control: no-cache
    Cookie: JSESSIONID=LkJcT8TfMT64SyQJT4Ls4sXhfDl0pTS7CXJVQ9KcryHGWJ1dhyfz!1373322659
    Connection: Keep-Alive
    Proxy-Client-IP: 192.168.9.179
    X-Forwarded-For: 192.168.9.179
    X-WebLogic-KeepAliveSecs: 30
    X-WebLogic-Force-JVMID: 1373322659
    ]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
    Thread-237 "[STUCK] ExecuteThread: '22' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, suspended, priority=1, DAEMON> {
    java.io.UnixFileSystem.getBooleanAttributes0(UnixFileSystem.java:???)
    java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:228)
    java.io.File.exists(File.java:729)
    weblogic.utils.classloaders.ClasspathClassFinder.getFileSource(ClasspathClassFinder.java:383)
    weblogic.utils.classloaders.ClasspathClassFinder.getSourcesInternal(ClasspathClassFinder.java:333)
    weblogic.utils.classloaders.ClasspathClassFinder.getSource(ClasspathClassFinder.java:316)
    weblogic.utils.classloaders.JarClassFinder.getSource(JarClassFinder.java:58)
    weblogic.utils.classloaders.AbstractClassFinder.getClassSource(AbstractClassFinder.java:22)
    weblogic.utils.classloaders.MultiClassFinder.getClassSource(MultiClassFinder.java:56)
    weblogic.utils.classloaders.CodeGenClassFinder.getClassSource(CodeGenClassFinder.java:26)
    weblogic.utils.classloaders.GenericClassLoader.findLocalClass(GenericClassLoader.java:268)
    *^-- Holding lock: weblogic.utils.classloaders.GenericClassLoader@1e3295a1[thin lock]*
    weblogic.utils.classloaders.GenericClassLoader.findClass(GenericClassLoader.java:237)
    java.lang.ClassLoader.loadClass(ClassLoader.java:292)
    *^-- Holding lock: weblogic.utils.classloaders.GenericClassLoader@1e3295a1[thin lock]*
    java.lang.ClassLoader.loadClass(ClassLoader.java:292)
    java.lang.ClassLoader.loadClass(ClassLoader.java:248)
    weblogic.utils.classloaders.GenericClassLoader.loadClass(GenericClassLoader.java:177)
    weblogic.utils.classloaders.ChangeAwareClassLoader.loadClass(ChangeAwareClassLoader.java:37)
    *^-- Holding lock: weblogic.utils.classloaders.ChangeAwareClassLoader@1e553474[thin lock]*
    java.lang.Class.getDeclaredConstructors0(Class.java:???)
    java.lang.Class.privateGetDeclaredConstructors(Class.java:2370)
    java.lang.Class.getConstructor0(Class.java:2699)
    java.lang.Class.newInstance0(Class.java:318)
    java.lang.Class.newInstance(Class.java:305)
    org.apache.myfaces.trinidadinternal.renderkit.ClassRendererInstantiator.instantiate(ClassRendererInstantiator.java:48)
    org.apache.myfaces.trinidadinternal.renderkit.RenderKitBase.findRenderer(RenderKitBase.java:146)
    org.apache.myfaces.trinidadinternal.renderkit.RenderKitDecorator.findRenderer(RenderKitDecorator.java:84)
    oracle.adfinternal.view.faces.unified.renderkit.UnifiedRenderKit._getRenderer(UnifiedRenderKit.java:218)
    oracle.adfinternal.view.faces.unified.renderkit.UnifiedRenderKit.findRenderer(UnifiedRenderKit.java:156)
    org.apache.myfaces.trinidadinternal.renderkit.RenderKitBase.getRenderer(RenderKitBase.java:129)
    org.apache.myfaces.trinidad.component.UIXComponentBase._getRendererImpl(UIXComponentBase.java:1143)
    org.apache.myfaces.trinidad.component.UIXComponentBase.getRenderer(UIXComponentBase.java:1173)
    org.apache.myfaces.trinidad.component.UIXComponentBase.getClientId(UIXComponentBase.java:316)
    javax.faces.webapp.UIComponentClassicTagBase.doStartTag(UIComponentClassicTagBase.java:1095)
    org.apache.myfaces.trinidad.webapp.UIXComponentELTag.doStartTag(UIXComponentELTag.java:70)
    oracle.adfinternal.view.faces.unified.taglib.input.UnifiedSelectBooleanCheckboxTag.doStartTag(UnifiedSelectBooleanCheckboxTag.java:45)
    oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
    oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
    oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
    oracle.jsp.runtime.tree.OracleJspIterationTagNode.executeHandler(OracleJspIterationTagNode.java:41)
    oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
    oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
    oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
    oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
    oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
    oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
    oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
    oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
    oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
    oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
    oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
    oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
    oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
    oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
    oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
    oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
    oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
    oracle.jsp.runtime.tree.OracleJspIterationTagNode.executeHandler(OracleJspIterationTagNode.java:41)
    oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
    oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
    oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
    oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
    oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
    oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
    oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
    oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
    oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
    oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
    oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:83)
    oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:42)
    oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:243)
    oracle.jsp.runtime.tree.OracleJspNode.execute(OracleJspNode.java:73)
    oracle.jsp.runtimev2.ShortCutServlet._jspService(ShortCutServlet.java:57)
    oracle.jsp.runtime.OracleJspBase.service(OracleJspBase.java:29)
    oracle.jsp.runtimev2.JspPageTable.compileAndServe(JspPageTable.java:650)
    oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.java:330)
    oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.java:680)
    oracle.jsp.runtimev2.JspServlet.service(JspServlet.java:653)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:815)
    weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
    weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
    weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:206)
    weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    oracle.adf.library.webapp.LibraryFilter.doFilter(LibraryFilter.java:147)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    weblogic.servlet.internal.RequestDispatcherImpl.invokeServlet(RequestDispatcherImpl.java:520)
    weblogic.servlet.internal.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:112)
    com.sun.faces.context.ExternalContextImpl.dispatch(ExternalContextImpl.java:402)
    org.apache.myfaces.trinidad.context.ExternalContextDecorator.dispatch(ExternalContextDecorator.java:44)
    org.apache.myfaces.trinidad.context.ExternalContextDecorator.dispatch(ExternalContextDecorator.java:44)
    org.apache.myfaces.trinidad.context.ExternalContextDecorator.dispatch(ExternalContextDecorator.java:44)
    org.apache.myfaces.trinidad.context.ExternalContextDecorator.dispatch(ExternalContextDecorator.java:44)
    org.apache.myfaces.trinidadinternal.context.FacesContextFactoryImpl$OverrideDispatch.dispatch(FacesContextFactoryImpl.java:261)
    com.sun.faces.application.ViewHandlerImpl.executePageToBuildView(ViewHandlerImpl.java:428)
    com.sun.faces.application.ViewHandlerImpl.renderView(ViewHandlerImpl.java:132)
    javax.faces.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:189)
    org.apache.myfaces.trinidadinternal.application.ViewHandlerImpl.renderView(ViewHandlerImpl.java:155)
    oracle.adfinternal.view.faces.lifecycle.LifecycleImpl._renderResponse(LifecycleImpl.java:706)
    oracle.adfinternal.view.faces.lifecycle.LifecycleImpl._executePhase(LifecycleImpl.java:244)
    oracle.adfinternal.view.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:199)
    javax.faces.webapp.FacesServlet.service(FacesServlet.java:246)
    weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
    weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
    weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:206)
    weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    oracle.adf.model.servlet.ADFBindingFilter.doFilter(ADFBindingFilter.java:134)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    oracle.adfinternal.view.faces.webapp.rich.RegistrationFilter.doFilter(RegistrationFilter.java:69)
    org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:411)
    oracle.adfinternal.view.faces.activedata.AdsFilter.doFilter(AdsFilter.java:44)
    org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:411)
    org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl._doFilterImpl(TrinidadFilterImpl.java:210)
    org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl.doFilter(TrinidadFilterImpl.java:126)
    org.apache.myfaces.trinidad.webapp.TrinidadFilter.doFilter(TrinidadFilter.java:91)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:94)
    oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:299)
    oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:406)
    oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:82)
    oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:70)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    oracle.adf.library.webapp.LibraryFilter.doFilter(LibraryFilter.java:147)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    oracle.dms.wls.DMSServletFilter.doFilter(DMSServletFilter.java:114)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:25)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.doIt(WebAppServletContext.java:3656)
    weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3650)
    weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:308)
    weblogic.security.service.SecurityManager.runAs(SecurityManager.java:117)
    weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2204)
    weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2126)
    weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1412)
    weblogic.work.ExecuteThread.execute(ExecuteThread.java:198)
    weblogic.work.ExecuteThread.run(ExecuteThread.java:165)
    SUBSYSTEM = WebLogicServer USERID = <WLS Kernel> SEVERITY = Error THREAD = [ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)' MSGID = BEA-000337 MACHINE = sis-prosun11.sis.ad TXID = CONTEXTID = 11d1def534ea1be0:3bbe257e:13286031cbf:-7ffd-0000000000002fe7 TIMESTAMP = 1316753908672
    WatchAlarmType: AutomaticReset
    WatchAlarmResetPeriod: 60000
    SMTPNotificationName: MS5-Health-Notification

    I found something interesting to the question I posted from the below link:
    http://www.mail-archive.com/[email protected]/msg03217.html

  • Help on stuck thread

    Hi, From my server log I noticed that there are some stuck threads. After looked at the thread dump thread 18 is blocking. MainPortalServlet.java is extends PortalServlet and line 96 is super.service(request,response); Please help me to analyse why it is blocking. Thanks.
    "ExecuteThread: '20' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x0090e6b0 nid=0x3b waiting for monitor entry [0x600fd000..0x600ffc28]
         at com.bea.wlw.netui.script.el.util.ParseUtils.parse(ParseUtils.java:100)
         - waiting to lock <0x8d3a3f18> (a com.bea.wlw.netui.script.el.util.ParseUtils$Cache)
         at com.bea.wlw.netui.script.el.ExpressionEvaluatorImpl.containsExpression(ExpressionEvaluatorImpl.java:239)
         at com.bea.wlw.netui.pageflow.ProcessPopulate.populate(ProcessPopulate.java:295)
         at com.bea.wlw.netui.pageflow.PageFlowRequestProcessor.processPopulate(PageFlowRequestProcessor.java:402)
         at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:254)
         at com.bea.wlw.netui.pageflow.PageFlowRequestProcessor.process(PageFlowRequestProcessor.java:691)
         at com.bea.wlw.netui.pageflow.AutoRegisterActionServlet.process(AutoRegisterActionServlet.java:527)
         at com.bea.wlw.netui.pageflow.PageFlowActionServlet.process(PageFlowActionServlet.java:152)
         at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:507)
         at com.bea.wlw.netui.pageflow.PageFlowUtils.strutsLookup(PageFlowUtils.java:1786)
         at com.bea.portlet.adapter.scopedcontent.ScopedContentCommonSupport.executeAction(ScopedContentCommonSupport.java:561)
         at com.bea.portlet.adapter.scopedcontent.ScopedContentCommonSupport.processActionInternal(ScopedContentCommonSupport.java:121)
         at com.bea.portlet.adapter.scopedcontent.PageFlowStubImpl.processAction(PageFlowStubImpl.java:98)
         at com.bea.netuix.servlets.controls.content.NetuiContent.raiseScopedAction(NetuiContent.java:154)
         at com.bea.netuix.servlets.controls.content.NetuiContent.raiseScopedAction(NetuiContent.java:113)
         at com.bea.netuix.servlets.controls.content.NetuiContent.handlePostbackData(NetuiContent.java:228)
         at com.bea.netuix.nf.ControlLifecycle$3.visit(ControlLifecycle.java:171)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:356)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:126)
         at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:105)
         at com.bea.netuix.nf.Lifecycle.runInbound(Lifecycle.java:173)
         at com.bea.netuix.nf.Lifecycle.run(Lifecycle.java:137)
         at com.bea.netuix.servlets.manager.UIServlet.runLifecycle(UIServlet.java:324)
         at com.bea.netuix.servlets.manager.UIServlet.processControlTree(UIServlet.java:220)
         at com.bea.netuix.servlets.manager.PortalServlet.doPost(PortalServlet.java:820)
         at com.bea.netuix.servlets.manager.UIServlet.service(UIServlet.java:150)
         at com.cd.servlets.portal.MainPortalServlet.service(MainPortalServlet.java:96)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
         at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
         at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:28)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
         at com.bea.p13n.servlets.PortalServletFilter.doFilter(PortalServletFilter.java:293)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
         at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:7053)
         at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
         at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
         at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3902)
         at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2773)
         at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)
    "ExecuteThread: '19' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x0090dab8 nid=0x3a runnable [0x601fb000..0x601ffc28]
         at java.util.HashMap.getEntry(HashMap.java:361)
         at java.util.LinkedHashMap.get(LinkedHashMap.java:266)
         at com.bea.wlw.netui.script.el.util.ParseUtils.parse(ParseUtils.java:84)
         at com.bea.wlw.netui.script.el.util.ParseUtils.evaluate(ParseUtils.java:123)
         at com.bea.wlw.netui.script.el.ExpressionEvaluatorImpl.evaluateStrict(ExpressionEvaluatorImpl.java:87)
         at com.bea.wlw.netui.tags.AbstractBaseTag.evaluateExpressionInternal(AbstractBaseTag.java:573)
         at com.bea.wlw.netui.tags.AbstractBaseTag.evaluateExpression(AbstractBaseTag.java:362)
         at com.bea.wlw.netui.tags.databinding.script.GetData.doEndTag(GetData.java:249)
         at jsp_servlet._portlets.___portlet1._jspService(_portlet1.jsp:27)
         at weblogic.servlet.jsp.JspBase.service(JspBase.java:33)
         at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
         at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:28)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
         at com.bea.p13n.servlets.PortalServletFilter.doFilter(PortalServletFilter.java:293)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
         at com.bea.wlw.netui.pageflow.PageFlowJspFilter.doFilter(PageFlowJspFilter.java:250)
         - locked <0x78ccf6c8> (a portlets.Portlet1Controller)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
         at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:652)
         at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:431)
         at com.bea.wlw.netui.pageflow.scoping.internal.ScopedRequestDispatcher.include(ScopedRequestDispatcher.java:120)
         at com.bea.netuix.servlets.controls.content.JspContent.beginRender(JspContent.java:534)
         at com.bea.netuix.servlets.controls.content.NetuiContent.beginRender(NetuiContent.java:425)
         at com.bea.netuix.nf.ControlLifecycle$1.visit(ControlLifecycle.java:495)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:543)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:247)
         at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:105)
         at com.bea.netuix.nf.Lifecycle.run(Lifecycle.java:356)
         at com.bea.netuix.nf.UIControl.render(UIControl.java:536)
         at com.bea.netuix.servlets.controls.PresentationContext.render(PresentationContext.java:405)
         at com.bea.netuix.servlets.util.RenderToolkit.renderChild(RenderToolkit.java:123)
         at com.bea.netuix.servlets.jsp.taglib.RenderChild.doStartTag(RenderChild.java:58)
         at jsp_servlet._framework._skeletons._cd.__flowlayout._jspService(flowlayout.jsp:30)
         at weblogic.servlet.jsp.JspBase.service(JspBase.java:33)
         at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:348)
         at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:646)
         at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:431)
         at com.bea.netuix.servlets.controls.JspRenderer.renderAlt(JspRenderer.java:194)
         at com.bea.netuix.servlets.controls.JspRenderer.beginRender(JspRenderer.java:96)
         at com.bea.netuix.nf.ControlLifecycle$1.visit(ControlLifecycle.java:491)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:543)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:247)
         at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:105)
         at com.bea.netuix.nf.Lifecycle.run(Lifecycle.java:356)
         at com.bea.netuix.nf.UIControl.render(UIControl.java:536)
         at com.bea.netuix.servlets.controls.PresentationContext.render(PresentationContext.java:405)
         at com.bea.netuix.servlets.util.RenderToolkit.renderChild(RenderToolkit.java:123)
         at com.bea.netuix.servlets.jsp.taglib.RenderChild.doStartTag(RenderChild.java:58)
         at jsp_servlet._framework._skeletons._cd.__flowlayout._jspService(flowlayout.jsp:30)
         at weblogic.servlet.jsp.JspBase.service(JspBase.java:33)
         at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:348)
         at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:646)
         at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:431)
         at com.bea.netuix.servlets.controls.JspRenderer.renderAlt(JspRenderer.java:194)
         at com.bea.netuix.servlets.controls.JspRenderer.beginRender(JspRenderer.java:96)
         at com.bea.netuix.nf.ControlLifecycle$1.visit(ControlLifecycle.java:491)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:543)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:247)
         at com.bea.netuix.nf.Lifecycle.runOutbound(Lifecycle.java:204)
         at com.bea.netuix.nf.Lifecycle.run(Lifecycle.java:153)
         at com.bea.netuix.servlets.manager.UIServlet.runLifecycle(UIServlet.java:324)
         at com.bea.netuix.servlets.manager.UIServlet.processControlTree(UIServlet.java:220)
         at com.bea.netuix.servlets.manager.PortalServlet.doPost(PortalServlet.java:820)
         at com.bea.netuix.servlets.manager.PortalServlet.doGet(PortalServlet.java:671)
         at com.bea.netuix.servlets.manager.UIServlet.service(UIServlet.java:147)
         at com.cd.servlets.portal.MainPortalServlet.service(MainPortalServlet.java:116)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
         at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
         at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:28)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
         at com.bea.p13n.servlets.PortalServletFilter.doFilter(PortalServletFilter.java:293)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
         at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:7053)
         at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
         at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
         at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3902)
         at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2773)
         at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)
    "ExecuteThread: '18' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x0090cec0 nid=0x39 runnable [0x602fd000..0x602ffc28]
         at java.util.LinkedHashMap.transfer(LinkedHashMap.java:224)
         at java.util.HashMap.resize(HashMap.java:452)
         at java.util.LinkedHashMap.addEntry(LinkedHashMap.java:399)
         at java.util.HashMap.put(HashMap.java:392)
         at com.bea.wlw.netui.script.el.util.ParseUtils.parse(ParseUtils.java:100)
         - locked <0x8d3a3f18> (a com.bea.wlw.netui.script.el.util.ParseUtils$Cache)
         at com.bea.wlw.netui.script.el.ExpressionEvaluatorImpl.containsExpression(ExpressionEvaluatorImpl.java:239)
         at com.bea.wlw.netui.pageflow.ProcessPopulate.populate(ProcessPopulate.java:295)
         at com.bea.wlw.netui.pageflow.PageFlowRequestProcessor.processPopulate(PageFlowRequestProcessor.java:402)
         at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:254)
         at com.bea.wlw.netui.pageflow.PageFlowRequestProcessor.process(PageFlowRequestProcessor.java:691)
         at com.bea.wlw.netui.pageflow.AutoRegisterActionServlet.process(AutoRegisterActionServlet.java:527)
         at com.bea.wlw.netui.pageflow.PageFlowActionServlet.process(PageFlowActionServlet.java:152)
         at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:507)
         at com.bea.wlw.netui.pageflow.PageFlowUtils.strutsLookup(PageFlowUtils.java:1786)
         at com.bea.portlet.adapter.scopedcontent.ScopedContentCommonSupport.executeAction(ScopedContentCommonSupport.java:561)
         at com.bea.portlet.adapter.scopedcontent.ScopedContentCommonSupport.processActionInternal(ScopedContentCommonSupport.java:121)
         at com.bea.portlet.adapter.scopedcontent.PageFlowStubImpl.processAction(PageFlowStubImpl.java:98)
         at com.bea.netuix.servlets.controls.content.NetuiContent.raiseScopedAction(NetuiContent.java:154)
         at com.bea.netuix.servlets.controls.content.NetuiContent.raiseScopedAction(NetuiContent.java:113)
         at com.bea.netuix.servlets.controls.content.NetuiContent.handlePostbackData(NetuiContent.java:228)
         at com.bea.netuix.nf.ControlLifecycle$3.visit(ControlLifecycle.java:171)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:356)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:126)
         at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:105)
         at com.bea.netuix.nf.Lifecycle.runInbound(Lifecycle.java:173)
         at com.bea.netuix.nf.Lifecycle.run(Lifecycle.java:137)
         at com.bea.netuix.servlets.manager.UIServlet.runLifecycle(UIServlet.java:324)
         at com.bea.netuix.servlets.manager.UIServlet.processControlTree(UIServlet.java:220)
         at com.bea.netuix.servlets.manager.PortalServlet.doPost(PortalServlet.java:820)
         at com.bea.netuix.servlets.manager.UIServlet.service(UIServlet.java:150)
         at com.cd.servlets.portal.MainPortalServlet.service(MainPortalServlet.java:96)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
         at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
         at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:28)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
         at com.bea.p13n.servlets.PortalServletFilter.doFilter(PortalServletFilter.java:293)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
         at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:7053)
         at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
         at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
         at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3902)
         at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2773)
         at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)

    Hi user9503787,
    Can you please confirm if the Weblogic instance / java process you took the Thread Dump from was showing high CPU utilization?
    Your Thread Dump data is quite interesting and a perfect example of a non Thread safe HashMap combined with a read/write race condition:
    Thread #23 (hanging for 600 seconds+) is showing a potential infinite looping condition in at java.util.HashMap.getEntry(HashMap.java:361)
    Thread #22 (hanging for 600 seconds+) is showing a potential infinite looping condition while performing a resize() following a put() operation
    Weblogic Portal has some known issues which regards to usage of non Thread safe HashHap data structures; which under load can cause infinite looping and sharp surge of CPU (due to corruption of the internal HashMap index preventing exiting the internal loop); ultimately hanging the whole JVM.
    We had the same issue for one of our Weblogic Portal 10.0 production environment and Oracle provided a patch (via Collections.synchronizedMap() ).
    Please open a SR and Oracle, share these 2 Threads and discuss with the engineer about a patch / know issues regarding non Thread safe HashMap and infinite looping.
    Regards,
    P-H
    http://javaeesupportpatterns.blogspot.com/

  • Stuck threads reading blob column from db table

    WLS 10.3.5, JDK 1.6u29, Oracle 11g RAC, ojdbc6 latest driver
    We're having problems with stuck threads trying to read a blob column from a DB table. The query to extract the blob is a simple select, without any locking such as "for update" clauses or whatever. The blob's size is <= 100k.
    The thread dump shows the following stack trace:
    +"[STUCK] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" RUNNABLE native+
    +     java.net.SocketInputStream.socketRead0(Native Method)+
    +     java.net.SocketInputStream.read(SocketInputStream.java:129)+
    +     oracle.net.ns.Packet.receive(Packet.java:300)+
    +     oracle.net.ns.DataPacket.receive(DataPacket.java:106)+
    +     oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:315)+
    +     oracle.net.ns.NetInputStream.read(NetInputStream.java:260)+
    +     oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:105)+
    +     oracle.jdbc.driver.T4CMAREngine.getNBytes(T4CMAREngine.java:1517)+
    +     oracle.jdbc.driver.T4C8TTILobd.unmarshalLobData(T4C8TTILobd.java:476)+
    +     oracle.jdbc.driver.T4C8TTILob.readLOBD(T4C8TTILob.java:770)+
    +     oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:361)+
    +     oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)+
    +     oracle.jdbc.driver.T4C8TTILob.read(T4C8TTILob.java:146)+
    +     oracle.jdbc.driver.T4CConnection.getBytes(T4CConnection.java:2392)+
    +     oracle.sql.BLOB.getBytes(BLOB.java:348)+
    +     oracle.sql.BLOB.getBytes(BLOB.java:222)+
    +     weblogic.jdbc.wrapper.Blob_oracle_sql_BLOB.getBytes(Unknown Source)+
    +     com.ibatis.sqlmap.engine.type.BlobTypeHandlerCallback.getResult(BlobTypeHandlerCallback.java:33)+
    +     com.ibatis.sqlmap.engine.type.CustomTypeHandler.getResult(CustomTypeHandler.java:52)+
    +     com.ibatis.sqlmap.engine.mapping.result.ResultMap.getPrimitiveResultMappingValue(ResultMap.java:619)+
    +     com.ibatis.sqlmap.engine.mapping.result.ResultMap.getResults(ResultMap.java:345)+
    +     com.ibatis.sqlmap.engine.execution.SqlExecutor.handleResults(SqlExecutor.java:384)+
    +     com.ibatis.sqlmap.engine.execution.SqlExecutor.handleMultipleResults(SqlExecutor.java:300)+
    +     com.ibatis.sqlmap.engine.execution.SqlExecutor.executeQuery(SqlExecutor.java:189)+
    +     com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.sqlExecuteQuery(MappedStatement.java:221)+
    +     com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.executeQueryWithCallback(MappedStatement.java:189)+
    +     com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.executeQueryForList(MappedStatement.java:139)+
    +     com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryForList(SqlMapExecutorDelegate.java:567)+
    +     com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryForList(SqlMapExecutorDelegate.java:541)+
    +     com.ibatis.sqlmap.engine.impl.SqlMapSessionImpl.queryForList(SqlMapSessionImpl.java:118)+
    +     com.ibatis.sqlmap.engine.impl.SqlMapClientImpl.queryForList(SqlMapClientImpl.java:94)+
    +     com.ibatis.dao.client.template.SqlMapDaoTemplate.queryForList(SqlMapDaoTemplate.java:282)"[STUCK] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" RUNNABLE native+
    +     java.net.SocketInputStream.socketRead0(Native Method)+
    +     java.net.SocketInputStream.read(SocketInputStream.java:129)+
    +     oracle.net.ns.Packet.receive(Packet.java:300)+
    +     oracle.net.ns.DataPacket.receive(DataPacket.java:106)+
    +     oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:315)+
    +     oracle.net.ns.NetInputStream.read(NetInputStream.java:260)+
    +     oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:105)+
    +     oracle.jdbc.driver.T4CMAREngine.getNBytes(T4CMAREngine.java:1517)+
    +     oracle.jdbc.driver.T4C8TTILobd.unmarshalLobData(T4C8TTILobd.java:476)+
    +     oracle.jdbc.driver.T4C8TTILob.readLOBD(T4C8TTILob.java:770)+
    +     oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:361)+
    +     oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)+
    +     oracle.jdbc.driver.T4C8TTILob.read(T4C8TTILob.java:146)+
    +     oracle.jdbc.driver.T4CConnection.getBytes(T4CConnection.java:2392)+
    +     oracle.sql.BLOB.getBytes(BLOB.java:348)+
    +     oracle.sql.BLOB.getBytes(BLOB.java:222)+
    +     weblogic.jdbc.wrapper.Blob_oracle_sql_BLOB.getBytes(Unknown Source)+
    +     com.ibatis.sqlmap.engine.type.BlobTypeHandlerCallback.getResult(BlobTypeHandlerCallback.java:33)+
    +     com.ibatis.sqlmap.engine.type.CustomTypeHandler.getResult(CustomTypeHandler.java:52)+
    +     com.ibatis.sqlmap.engine.mapping.result.ResultMap.getPrimitiveResultMappingValue(ResultMap.java:619)+
    +     com.ibatis.sqlmap.engine.mapping.result.ResultMap.getResults(ResultMap.java:345)+
    +     com.ibatis.sqlmap.engine.execution.SqlExecutor.handleResults(SqlExecutor.java:384)+
    +     com.ibatis.sqlmap.engine.execution.SqlExecutor.handleMultipleResults(SqlExecutor.java:300)+
    +     com.ibatis.sqlmap.engine.execution.SqlExecutor.executeQuery(SqlExecutor.java:189)+
    +     com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.sqlExecuteQuery(MappedStatement.java:221)+
         com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.executeQueryWithCallback(MappedStatement.java:189)
    +     com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.executeQueryForList(MappedStatement.java:139)+
    +     com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryForList(SqlMapExecutorDelegate.java:567)+
    +     com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryForList(SqlMapExecutorDelegate.java:541)+
    +     com.ibatis.sqlmap.engine.impl.SqlMapSessionImpl.queryForList(SqlMapSessionImpl.java:118)+
    +     com.ibatis.sqlmap.engine.impl.SqlMapClientImpl.queryForList(SqlMapClientImpl.java:94)+
    +     com.ibatis.dao.client.template.SqlMapDaoTemplate.queryForList(SqlMapDaoTemplate.java:282)+
    Some threads eventually end (after 1-2 hours), most of them remain there for days.
    Any hint would be quite useful, thanks.

    Threads are executing the actual allocated request from the Weblogic Kernel. Most of the problems happen when the Thread execution is reaching the application or business layer.
    At this point your application Java code module is sending or receiving data from external sources such as a an Oracle database for example. Any problem with such external system will cause the Thread to hang and wait for data to come back.
    Other situations can occur such as internal deadlock, infinite looping, heavy IO contention on your server etc.
    Doesn't loo like a driver issue.
    http://docs.oracle.com/cd/E21764_01/doc.1111/e14770/weblogic_server_issues.htm#autoId2
    Check at the Database end
    Cheers ...

  • How does WebLogic clean up stuck threads?

    Hi,
    Using WebLogic 9.2.2 on Linux OS. In our logs, we are finding some stuck threads and it seems to be due to something WebLogic is doing. While we don't understand the cause (maybe the stack trace helps), does WebLogic do anything to remove stuck threads from its pool, or is there anything we can do to configure it to?
    <Jul 21, 2009 2:50:21 PM UTC> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "605" seconds working on the request "Http Request: /RoutingEngine/pcFlow.do", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
    Thread-69 "[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, priority=1, DAEMON> {
    jrockit.net.SocketNativeIO.readBytesPinned(Unknown Source)
    jrockit.net.SocketNativeIO.socketRead(Unknown Source)
    java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)
    java.net.SocketInputStream.read(SocketInputStream.java:107)
    java.io.BufferedInputStream.fill(BufferedInputStream.java:189)
    java.io.BufferedInputStream.read(BufferedInputStream.java:234)
    ^-- Holding lock: java.io.BufferedInputStream@3537712[thin lock]
    weblogic.net.http.MessageHeader.isHTTP(MessageHeader.java:214)
    weblogic.net.http.MessageHeader.parseHeader(MessageHeader.java:141)
    weblogic.net.http.HttpClient.parseHTTP(HttpClient.java:473)
    ^-- Holding lock: java.io.BufferedInputStream@3537712[thin lock]
    weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:332)
    ^-- Holding lock: java.io.BufferedInputStream@3537712[thin lock]
    weblogic.net.http.SOAPHttpURLConnection.getInputStream(SOAPHttpURLConnection.java:36)
    weblogic.wsee.util.dom.DOMParser.getWebLogicDocumentImpl(DOMParser.java:92)
    weblogic.wsee.util.dom.DOMParser.getDocument(DOMParser.java:56)
    weblogic.wsee.wsdl.WsdlReader.getDocument(WsdlReader.java:311)
    weblogic.wsee.wsdl.WsdlReader.getDocument(WsdlReader.java:305)
    weblogic.wsee.wsdl.WsdlSchema.parse(WsdlSchema.java:120)
    weblogic.wsee.wsdl.WsdlSchemaImport.parse(WsdlSchemaImport.java:55)
    weblogic.wsee.wsdl.WsdlSchema.parse(WsdlSchema.java:82)
    weblogic.wsee.wsdl.WsdlSchema.parse(WsdlSchema.java:62)
    weblogic.wsee.wsdl.WsdlTypes.parse(WsdlTypes.java:145)
    weblogic.wsee.wsdl.WsdlDefinitions.parseChild(WsdlDefinitions.java:452)
    weblogic.wsee.wsdl.WsdlExtensible.parse(WsdlExtensible.java:84)
    weblogic.wsee.wsdl.WsdlDefinitions.parse(WsdlDefinitions.java:378)
    weblogic.wsee.wsdl.WsdlDefinitions.parse(WsdlDefinitions.java:358)
    weblogic.wsee.wsdl.WsdlFactory.parse(WsdlFactory.java:50)
    weblogic.wsee.jaxrpc.ServiceImpl.loadWsdlDefinition(ServiceImpl.java:437)
    weblogic.wsee.jaxrpc.ServiceImpl.<init>(ServiceImpl.java:95)
    com.myco.regui.provisioning.client.ProvisioningAccountService_Impl.<init>(ProvisioningAccountService_Impl.java:21)
    com.myco.regui.provisioning.client.ProvisioningAccountService_Impl.<init>(ProvisioningAccountService_Impl.java:17)
    com.myco.regui.struts.accounts.AccountsForm.contactWPS(AccountsForm.java:208)
    com.myco.regui.struts.accounts.AccountsForm.validateAccountid(AccountsForm.java:161)
    com.myco.regui.struts.accounts.AccountsForm.validatePCAccountId(AccountsForm.java:144)
    com.myco.regui.struts.accounts.AccountsForm._validatePCFields(AccountsForm.java:112)
    com.myco.regui.struts.accounts.AccountsForm.validate(AccountsForm.java:83)
    org.apache.struts.chain.commands.servlet.ValidateActionForm.validate(ValidateActionForm.java:57)
    org.apache.struts.chain.commands.AbstractValidateActionForm.execute(AbstractValidateActionForm.java:94)
    org.apache.struts.chain.commands.ActionCommandBase.execute(ActionCommandBase.java:48)
    org.apache.commons.chain.impl.ChainBase.execute(ChainBase.java:176)
    org.apache.commons.chain.generic.LookupCommand.execute(LookupCommand.java:303)
    org.apache.commons.chain.impl.ChainBase.execute(ChainBase.java:176)
    org.apache.struts.chain.ComposableRequestProcessor.process(ComposableRequestProcessor.java:272)
    org.apache.struts.action.ActionServlet.process(ActionServlet.java:1903)
    org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:462)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:736)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:851)
    weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
    weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
    weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:198)
    weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:41)
    com.myco.regui.servlets.filters.TransactionFilter.doFilter(TransactionFilter.java:23)
    .......

    You can configure these actions to occur when not all threads are stuck, but the number of stuck threads have exceeded a configured threshold:
    * Shut down the Work Manager if it has stuck threads. A Work Manager that is shut down will refuse new work and reject existing work in
    the queue by sending a rejection message. In a cluster, clustered clients will fail over to another cluster member.
    * Shut down the application if there are stuck threads in the application. The application is shutdown by bringing it into admin mode. All
    Work Managers belonging to the application are shut down, and behave as described above.
    * Mark the server instance as failed and shut it down it down if there are stuck threads in the server. In a cluster, clustered clients that
    are connected or attempting to connect will fail over to another cluster member.
    Link : [http://e-docs.bea.com/wls/docs103/config_wls/overload.html]

  • Mapping Stuck Threads to Proxy Services

    Hello OSB Experts:
    Is there a way I can map the information of a thread dump showing stuck threads to the proxy service that those threads are executing?
    I am facing a situation that seems to be usual: one of several backends is performing poorly, producing stuck threads in the OSB app. servers. After a while, the performance of the whole OSB is affected because of the number of threads in stuck state.
    I would like to be able to pinpoint the backend with problems in order to solve the problem before the health of the whole OSB is affected.
    Can this be done without 3rd party tools?
    Thanks
    Sebastián

    No, it is not possible to get the the proxy service name from the thread dump. The OSB stuff is handled under a weblogic wrapper thread.
    One of the main reason for stuck threads is some of the proxy service calls taking a lot of time through the business services and there is no appropriate timeout set for the calls on the business service.
    You have couple of options to find out the culprit. You can enable tracing on the proxy services and the business services and capture the metrics that should help you find a pattern for the performance. The numbers will help you find which backend is taking more time.
    You can get nmore details on OSB Monitoring @ http://docs.oracle.com/cd/E23943_01/admin.1111/e15867/monitoring_ops.htm#OSBAG472
    Secondly, always set some realistic timeout value (like 30 seconds/60 seconds depending on the sla of the back-end) on the business service. Leaving it to default (that is 0) is like configuring a indefinite timeout (in which case it uses the default weblogic connection timeout setting of 10/15 minutes).
    This way the threads are not stuck waiting for the response and dont cause issues.
    Additionally you use OSB alerts to notify you for service behavior discrepancies w.r.t response times, number of errors etc .. Refer http://docs.oracle.com/cd/E23943_01/admin.1111/e15867/monitoring_console.htm#CACDCDAH for more details.
    Hope this information helps.
    Thanks,
    Patrick
    It is considered good etiquette to reward answerers with points (as "helpful" - 5 pts - or "correct" - 10pts).+
    https://forums.oracle.com/forums/ann.jspa?annID=893

  • Stuck Thread - JMS Business proxy Service (ALSB 2.5)

    Hi All,
    We are having a jms proxy service which uses a JMS Business Service to send messages to a jms queue on weblogic 8.1 sp6. The alsb domain runs on weblogic 9.2.The jms proxy service is made single threaded by configuring a suitable work manager.
    Now and then, we get a stuck thread issue in production and the proxy service stops consuming the messaging off the jms queue in wls 9.2. These are the errors in the logs:
    <[ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1235437298516> <BEA-000337> <[STUCK] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "695" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@5321caf", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
    Thread-70 "[STUCK] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, waiting, priority=1, DAEMON> {
    -- Waiting for notification on: weblogic.rjvm.ResponseImpl@51961a5[fat lock]
    jrockit.vm.Threads.waitForSignal(Native Method)
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Native Method)
    weblogic.rjvm.ResponseImpl.waitForData(ResponseImpl.java:73)
    ^-- Lock released while waiting: weblogic.rjvm.ResponseImpl@51961a5[fat lock]
    weblogic.rjvm.ResponseImpl.getTxContext(ResponseImpl.java:100)
    weblogic.rjvm.BasicOutboundRequest.sendReceive(BasicOutboundRequest.java:109)
    weblogic.rmi.internal.BasicRemoteRef.invoke(BasicRemoteRef.java:223)
    weblogic.jms.dispatcher.DispatcherImpl_816_WLStub.dispatchSyncTranFuture(Unknown Source)
    weblogic.jms.dispatcher.DispatcherInteropAdapter.dispatchSyncTranFuture(DispatcherInteropAdapter.java:80)
    weblogic.messaging.dispatcher.DispatcherWrapperState.dispatchSyncTran(DispatcherWrapperState.java:230)
    weblogic.jms.dispatcher.DispatcherAdapter.dispatchSyncTran(DispatcherAdapter.java:51)
    weblogic.jms.client.JMSProducer.toFEProducer(JMSProducer.java:1047)
    weblogic.jms.client.JMSProducer.deliveryInternal(JMSProducer.java:670)
    weblogic.jms.client.JMSProducer.sendInternal(JMSProducer.java:455)
    weblogic.jms.client.JMSProducer.send(JMSProducer.java:352)
    weblogic.jms.client.WLProducerImpl.send(WLProducerImpl.java:1009)
    weblogic.deployment.jms.WrappedMessageProducer.send(WrappedMessageProducer.java:157)
    com.bea.wli.sb.transports.jms.JmsOutboundMessageContext$SendAction.run(JmsOutboundMessageContext.java:795)
    com.bea.wli.sb.transports.jms.JmsOutboundMessageContext.send(JmsOutboundMessageContext.java:500)
    com.bea.wli.sb.transports.jms.JmsTransportProvider.sendMessageAsync(JmsTransportProvider.java:612)
    jrockit.reflect.CompiledMethodInvoker.invoke0(Unknown Source)
    jrockit.reflect.CompiledMethodInvoker.invoke(Unknown Source)
    java.lang.reflect.Method.invoke(Unknown Source)
    com.bea.wli.sb.transports.Util$1.invoke(Util.java:241)
    $Proxy39.sendMessageAsync(Unknown Source)
    com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageAsync(LoadBalanceFailoverListener.java:137)
    com.bea.wli.sb.transports.TransportManagerImpl.sendMessageToService(TransportManagerImpl.java:437)
    com.bea.wli.sb.transports.TransportManagerImpl.sendMessageAsync(TransportManagerImpl.java:328)
    com.bea.wli.sb.pipeline.PipelineContextImpl.doDispatch(PipelineContextImpl.java:503)
    com.bea.wli.sb.pipeline.PipelineContextImpl.dispatch(PipelineContextImpl.java:415)
    stages.publish.runtime.PublishRuntimeStep.processMessage(PublishRuntimeStep.java:106)
    com.bea.wli.sb.stages.StageMetadataImpl$WrapperRuntimeStep.processMessage(StageMetadataImpl.java:327)
    com.bea.wli.sb.stages.impl.SequenceRuntimeStep.processMessage(SequenceRuntimeStep.java:33)
    com.bea.wli.sb.pipeline.PipelineStage.processMessage(PipelineStage.java:100)
    com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:771)
    com.bea.wli.sb.pipeline.Pipeline.processMessage(Pipeline.java:73)
    com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:771)
    com.bea.wli.sb.pipeline.PipelineNode.doRequest(PipelineNode.java:47)
    com.bea.wli.sb.pipeline.Node.processMessage(Node.java:57)
    com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:771)
    com.bea.wli.sb.pipeline.Router.processMessage(Router.java:141)
    com.bea.wli.sb.pipeline.MessageProcessor.processRequest(MessageProcessor.java:75)
    com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:646)
    com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:645)
    weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
    weblogic.security.service.SecurityManager.runAs(SecurityManager.java:147)
    com.bea.wli.sb.pipeline.RouterManager.processMessage(RouterManager.java:627)
    com.bea.wli.sb.transports.TransportManagerImpl.receiveMessage(TransportManagerImpl.java:264)
    com.bea.wli.sb.transports.jms.JmsInboundMDB.onMessage(JmsInboundMDB.java:108)
    weblogic.ejb.container.internal.MDListener.execute(MDListener.java:429)
    weblogic.ejb.container.internal.MDListener.transactionalOnMessage(MDListener.java:335)
    weblogic.ejb.container.internal.MDListener.onMessage(MDListener.java:292)
    weblogic.jms.client.JMSSession.onMessage(JMSSession.java:4060)
    weblogic.jms.client.JMSSession.execute(JMSSession.java:4012)
    weblogic.jms.client.JMSSession$UseForRunnable.run(JMSSession.java:4467)
    weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
    weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
    weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
    jrockit.vm.RNI.c2java(Native Method)
    Any one know the root cause of this issue or ways to prevent this?
    Also Is there a way to make the proxy service to reconsume messages off the queue without a server restart?
    Thanks,
    Atheek

    Hi atheek, did you find the solution??
    I already got the same/identic issue, do you mind giving me an aid?
    Really thanks
    Regards,
    Mr.K

  • Weblogic server + stuck threads

    Hi All,
    We have java application deployed on weblogic server 10.1.3.4 which is integrated with BPEL (SOA Oracle 11g). The application creates a task through BPEL. We have configured StuckThread Max Time to 600 secs. Now after certain time (5-7 days) the servers goes into warning state due to stuck threads and finally it gets crashed
    We noticed the OOM error in jrockit dump, could any one help me here as how to avoid this? your help is very much appreciated, thanks

    Regarding
    Sample of your Weblogic logs showing the java.lang.OutOfMemoryError along with Stack Trace if available
    Please find below
    [ACTIVE] ExecuteThread: '20' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <532776f80ba738ab:-48a8d4c8:133068f0cee:-8000-00000000000001a5> <1322080967160> <BEA-080003> <RuntimeException thrown by rmi server: javax.management.remote.rmi.RMIConnectionImpl.invoke(Ljavax.management.ObjectName;Ljava.lang.String;Ljava.rmi.MarshalledObject;[Ljava.lang.String;Ljavax.security.auth.Subject;)
    javax.management.RuntimeErrorException: java.lang.OutOfMemoryError: allocLargeObjectOrArray: [B, size 1335312.
    javax.management.RuntimeErrorException: java.lang.OutOfMemoryError: allocLargeObjectOrArray: [B, size 1335312
            at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:858)
            at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrowMaybeMBeanException(DefaultMBeanServerInterceptor.java:869)
            at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:838)
            at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
            at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$16.run(WLSMBeanServerInterceptorBase.java:449)
            at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:447)
            at weblogic.management.mbeanservers.internal.JMXContextInterceptor.invoke(JMXContextInterceptor.java:263)
            at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$16.run(WLSMBeanServerInterceptorBase.java:449)
            at weblogic.management.mbeanservers.internal.SecurityInterceptor.invoke(SecurityInterceptor.java:444)
            at weblogic.management.jmx.mbeanserver.WLSMBeanServer.invoke(WLSMBeanServer.java:323)
            at weblogic.management.mbeanservers.internal.JMXConnectorSubjectForwarder$11.run(JMXConnectorSubjectForwarder.java:661)
            at weblogic.management.mbeanservers.internal.JMXConnectorSubjectForwarder.invoke(JMXConnectorSubjectForwarder.java:654)
            at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
            at javax.management.remote.rmi.RMIConnectionImpl_WLSkel.invoke(Unknown Source)
            at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:668)
            at weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:118)
            at weblogic.work.ExecuteThread.execute(ExecuteThread.java:207)
            at weblogic.work.ExecuteThread.run(ExecuteThread.java:176)
    Caused By: java.lang.OutOfMemoryError: allocLargeObjectOrArray: [B, size 1335312
            at java.util.Arrays.copyOf(Arrays.java:2786)
            at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
            at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1847)
      at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
            at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1847)
            at java.io.ObjectOutputStream$BlockDataOutputStream.flush(ObjectOutputStream.java:1792)
            at java.io.ObjectOutputStream.flush(ObjectOutputStream.java:699)
            at java.io.ObjectOutputStream.close(ObjectOutputStream.java:720)
            at oracle.dms.jmx.app.ServerAggreMBeanImpl.getTableGroupInBytes(ServerAggreMBeanImpl.java:328)
            at oracle.dms.jmx.app.ServerAggreMBeanWrapper.invokeOperation(ServerAggreMBeanWrapper.java:71)
            at oracle.dms.jmx.app.AggreMBeanWrapper.invoke(AggreMBeanWrapper.java:167)
            at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
            at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
            at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$16.run(WLSMBeanServerInterceptorBase.java:449)
            at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:447)
            at weblogic.management.mbeanservers.internal.JMXContextInterceptor.invoke(JMXContextInterceptor.java:263)
            at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$16.run(WLSMBeanServerInterceptorBase.java:449)
            at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:447)
            at weblogic.management.mbeanservers.internal.SecurityInterceptor.invoke(SecurityInterceptor.java:444)
            at weblogic.management.jmx.mbeanserver.WLSMBeanServer.invoke(WLSMBeanServer.java:323)
            at weblogic.management.mbeanservers.internal.JMXConnectorSubjectForwarder$11$1.run(JMXConnectorSubjectForwarder.java:663)
            at weblogic.management.mbeanservers.internal.JMXConnectorSubjectForwarder$11.run(JMXConnectorSubjectForwarder.java:661)
            at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
            at weblogic.management.mbeanservers.internal.JMXConnectorSubjectForwarder.invoke(JMXConnectorSubjectForwarder.java:654)
            at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
            at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
            at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
            at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1367)
            at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
            at javax.management.remote.rmi.RMIConnectionImpl_WLSkel.invoke(Unknown Source)
            at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:667)
            at weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:522)
            at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
            at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:146)
            at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:518)
            at weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:118)
            at weblogic.work.ExecuteThread.execute(ExecuteThread.java:207)
            at weblogic.work.ExecuteThread.run(ExecuteThread.java:176)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               

  • STUCK THREADS DURING OIM DURING TRUSTED RECON USING FLAT FILE

    Hello All,
    I need some help with resolving this issue where stuck threads/hogging threads are spawn after I run the Trusted Recon which reads a flat file and creates/updates users in OIM.
    We have OIM 11.1.1.3 BP6
    Weblogic 10.3.3.0
    After the recon starts running for few hours I see ->.phd,.trc files created within the weblogic server path and server goes out of memory. eventually server goes down.
    My issue is similar to: https://forums.oracle.com/forums/message.jspa?messageID=10187076#10187076
    I have followed almost all of the performance tuning settings.
    Pls. let me know if you have any ideas.
    I have pasted below the heap dump.
    Thanks.
    ~VSN
    3XMTHREADINFO "[STUCK] ExecuteThread: '56' for queue: 'weblogic.kernel.Default (self-tuning)'" J9VMThread:0x000000013B49FC00, j9thread_t:0x00000001369D1760, java/lang/Thread:0x0700000062E79CB0, state:CW, prio=1
    3XMTHREADINFO1 (native thread ID:0x3F000B1, native priority:0x1, native policy:UNKNOWN)
    3XMTHREADINFO3 Java callstack:
    4XESTACKTRACE at java/lang/Object.wait(Native Method)
    4XESTACKTRACE at java/lang/Object.wait(Object.java:167(Compiled Code))
    4XESTACKTRACE at java/io/ObjectStreamClass$EntryFuture.get(ObjectStreamClass.java:428(Compiled Code))
    4XESTACKTRACE at java/io/ObjectStreamClass.lookup(ObjectStreamClass.java:314(Compiled Code))
    4XESTACKTRACE at java/io/ObjectOutputStream.writeObject0(ObjectOutputStream.java:1115(Compiled Code))
    4XESTACKTRACE at java/io/ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1518(Compiled Code))
    4XESTACKTRACE at java/io/ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1483(Compiled Code))
    4XESTACKTRACE at java/io/ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1401(Compiled Code))
    4XESTACKTRACE at java/io/ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159(Compiled Code))
    4XESTACKTRACE at java/io/ObjectOutputStream.writeObject(ObjectOutputStream.java:332(Compiled Code))
    4XESTACKTRACE at org/eclipse/persistence/mappings/converters/SerializedObjectConverter.convertObjectValueToDataValue(SerializedObjectConverter.java:85(Compiled Code))
    4XESTACKTRACE at org/eclipse/persistence/mappings/foundation/AbstractDirectMapping.getFieldValue(AbstractDirectMapping.java:808(Compiled Code))
    4XESTACKTRACE at org/eclipse/persistence/mappings/foundation/AbstractDirectMapping.buildCloneValue(AbstractDirectMapping.java:264(Compiled Code))
    4XESTACKTRACE at org/eclipse/persistence/mappings/foundation/AbstractDirectMapping.buildCloneValue(AbstractDirectMapping.java:239(Compiled Code))
    4XESTACKTRACE at org/eclipse/persistence/mappings/foundation/AbstractDirectMapping.buildClone(AbstractDirectMapping.java(Compiled Code))
    4XESTACKTRACE at org/eclipse/persistence/internal/descriptors/ObjectBuilder.populateAttributesForClone(ObjectBuilder.java:2698(Compiled Code))
    4XESTACKTRACE at org/eclipse/persistence/internal/sessions/UnitOfWorkImpl.populateAndRegisterObject(UnitOfWorkImpl.java:3682(Compiled Code))
    4XESTACKTRACE at org/eclipse/persistence/internal/sessions/UnitOfWorkImpl.cloneAndRegisterObject(UnitOfWorkImpl.java:996(Compiled Code))
    4XESTACKTRACE at org/eclipse/persistence/internal/sessions/UnitOfWorkImpl.cloneAndRegisterObject(UnitOfWorkImpl.java:905(Compiled Code))
    4XESTACKTRACE at org/eclipse/persistence/internal/sessions/UnitOfWorkIdentityMapAccessor.getAndCloneCacheKeyFromParent(UnitOfWorkIdentityMapAccessor.java:123(Compiled Code))
    4XESTACKTRACE at org/eclipse/persistence/internal/sessions/UnitOfWorkIdentityMapAccessor.getFromIdentityMap(UnitOfWorkIdentityMapAccessor.java:110(Compiled Code))
    4XESTACKTRACE at org/eclipse/persistence/internal/sessions/IdentityMapAccessor.getFromIdentityMap(IdentityMapAccessor.java(Compiled Code))
    4XESTACKTRACE at org/eclipse/persistence/internal/sessions/UnitOfWorkImpl.checkExistence(UnitOfWorkImpl.java:774(Compiled Code))
    4XESTACKTRACE at org/eclipse/persistence/internal/sessions/UnitOfWorkImpl.internalRegisterObject(UnitOfWorkImpl.java:2935(Compiled Code))
    4XESTACKTRACE at org/eclipse/persistence/internal/sessions/UnitOfWorkImpl.registerObject(UnitOfWorkImpl.java:4363(Compiled Code))
    4XESTACKTRACE at org/eclipse/persistence/internal/sessions/UnitOfWorkImpl.registerObject(UnitOfWorkImpl.java:4321(Compiled Code))
    4XESTACKTRACE at oracle/iam/platform/kernel/dao/OrchestrationDao.setProcessSeq(OrchestrationDao.java:682(Compiled Code))
    4XESTACKTRACE at oracle/iam/platform/kernel/impl/OrchProcessData.runActionEvents(OrchProcessData.java:1050(Compiled Code))
    4XESTACKTRACE at oracle/iam/platform/kernel/impl/OrchProcessData.runEvents(OrchProcessData.java:644(Compiled Code))
    4XESTACKTRACE at oracle/iam/platform/kernel/impl/OrchProcessData.executeEvents(OrchProcessData.java:227(Compiled Code))
    4XESTACKTRACE at oracle/iam/platform/kernel/impl/OrchestrationEngineImpl.resumeProcess(OrchestrationEngineImpl.java:668(Compiled Code))
    4XESTACKTRACE at oracle/iam/platform/kernel/impl/OrchestrationEngineImpl.resumeChildProcess(OrchestrationEngineImpl.java:738(Compiled Code))
    4XESTACKTRACE at oracle/iam/platform/kernel/impl/OrchestrationEngineImpl.resumeProcess(OrchestrationEngineImpl.java:689(Compiled Code))
    4XESTACKTRACE at oracle/iam/platform/kernel/impl/OrchestrationEngineImpl.notifyParentProcess(OrchestrationEngineImpl.java:828(Compiled Code))
    4XESTACKTRACE at oracle/iam/platform/kernel/impl/OrchProcessData.runEvents(OrchProcessData.java:771(Compiled Code))
    4XESTACKTRACE at oracle/iam/platform/kernel/impl/OrchProcessData.executeEvents(OrchProcessData.java:227(Compiled Code))
    4XESTACKTRACE at oracle/iam/platform/kernel/impl/OrchestrationEngineImpl.resumeProcess(OrchestrationEngineImpl.java:668(Compiled Code))
    4XESTACKTRACE at oracle/iam/platform/kernel/impl/OrchestrationEngineImpl.resumeChildProcess(OrchestrationEngineImpl.java:738(Compiled Code))
    4XESTACKTRACE at oracle/iam/platform/kernel/impl/OrchestrationEngineImpl.resumeProcess(OrchestrationEngineImpl.java:689(Compiled Code))
    4XESTACKTRACE at oracle/iam/platform/kernel/impl/OrchestrationEngineImpl.notifyParentProcess(OrchestrationEngineImpl.java:828(Compiled Code))
    4XESTACKTRACE at oracle/iam/platform/kernel/impl/OrchProcessData.runEvents(OrchProcessData.java:771(Compiled Code))
    4XESTACKTRACE at oracle/iam/platform/kernel/impl/OrchProcessData.executeEvents(OrchProcessData.java:227(Compiled Code))
    4XESTACKTRACE at oracle/iam/platform/kernel/impl/OrchestrationEngineImpl.resumeProcess(OrchestrationEngineImpl.java:668(Compiled Code))
    4XESTACKTRACE at oracle/iam/platform/kernel/impl/OrchestrationEngineImpl.resumeChildProcess(OrchestrationEngineImpl.java:738(Compiled Code))
    4XESTACKTRACE at oracle/iam/platform/kernel/impl/OrchestrationEngineImpl.resumeProcess(OrchestrationEngineImpl.java:689(Compiled Code))
    4XESTACKTRACE at oracle/iam/platform/kernel/impl/OrchestrationEngineImpl.notifyParentProcess(OrchestrationEngineImpl.java:828(Compiled Code))
    4XESTACKTRACE at oracle/iam/platform/kernel/impl/OrchProcessData.runEvents(OrchProcessData.java:771(Compiled Code))
    4XESTACKTRACE at oracle/iam/platform/kernel/impl/OrchProcessData.executeEvents(OrchProcessData.java:227(Compiled Code))
    4XESTACKTRACE at oracle/iam/platform/kernel/impl/OrchestrationEngineImpl.resumeProcess(OrchestrationEngineImpl.java:668(Compiled Code))
    4XESTACKTRACE at oracle/iam/platform/kernel/impl/OrchestrationEngineImpl.resumeChildProcess(OrchestrationEngineImpl.java:738(Compiled Code))
    4XESTACKTRACE at oracle/iam/platform/kernel/impl/OrchestrationEngineImpl.resumeProcess(OrchestrationEngineImpl.java:689(Compiled Code))
    4XESTACKTRACE at oracle/iam/platform/kernel/impl/OrchestrationEngineImpl.notifyParentProcess(OrchestrationEngineImpl.java:828(Compiled Code))
    4XESTACKTRACE at oracle/iam/platform/kernel/impl/OrchProcessData.runEvents(OrchProcessData.java:771(Compiled Code))
    4XESTACKTRACE at oracle/iam/platform/kernel/impl/OrchProcessData.executeEvents(OrchProcessData.java:227(Compiled Code))
    4XESTACKTRACE at oracle/iam/platform/kernel/impl/OrchestrationEngineImpl.resumeProcess(OrchestrationEngineImpl.java:668(Compiled Code))
    4XESTACKTRACE at oracle/iam/platform/kernel/impl/OrchestrationEngineImpl.resumeChildProcess(OrchestrationEngineImpl.java:738(Compiled Code))
    4XESTACKTRACE at oracle/iam/platform/kernel/impl/OrchestrationEngineImpl.resumeProcess(OrchestrationEngineImpl.java:689(Compiled Code))
    4XESTACKTRACE at oracle/iam/platform/kernel/impl/OrchestrationEngineImpl.notifyParentProcess(OrchestrationEngineImpl.java:828(Compiled Code))
    4XESTACKTRACE at oracle/iam/platform/kernel/impl/OrchProcessData.runEvents(OrchProcessData.java:771(Compiled Code))
    4XESTACKTRACE at oracle/iam/platform/kernel/impl/OrchProcessData.executeEvents(OrchProcessData.java:227(Compiled Code))
    4XESTACKTRACE at oracle/iam/platform/kernel/impl/OrchestrationEngineImpl.resumeProcess(OrchestrationEngineImpl.java:668(Compiled Code))
    4XESTACKTRACE at oracle/iam/platform/kernel/impl/OrchestrationEngineImpl.resumeChildProcess(OrchestrationEngineImpl.java:738(Compiled Code))
    4XESTACKTRACE at oracle/iam/platform/kernel/impl/OrchestrationEngineImpl.resumeProcess(OrchestrationEngineImpl.java:689(Compiled Code))
    4XESTACKTRACE at oracle/iam/platform/kernel/impl/OrchestrationEngineImpl.notifyParentProcess(OrchestrationEngineImpl.java:828(Compiled Code))
    4XESTACKTRACE at oracle/iam/platform/kernel/impl/OrchProcessData.runEvents(OrchProcessData.java:771(Compiled Code))
    4XESTACKTRACE at oracle/iam/platform/kernel/impl/OrchProcessData.executeEvents(OrchProcessData.java:227(Compiled Code))
    4XESTACKTRACE at oracle/iam/platform/kernel/impl/OrchestrationEngineImpl.resumeProcess(OrchestrationEngineImpl.java:668(Compiled Code))
    4XESTACKTRACE at oracle/iam/platform/kernel/impl/OrchestrationEngineImpl.resumeChildProcess(OrchestrationEngineImpl.java:738(Compiled Code))
    4XESTACKTRACE at oracle/iam/platform/kernel/impl/OrchestrationEngineImpl.resumeProcess(OrchestrationEngineImpl.java:689(Compiled Code))
    4XESTACKTRACE at oracle/iam/platform/kernel/impl/OrchestrationEngineImpl.notifyParentProcess(OrchestrationEngineImpl.java:828(Compiled Code))
    4XESTACKTRACE at oracle/iam/platform/kernel/impl/OrchProcessData.runEvents(OrchProcessData.java:771(Compiled Code))
    4XESTACKTRACE at oracle/iam/platform/kernel/impl/OrchProcessData.executeEvents(OrchProcessData.java:227(Compiled Code))

    Increase the memory your database is using. Using the Enterprise Manager of your database, watch the usage during the recon. If it is still maxing out on memory and using paging memory, then increase the memory some more. You can also increase the memory allocated to your application servers. Increase the number of available threads to your weblogic instance.
    -Kevin

Maybe you are looking for