What causes following stuck thread ?

The StuckThreadMaxTime = 600 seconds.
Here is:
            "[STUCK] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'" id=55 idx=0xdc tid=3147 prio=1 alive, native_blocked, daemon
                at java/util/HashMap.containsKey(HashMap.java:415)[optimized]
                at com/sun/faces/mgbean/BeanManager.isManaged(BeanManager.java:135)[inlined]
                at com/sun/faces/application/WebappLifecycleListener.handleAttributeEvent(WebappLifecycleListener.java:264)[inlined]
                at com/sun/faces/application/WebappLifecycleListener.requestDestroyed(WebappLifecycleListener.java:97)[inlined]
                at com/sun/faces/config/ConfigureListener.requestDestroyed(ConfigureListener.java:283)[optimized]
                at weblogic/servlet/internal/EventsManager.notifyRequestLifetimeEvent(EventsManager.java:335)[inlined]
                at weblogic/servlet/internal/RequestEventsFilter.doFilter(RequestEventsFilter.java:29)[optimized]
                at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[inlined]
                at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3715)[inlined]
                at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3681)[optimized]
                at weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)[optimized]
                at weblogic/security/service/SecurityManager.runAs(SecurityManager.java:120)[inlined]
                at weblogic/servlet/internal/WebAppServletContext.securedExecute(WebAppServletContext.java:2277)[inlined]
                at weblogic/servlet/internal/WebAppServletContext.execute(WebAppServletContext.java:2183)[optimized]
                at weblogic/servlet/internal/ServletRequestImpl.run(ServletRequestImpl.java:1454)[optimized]
                at weblogic/work/ExecuteThread.execute(ExecuteThread.java:209)[optimized]
                at weblogic/work/ExecuteThread.run(ExecuteThread.java:178)
                at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
                -- end of trace

Hi,
Tanks for response !
But, -XX:+PrintConcurrentLocks does not work with JRockit.
Is this default JRockit option, as noticed in the article ?
Pierluigi Vernetto wrote:
You should start the JVM option -XX:+PrintConcurrentLocks (see http://www.javamonamour.org/2011/09/nice-to-activate-sun-jvm-flags-for-gc.html ) , and get more info on who is holding a lock on the same object

Similar Messages

  • How can I find out what causes a program to NOT RESPONDING status suddenly?

    After a very long period of functioning without serious problems a program suddenly doesn't start anymore, showing this funny rotating ball and a message in the Activity Monitor that this program is not responding. I have searched the log files meticulously, hoping to find some clues what caused the stuck, but no chance of course. Is there a more or less "educated way" to find out, what happened and how to prevent it in the future?
    Thanks a lot in advance

    Some of your user files (not system files) have incorrect permissions or are locked. This procedure will unlock those files and reset their ownership, permissions, and access controls to the default. If you've intentionally set special values for those attributes, they will be reverted. In that case, either stop here, or be prepared to recreate the settings if necessary. Do so only after verifying that those settings didn't cause the problem. If none of this is meaningful to you, you don't need to worry about it, but you do need to follow the instructions below.
    Back up all data before proceeding.
    Step 1
    If you have more than one user, and the one in question is not an administrator, then go to Step 2.
    Enter the following command in the Terminal window in the same way as before (triple-click, copy, and paste):
    sudo find ~ $TMPDIR.. -exec chflags -h nouchg,nouappnd,noschg,nosappnd {} + -exec chown -h $UID {} + -exec chmod +rw {} + -exec chmod -h -N {} + -type d -exec chmod -h +x {} + 2>&-
    You'll be prompted for your login password, which won't be displayed when you type it. Type carefully and then press return. You may get a one-time warning to be careful. If you don’t have a login password, you’ll need to set one before you can run the command. If you see a message that your username "is not in the sudoers file," then you're not logged in as an administrator.
    The command may take several minutes to run, depending on how many files you have. Wait for a new line ending in a dollar sign ($) to appear, then quit Terminal.
    Step 2 (optional)
    Take this step only if you have trouble with Step 1, if you prefer not to take it, or if it doesn't solve the problem.
    Start up in Recovery mode. When the OS X Utilities screen appears, select
              Utilities ▹ Terminal
    from the menu bar. A Terminal window will open. In that window, type this:
    resetp
    Press the tab key. The partial command you typed will automatically be completed to this:
    resetpassword
    Press return. A Reset Password window will open. You’re not going to reset a password.
    Select your startup volume ("Macintosh HD," unless you gave it a different name) if not already selected.
    Select your username from the menu labeled Select the user account if not already selected.
    Under Reset Home Directory Permissions and ACLs, click the Reset button.
    Select
               ▹ Restart
    from the menu bar.

  • OSB STUCK THREAD at LoadBalanceFailoverListener

    Hi,
    We get the following stuck threads on our OSB servers.
    Can someone help?
    Due to this, our servers are experiencing a huge resource crunch.
    "[STUCK] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=3 tid=0x0000000110704800 nid=0x94 runnable [0xfffffffa1d0fc000]
    java.lang.Thread.State: RUNNABLE
    at java.util.HashMap.get(HashMap.java:303)
    at com.bea.wli.sb.service.ServiceStats.getOperationStats(ServiceStats.java:104)
    at com.bea.wli.sb.pipeline.MessageProcessor.startOperationalTimer(MessageProcessor.java:431)
    at com.bea.wli.sb.pipeline.MessageProcessor.processRequest(MessageProcessor.java:93)
    at com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:593)
    at com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:591)
    at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
    at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:146)
    at com.bea.wli.sb.security.WLSSecurityContextService.runAs(WLSSecurityContextService.java:55)
    at com.bea.wli.sb.pipeline.RouterManager.processMessage(RouterManager.java:590)
    at com.bea.wli.sb.transports.TransportManagerImpl.receiveMessage(TransportManagerImpl.java:375)
    at com.bea.wli.sb.transports.CoLocatedMessageContext$1.run(CoLocatedMessageContext.java:162)
    at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
    at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:146)
    at weblogic.security.Security.runAs(Security.java:61)
    at com.bea.wli.sb.util.security.SecurityUtils.runAs(SecurityUtils.java:58)
    at com.bea.wli.sb.transports.CoLocatedMessageContext.send(CoLocatedMessageContext.java:157)
    at com.bea.wli.sb.transports.http.wls.HttpTransportProvider.sendMessageAsync(HttpTransportProvider.java:215)
    at sun.reflect.GeneratedMethodAccessor914.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.bea.wli.sb.transports.Util$1.invoke(Util.java:83)
    at $Proxy123.sendMessageAsync(Unknown Source)
    at com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageAsync(LoadBalanceFailoverListener.java:148)
    at com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageToServiceAsync(LoadBalanceFailoverListener.java:603)
    at com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageToService(LoadBalanceFailoverListener.java:538)
    at com.bea.wli.sb.transports.TransportManagerImpl.sendMessageToService(TransportManagerImpl.java:558)
    at com.bea.wli.sb.transports.TransportManagerImpl.sendMessageAsync(TransportManagerImpl.java:426)
    at com.bea.wli.sb.pipeline.PipelineContextImpl.doDispatch(PipelineContextImpl.java:670)
    at com.bea.wli.sb.pipeline.PipelineContextImpl.dispatch(PipelineContextImpl.java:585)
    at stages.routing.runtime.DynamicRouteRuntimeStep.processMessage(DynamicRouteRuntimeStep.java:161)
    at com.bea.wli.sb.stages.StageMetadataImpl$WrapperRuntimeStep.processMessage(StageMetadataImpl.java:346)
    at com.bea.wli.sb.pipeline.RouteNode.doRequest(RouteNode.java:106)
    at com.bea.wli.sb.pipeline.Node.processMessage(Node.java:67)
    at com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:1055)
    at com.bea.wli.sb.pipeline.Router.processMessage(Router.java:214)
    at com.bea.wli.sb.pipeline.MessageProcessor.processRequest(MessageProcessor.java:96)
    at com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:593)
    at com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:591)
    at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
    at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:146)
    at com.bea.wli.sb.security.WLSSecurityContextService.runAs(WLSSecurityContextService.java:55)
    at com.bea.wli.sb.pipeline.RouterManager.processMessage(RouterManager.java:590)
    at com.bea.wli.sb.transports.TransportManagerImpl.receiveMessage(TransportManagerImpl.java:375)
    at com.bea.wli.sb.transports.jms.JmsInboundMDB.onMessage(JmsInboundMDB.java:132)
    at sun.reflect.GeneratedMethodAccessor915.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310)
    at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
    at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
    at com.bea.core.repackaged.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
    at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
    at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
    at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)

    What i have not understood is how disabling log tracing will prevent the STUCK thread?The stuck threads which you are seeing are because of a known bug in the tracing functionality. So if you will disable the tracing then you will not hit that bug and hence no stuck threads will appear. To use tracing, get a fix for this problem from Oracle Support by raising a SR with them.
    Regards,
    Anuj

  • Need help with STUCK Threads.  What's causing them?

    In our production environment, we have a soa suite cluster with 3 nodes. All 3 of the servers keep turning up stuck and hogging threads, and eventually they all grind to a halt where I can't even view composite instances through em. There is a bpel process that is using a dbadapter to poll for records every 5 seconds and then performs some service call outs and other dbadapters. This process is generating thousands of instances a day and I think it is the culprit. If I view the logs, I see one of the dbadapters that it calls out to fails occasionally with a PK violation. I looked at the developer's code, and they have no error handling in the process at all. In the logs, there will the a hit for the PK violation, then about 10 minutes later the a STUCK thread gets logged(the configured timeout is 600 seconds, or 10 minutes). Would not handling an error in a BPEL process hog/STUCK a thread? I need to be able to show what code is causing the issue so I can have a resolution path.
    I also see a note in the trace about something holding a lock:
    Holding lock: oracle.jdbc.driver.T4CConnection@1fbc3486[thin lock]....would that be an un-rolled back transaction?
    Here is the dump of the STUCK thread. Any help is greatly appreciated. Thanks!
    <Aug 5, 2011 10:30:02 AM EDT> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '64' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "634" seconds working on the request "oracle.soa.management.internal.ejb.impl.FacadeFinderBean_4vacyo_FacadeFinderBeanImpl", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
    Thread-517 "[STUCK] ExecuteThread: '64' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, priority=1, DAEMON> {
    jrockit.net.SocketNativeIO.readBytesPinned(SocketNativeIO.java:???)
    jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:24)
    java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)
    java.net.SocketInputStream.read(SocketInputStream.java:107)
    oracle.net.nt.MetricsEnabledInputStream.read(TcpNTAdapter.java:553)
    oracle.net.ns.Packet.receive(Packet.java:249)
    oracle.net.ns.DataPacket.receive(DataPacket.java:94)
    oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:168)
    oracle.net.ns.NetInputStream.read(NetInputStream.java:113)
    oracle.net.ns.NetInputStream.read(NetInputStream.java:96)
    oracle.net.ns.NetInputStream.read(NetInputStream.java:81)
    oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:120)
    oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:75)
    oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1081)
    oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1062)
    oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:588)
    oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:70)
    oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:854)
    oracle.jdbc.driver.T4CPreparedStatement.executeMaybeDescribe(T4CPreparedStatement.java:918)
    oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1252)
    oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3541)
    oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3605)
    ^-- Holding lock: oracle.jdbc.driver.T4CConnection@1fbc3486[thin lock]
    oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1667)
    weblogic.jdbc.wrapper.PreparedStatement.executeQuery(PreparedStatement.java:123)
    org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeSelect(DatabaseAccessor.java:887)
    org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:541)
    org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:530)
    org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:521)
    org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:202)
    org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:192)
    org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelectCall(DatasourceCallQueryMechanism.java:254)
    org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelect(DatasourceCallQueryMechanism.java:245)
    org.eclipse.persistence.queries.DataReadQuery.executeNonCursor(DataReadQuery.java:188)
    org.eclipse.persistence.queries.DataReadQuery.executeDatabaseQuery(DataReadQuery.java:141)
    org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:603)
    org.eclipse.persistence.queries.DataReadQuery.execute(DataReadQuery.java:115)
    org.eclipse.persistence.queries.DatabaseQuery.executeInUnitOfWork(DatabaseQuery.java:589)
    org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2898)
    org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1218)
    org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1190)
    org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1176)
    org.eclipse.persistence.internal.jpa.EJBQueryImpl.executeReadQuery(EJBQueryImpl.java:378)
    org.eclipse.persistence.internal.jpa.EJBQueryImpl.getSingleResult(EJBQueryImpl.java:701)
    weblogic.deployment.TransactionalEntityManagerProxyImpl$QueryImpl.getSingleResult(TransactionalEntityManagerProxyImpl.java:232)
    com.collaxa.cube.persistence.dao.impl.jpa.JpaCubeInstanceDaoImpl.getNumberOfInstances(JpaCubeInstanceDaoImpl.java:439)
    oracle.fabric.CubeServiceEngine.getNumberOfComponentInstances(CubeServiceEngine.java:1962)
    oracle.soa.management.internal.ejb.impl.FacadeFinderBeanImpl.getNumberOfComponentInstances(FacadeFinderBeanImpl.java:1618)
    oracle.soa.management.internal.ejb.impl.FacadeFinderBeanImpl.getComponentSnapShot(FacadeFinderBeanImpl.java:1653)
    sun.reflect.GeneratedMethodAccessor5852.invoke(Unknown Source)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    java.lang.reflect.Method.invoke(Method.java:575)
    com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.jee.intercept.MethodInvocationInvocationContext.proceed(MethodInvocationInvocationContext.java:104)
    oracle.security.jps.ee.ejb.JpsAbsInterceptor$1.run(JpsAbsInterceptor.java:88)
    oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:299)
    oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:406)
    oracle.security.jps.wls.JpsWeblogicEjbInterceptor.runJaasMode(JpsWeblogicEjbInterceptor.java:61)
    oracle.security.jps.ee.ejb.JpsAbsInterceptor.intercept(JpsAbsInterceptor.java:81)
    oracle.security.jps.ee.ejb.JpsInterceptor.intercept(JpsInterceptor.java:101)
    sun.reflect.GeneratedMethodAccessor863.invoke(Unknown Source)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    java.lang.reflect.Method.invoke(Method.java:575)
    com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
    com.bea.core.repackaged.springframework.jee.intercept.JeeInterceptorInterceptor.invoke(JeeInterceptorInterceptor.java:66)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
    com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:102)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.jee.spi.MethodInvocationVisitorImpl.visit(MethodInvocationVisitorImpl.java:37)
    weblogic.ejb.container.injection.EnvironmentInterceptorCallbackImpl.callback(EnvironmentInterceptorCallbackImpl.java:22)
    com.bea.core.repackaged.springframework.jee.spi.EnvironmentInterceptor.invoke(EnvironmentInterceptor.java:49)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:86)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
    com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:102)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:151)
    $Proxy309.getComponentSnapShot(Unknown Source)
    oracle.soa.management.internal.ejb.impl.FacadeFinderBean_4vacyo_FacadeFinderBeanImpl.getComponentSnapShot(FacadeFinderBean_4vacyo_FacadeFinderBeanImpl.java:1604)
    oracle.soa.management.internal.ejb.impl.FacadeFinderBean_4vacyo_FacadeFinderBeanImpl_WLSkel.invoke(Unknown Source)
    weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:567)
    weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:477)
    weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:350)
    weblogic.security.service.SecurityManager.runAs(SecurityManager.java:143)
    weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:425)
    weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:117)
    weblogic.work.ExecuteThread.execute(ExecuteThread.java:198)
    weblogic.work.ExecuteThread.run(ExecuteThread.java:165)
    >
    <Aug 5, 2011 10:30:02 AM EDT> <Notice> <Diagnostics> <BEA-320068> <Watch 'StuckThread' with severity 'Notice' on server 'soa_server1' has triggered at Aug 5, 2011 10:30:02 AM EDT. Notification details:
    WatchRuleType: Log
    WatchRule: (SEVERITY = 'Error') AND (MSGID = 'BEA-000337')
    WatchData: DATE = Aug 5, 2011 10:30:02 AM EDT SERVER = soa_server1 MESSAGE = [STUCK] ExecuteThread: '64' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "634" seconds working on the request "oracle.soa.management.internal.ejb.impl.FacadeFinderBean_4vacyo_FacadeFinderBeanImpl", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
    Thread-517 "[STUCK] ExecuteThread: '64' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, priority=1, DAEMON> {
    jrockit.net.SocketNativeIO.readBytesPinned(SocketNativeIO.java:???)
    jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:24)
    java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)
    java.net.SocketInputStream.read(SocketInputStream.java:107)
    oracle.net.nt.MetricsEnabledInputStream.read(TcpNTAdapter.java:553)
    oracle.net.ns.Packet.receive(Packet.java:249)
    oracle.net.ns.DataPacket.receive(DataPacket.java:94)
    oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:168)
    oracle.net.ns.NetInputStream.read(NetInputStream.java:113)
    oracle.net.ns.NetInputStream.read(NetInputStream.java:96)
    oracle.net.ns.NetInputStream.read(NetInputStream.java:81)
    oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:120)
    oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:75)
    oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1081)
    oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1062)
    oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:588)
    oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:70)
    oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:854)
    oracle.jdbc.driver.T4CPreparedStatement.executeMaybeDescribe(T4CPreparedStatement.java:918)
    oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1252)
    oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3541)
    oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3605)
    ^-- Holding lock: oracle.jdbc.driver.T4CConnection@1fbc3486[thin lock]
    oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1667)
    weblogic.jdbc.wrapper.PreparedStatement.executeQuery(PreparedStatement.java:123)
    org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeSelect(DatabaseAccessor.java:887)
    org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:541)
    org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:530)
    org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:521)
    org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:202)
    org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:192)
    org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelectCall(DatasourceCallQueryMechanism.java:254)
    org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelect(DatasourceCallQueryMechanism.java:245)
    org.eclipse.persistence.queries.DataReadQuery.executeNonCursor(DataReadQuery.java:188)
    org.eclipse.persistence.queries.DataReadQuery.executeDatabaseQuery(DataReadQuery.java:141)
    org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:603)
    org.eclipse.persistence.queries.DataReadQuery.execute(DataReadQuery.java:115)
    org.eclipse.persistence.queries.DatabaseQuery.executeInUnitOfWork(DatabaseQuery.java:589)
    org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2898)
    org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1218)
    org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1190)
    org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1176)
    org.eclipse.persistence.internal.jpa.EJBQueryImpl.executeReadQuery(EJBQueryImpl.java:378)
    org.eclipse.persistence.internal.jpa.EJBQueryImpl.getSingleResult(EJBQueryImpl.java:701)
    weblogic.deployment.TransactionalEntityManagerProxyImpl$QueryImpl.getSingleResult(TransactionalEntityManagerProxyImpl.java:232)
    com.collaxa.cube.persistence.dao.impl.jpa.JpaCubeInstanceDaoImpl.getNumberOfInstances(JpaCubeInstanceDaoImpl.java:439)
    oracle.fabric.CubeServiceEngine.getNumberOfComponentInstances(CubeServiceEngine.java:1962)
    oracle.soa.management.internal.ejb.impl.FacadeFinderBeanImpl.getNumberOfComponentInstances(FacadeFinderBeanImpl.java:1618)
    oracle.soa.management.internal.ejb.impl.FacadeFinderBeanImpl.getComponentSnapShot(FacadeFinderBeanImpl.java:1653)
    sun.reflect.GeneratedMethodAccessor5852.invoke(Unknown Source)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    java.lang.reflect.Method.invoke(Method.java:575)
    com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.jee.intercept.MethodInvocationInvocationContext.proceed(MethodInvocationInvocationContext.java:104)
    oracle.security.jps.ee.ejb.JpsAbsInterceptor$1.run(JpsAbsInterceptor.java:88)
    oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:299)
    oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:406)
    oracle.security.jps.wls.JpsWeblogicEjbInterceptor.runJaasMode(JpsWeblogicEjbInterceptor.java:61)
    oracle.security.jps.ee.ejb.JpsAbsInterceptor.intercept(JpsAbsInterceptor.java:81)
    oracle.security.jps.ee.ejb.JpsInterceptor.intercept(JpsInterceptor.java:101)
    sun.reflect.GeneratedMethodAccessor863.invoke(Unknown Source)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    java.lang.reflect.Method.invoke(Method.java:575)
    com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
    com.bea.core.repackaged.springframework.jee.intercept.JeeInterceptorInterceptor.invoke(JeeInterceptorInterceptor.java:66)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
    com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:102)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.jee.spi.MethodInvocationVisitorImpl.visit(MethodInvocationVisitorImpl.java:37)
    weblogic.ejb.container.injection.EnvironmentInterceptorCallbackImpl.callback(EnvironmentInterceptorCallbackImpl.java:22)
    com.bea.core.repackaged.springframework.jee.spi.EnvironmentInterceptor.invoke(EnvironmentInterceptor.java:49)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:86)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
    com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:102)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:151)
    $Proxy309.getComponentSnapShot(Unknown Source)
    oracle.soa.management.internal.ejb.impl.FacadeFinderBean_4vacyo_FacadeFinderBeanImpl.getComponentSnapShot(FacadeFinderBean_4vacyo_FacadeFinderBeanImpl.java:1604)
    oracle.soa.management.internal.ejb.impl.FacadeFinderBean_4vacyo_FacadeFinderBeanImpl_WLSkel.invoke(Unknown Source)
    weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:567)
    weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:477)
    weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:350)
    weblogic.security.service.SecurityManager.runAs(SecurityManager.java:143)
    weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:425)
    weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:117)
    weblogic.work.ExecuteThread.execute(ExecuteThread.java:198)
    weblogic.work.ExecuteThread.run(ExecuteThread.java:165)
    SUBSYSTEM = WebLogicServer USERID = <WLS Kernel> SEVERITY = Error THREAD = [ACTIVE] ExecuteThread: '96' for queue: 'weblogic.kernel.Default (self-tuning)' MSGID = BEA-000337 MACHINE = soaprod TXID = CONTEXTID = f5d0b85a929630a2:6eada0f7:1304e0a2bdc:-7ffb-00000000017e1605 TIMESTAMP = 1312554602698
    WatchAlarmType: AutomaticReset
    WatchAlarmResetPeriod: 600000
    >
    Edited by: Brutus35 on Aug 5, 2011 8:04 AM

    my impression is that you have trouble on the DB,
    monitor the sessions to see if there is some table locked, and increase the number of processes on the DB
    you can use SQLDeveloper for that
    anyway your TX should rollback after some time, check that you have a timeout in place, it's better to rollback a TX than having the entire server crash

  • Stuck Thread causing heap to grow steeply and resulting in crash

    Hi,
    We have an ADF application deployed in weblogic 10.3.2.
    We have few PL/SQL's and SQL's which are complex and take around 5 minutes of time to complete at times. We also have the script running in database to kill long running queries so that it helps us to get rid of weblogic stuck threads to some extent.
    Now when ever there is an SQL running for long time , we see a stuck thread casuing heap to grow very fast and casuing server to crash.
    I know after a particular point of idle time thread is marked as stuck , but why does it filling the heap space - does stuck threads always causes heap to grow or is it just my specific case
    any pointers on how to debug this issue to find the root cause.
    Thanks,
    Naresh

    I think this setup needs profiling. To start-off, i would recommend collecting heap dump or jrf (if jrockit), instructions for them are available on net, as it varies with jvm versions and vendors.
    Post that you should look for the thread which you believe is stuck and a possible cause using OQL in memory analyzer tool over the heap dumps.
    Conceptually, till the time the objects (or data if you say) is being referenced, JVM cannot garbage collect it. So, I would say we need to figure out what are the contents of the heap, before coming to a conclusion that the threads which are stuck is causing the issue. Why I say so it that, I have seen in past that many-a-times, we suspect stuck threads to be an issue but it turns out to be that we are "knocking the wrong door".
    Long running threads (aka STUCK thread in WLS), will normally cause memory issues, if the references/task that its trying to perform has a large local data (i.e. data with the thread scope), so it the time the thread is not released the data is alive (irrespective of the fact that the data is used or not).
    Now, my first thought of this situation is that stuck thread is not an issue, considering that you are not able to replicate this in UAT post simulating stuck threads. Thus please collect some heap dumps (may be when you start observing memory increase) and then review them comparatively, I think that should be helpful.
    AJ

  • What causes four beeps followed by a system shut down?

    What causes forced shut down proceed by four beeps?

    For products produced after 1999:
    4 beeps = no good boot images in the boot ROM (and/or bad sys config block)

  • Weblogic 8.14: Stuck Threads during JMS - IO native write operations

    G'day!
    Hope somebody can advise a solution on the following...
    Recently message loads have started to increase on our weblogic 8.14 cluster (3 servers) and we started to see an almost daily occurrence of Stuck Thread issues. From thread dumps we could see these were caused by jrockit/net/SocketNativeIO.write processes. We concluded we had reached the max performance limit for the file based JMS stores and so we moved these to database based JMS stores. This certainly seemed to improve matters - but we have recently seen the issue return. Which is puzzling, as technically we thought all messages would be persisted to the database, and any file IO on the application server would be eliminated. Our database resides on a separate dedicated oracle server. We tracked a recent cause to a large message (200K) which caused the Struck Thread - resulting in a build up of pending messages. After bouncing the server the message processed ok. Any thoughts on how to overcome these issues - greatly appreciated. Here's some detail on the stuck thread from the thread dump: (+Why does weblogic use IO when the JMS store is on a database?+). All messages are persistant.
    "ExecuteThread: '6' for queue: 'weblogic.kernel.Default'" prio=5 id=0xc00 pid=27544 active, daemon
    at jrockit/net/SocketNativeIO.write(Native Method)@0xf3aeae10
    at jrockit/net/SocketNativeIO.write(Unknown Source)@0xf3aeae9c
    at jrockit/io/NativeIO.write(Unknown Source)@0xf3ae62c6
    at java/net/AbstractSocketImpl$2.write(Unknown Source)@0xf3aeada4
    at jrockit/io/NativeIOOutputStream.write(Unknown Source)@0xf3ae61ec
    at jrockit/io/NativeIOOutputStream.write(Unknown Source)@0xf3ae618d
    at java/io/DataOutputStream.write(Unknown Source)@0xf4e3ec25
    ^-- Holding lock: java/io/DataOutputStream@0x4309fc80[thin lock]
    at com/ibm/mq/MQInternalCommunications.send(MQInternalCommunications.java:1022)@0xed34364a
    ^-- Holding lock: com/ibm/mq/MQInternalCommunications@0x4309f7e0[thin lock]
    at com/ibm/mq/MQSESSIONClient.lowLevelComms(MQSESSIONClient.java:2832)@0xed342fbd
    ^-- Holding lock: java/lang/Integer@0x4309f8a0[thin lock]
    at com/ibm/mq/MQSESSIONClient.MQPUT(MQSESSIONClient.java:3844)@0xed34271b
    at com/ibm/mq/MQQueue.putMsg2(MQQueue.java:1486)@0xed342262
    ^-- Holding lock: com/ibm/mq/MQSPIQueue@0x4312ce80[thin lock]
    at com/ibm/mq/jms/MQMessageProducer.sendInternal(MQMessageProducer.java:1569)@0xed33d1ab
    at com/ibm/mq/jms/MQMessageProducer.send(MQMessageProducer.java:1012)@0xed860a49
    at com/ibm/mq/jms/MQMessageProducer.send(MQMessageProducer.java:1046)@0xed8605f4
    at weblogic/jms/adapter/JMSBaseConnection.sendInternal(JMSBaseConnection.java:667)@0xef7df0bf
    ^-- Holding lock: weblogic/jms/adapter/JMSBaseConnection@0x43093748[thin lock]
    at weblogic/jms/adapter/JMSBaseConnection.access$200(JMSBaseConnection.java:80)@0xef7dee28
    at weblogic/jms/adapter/JMSBaseConnection$6.run(JMSBaseConnection.java:647)@0xef7dee1d
    at weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)@0xf2fcbb88
    at weblogic/security/service/SecurityManager.runAs(SecurityManager.java:147)@0xf2fcbb0a
    at weblogic/jms/adapter/JMSBaseConnection.send(JMSBaseConnection.java:644)@0xef7decf3
    at weblogic/jms/adapter/JMSConnectionHandle.send(JMSConnectionHandle.java:144)@0xef7dec7c
    at jrockit/reflect/NativeMethodInvoker.invoke0(Native Method)@0xf4f10d10
    at jrockit/reflect/NativeMethodInvoker.invoke(Unknown Source)@0xf4f10e98
    at jrockit/reflect/VirtualNativeMethodInvoker.invoke(Unknown Source)@0xf4e86d70
    at java/lang/reflect/Method.invoke(Unknown Source)@0xf4f104c3
    at weblogic/connector/common/internal/ConnectionWrapper.invoke(ConnectionWrapper.java:149)@0xed96a498
    at $Proxy14.send(Unknown Source)@0xef7dec04
    at weblogic/jms/bridge/internal/MessagingBridge.onMessageInternal(MessagingBridge.java:1258)@0xed857b25
    at weblogic/jms/bridge/internal/MessagingBridge.onMessage(MessagingBridge.java:1181)@0xed857864
    at weblogic/jms/adapter/JMSBaseConnection$27.run(JMSBaseConnection.java:1943)@0xef7de623
    at weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)@0xf2fcbb88
    at weblogic/security/service/SecurityManager.runAs(SecurityManager.java:147)@0xf2fcbb0a
    at weblogic/jms/adapter/JMSBaseConnection.onMessage(JMSBaseConnection.java:1939)@0xef7de586
    at weblogic/jms/client/JMSSession.onMessage(JMSSession.java:2678)@0xeee7e30c
    at weblogic/jms/client/JMSSession.execute(JMSSession.java:2598)@0xeee7d25d
    at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:219)@0xf4e14070
    at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:178)@0xf4e1fddb
    at java/lang/Thread.startThreadFromVM(Unknown Source)@0xf4f3c9f3
    Thanks
    C.

    I'm not sure what operating system you're running, but I only have experience of WLS8.1 on Windows.
    We did extensive load testing of JMS before we went live, and found JDBC file stores to be rubbish in comparison to file store. When a message is persisted, it has to be written to the database, then when it is consumed, it must be deleted from the table. This simply killed our system when we load tested it.
    One thing we did do though, was use a second JMS server and second file store for one particularly heavy queue, where we did have IO issues. This was with a JMS distributed destination. We also use a synchronous write policy of cache-flush on the file store and we also have configured a paging store to page messages so the app server does not run out of memory.
    200K isn't by any means a huge message though, and shouldn't be any problem at all, we have 1mb messages quite regularly.
    I notice from the stack trace, that you have MQ series aswell. I'm not experienced with MQ, so can't give any information, but what I would try is the following.
    Using a JMS server with JMS File storage, develop a small java client to write as many messages to the queue and effectively stress the JMS subsystem just to establish if the filestore is the problem. This should be done without a connection to MQ and also use large messages. if you don't get any issues and WebLogic JMS with file store looks ok, then attach a consumer to the queue, which should also be a simple java client. Then you have production and consumption going on, which will result in writes and read's/deletes from the jms file store.
    Once you've stress tested JMS and you are satisfied that its ok or not, then add the MQseries connection and see how that affects things.
    You may well have already done all this, so please feel free to ignore me! I did a load of this kind of stuff before we went live to try and bash JMS to bits, but I didn't manage it! The only thing we did was add a second file store and second JMS server.
    I just looked back in my support cases, and found we'd had a similar issue - disk fragmentation seemed to be one of the factors here too. The JMS file store was heavily fragmented. That's windows for you though!!!
    Hope that helps.
    Pete

  • Help on stuck thread

    Hi, From my server log I noticed that there are some stuck threads. After looked at the thread dump thread 18 is blocking. MainPortalServlet.java is extends PortalServlet and line 96 is super.service(request,response); Please help me to analyse why it is blocking. Thanks.
    "ExecuteThread: '20' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x0090e6b0 nid=0x3b waiting for monitor entry [0x600fd000..0x600ffc28]
         at com.bea.wlw.netui.script.el.util.ParseUtils.parse(ParseUtils.java:100)
         - waiting to lock <0x8d3a3f18> (a com.bea.wlw.netui.script.el.util.ParseUtils$Cache)
         at com.bea.wlw.netui.script.el.ExpressionEvaluatorImpl.containsExpression(ExpressionEvaluatorImpl.java:239)
         at com.bea.wlw.netui.pageflow.ProcessPopulate.populate(ProcessPopulate.java:295)
         at com.bea.wlw.netui.pageflow.PageFlowRequestProcessor.processPopulate(PageFlowRequestProcessor.java:402)
         at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:254)
         at com.bea.wlw.netui.pageflow.PageFlowRequestProcessor.process(PageFlowRequestProcessor.java:691)
         at com.bea.wlw.netui.pageflow.AutoRegisterActionServlet.process(AutoRegisterActionServlet.java:527)
         at com.bea.wlw.netui.pageflow.PageFlowActionServlet.process(PageFlowActionServlet.java:152)
         at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:507)
         at com.bea.wlw.netui.pageflow.PageFlowUtils.strutsLookup(PageFlowUtils.java:1786)
         at com.bea.portlet.adapter.scopedcontent.ScopedContentCommonSupport.executeAction(ScopedContentCommonSupport.java:561)
         at com.bea.portlet.adapter.scopedcontent.ScopedContentCommonSupport.processActionInternal(ScopedContentCommonSupport.java:121)
         at com.bea.portlet.adapter.scopedcontent.PageFlowStubImpl.processAction(PageFlowStubImpl.java:98)
         at com.bea.netuix.servlets.controls.content.NetuiContent.raiseScopedAction(NetuiContent.java:154)
         at com.bea.netuix.servlets.controls.content.NetuiContent.raiseScopedAction(NetuiContent.java:113)
         at com.bea.netuix.servlets.controls.content.NetuiContent.handlePostbackData(NetuiContent.java:228)
         at com.bea.netuix.nf.ControlLifecycle$3.visit(ControlLifecycle.java:171)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:356)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:126)
         at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:105)
         at com.bea.netuix.nf.Lifecycle.runInbound(Lifecycle.java:173)
         at com.bea.netuix.nf.Lifecycle.run(Lifecycle.java:137)
         at com.bea.netuix.servlets.manager.UIServlet.runLifecycle(UIServlet.java:324)
         at com.bea.netuix.servlets.manager.UIServlet.processControlTree(UIServlet.java:220)
         at com.bea.netuix.servlets.manager.PortalServlet.doPost(PortalServlet.java:820)
         at com.bea.netuix.servlets.manager.UIServlet.service(UIServlet.java:150)
         at com.cd.servlets.portal.MainPortalServlet.service(MainPortalServlet.java:96)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
         at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
         at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:28)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
         at com.bea.p13n.servlets.PortalServletFilter.doFilter(PortalServletFilter.java:293)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
         at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:7053)
         at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
         at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
         at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3902)
         at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2773)
         at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)
    "ExecuteThread: '19' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x0090dab8 nid=0x3a runnable [0x601fb000..0x601ffc28]
         at java.util.HashMap.getEntry(HashMap.java:361)
         at java.util.LinkedHashMap.get(LinkedHashMap.java:266)
         at com.bea.wlw.netui.script.el.util.ParseUtils.parse(ParseUtils.java:84)
         at com.bea.wlw.netui.script.el.util.ParseUtils.evaluate(ParseUtils.java:123)
         at com.bea.wlw.netui.script.el.ExpressionEvaluatorImpl.evaluateStrict(ExpressionEvaluatorImpl.java:87)
         at com.bea.wlw.netui.tags.AbstractBaseTag.evaluateExpressionInternal(AbstractBaseTag.java:573)
         at com.bea.wlw.netui.tags.AbstractBaseTag.evaluateExpression(AbstractBaseTag.java:362)
         at com.bea.wlw.netui.tags.databinding.script.GetData.doEndTag(GetData.java:249)
         at jsp_servlet._portlets.___portlet1._jspService(_portlet1.jsp:27)
         at weblogic.servlet.jsp.JspBase.service(JspBase.java:33)
         at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
         at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:28)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
         at com.bea.p13n.servlets.PortalServletFilter.doFilter(PortalServletFilter.java:293)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
         at com.bea.wlw.netui.pageflow.PageFlowJspFilter.doFilter(PageFlowJspFilter.java:250)
         - locked <0x78ccf6c8> (a portlets.Portlet1Controller)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
         at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:652)
         at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:431)
         at com.bea.wlw.netui.pageflow.scoping.internal.ScopedRequestDispatcher.include(ScopedRequestDispatcher.java:120)
         at com.bea.netuix.servlets.controls.content.JspContent.beginRender(JspContent.java:534)
         at com.bea.netuix.servlets.controls.content.NetuiContent.beginRender(NetuiContent.java:425)
         at com.bea.netuix.nf.ControlLifecycle$1.visit(ControlLifecycle.java:495)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:543)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:247)
         at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:105)
         at com.bea.netuix.nf.Lifecycle.run(Lifecycle.java:356)
         at com.bea.netuix.nf.UIControl.render(UIControl.java:536)
         at com.bea.netuix.servlets.controls.PresentationContext.render(PresentationContext.java:405)
         at com.bea.netuix.servlets.util.RenderToolkit.renderChild(RenderToolkit.java:123)
         at com.bea.netuix.servlets.jsp.taglib.RenderChild.doStartTag(RenderChild.java:58)
         at jsp_servlet._framework._skeletons._cd.__flowlayout._jspService(flowlayout.jsp:30)
         at weblogic.servlet.jsp.JspBase.service(JspBase.java:33)
         at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:348)
         at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:646)
         at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:431)
         at com.bea.netuix.servlets.controls.JspRenderer.renderAlt(JspRenderer.java:194)
         at com.bea.netuix.servlets.controls.JspRenderer.beginRender(JspRenderer.java:96)
         at com.bea.netuix.nf.ControlLifecycle$1.visit(ControlLifecycle.java:491)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:543)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:247)
         at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:105)
         at com.bea.netuix.nf.Lifecycle.run(Lifecycle.java:356)
         at com.bea.netuix.nf.UIControl.render(UIControl.java:536)
         at com.bea.netuix.servlets.controls.PresentationContext.render(PresentationContext.java:405)
         at com.bea.netuix.servlets.util.RenderToolkit.renderChild(RenderToolkit.java:123)
         at com.bea.netuix.servlets.jsp.taglib.RenderChild.doStartTag(RenderChild.java:58)
         at jsp_servlet._framework._skeletons._cd.__flowlayout._jspService(flowlayout.jsp:30)
         at weblogic.servlet.jsp.JspBase.service(JspBase.java:33)
         at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:348)
         at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:646)
         at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:431)
         at com.bea.netuix.servlets.controls.JspRenderer.renderAlt(JspRenderer.java:194)
         at com.bea.netuix.servlets.controls.JspRenderer.beginRender(JspRenderer.java:96)
         at com.bea.netuix.nf.ControlLifecycle$1.visit(ControlLifecycle.java:491)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:543)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:247)
         at com.bea.netuix.nf.Lifecycle.runOutbound(Lifecycle.java:204)
         at com.bea.netuix.nf.Lifecycle.run(Lifecycle.java:153)
         at com.bea.netuix.servlets.manager.UIServlet.runLifecycle(UIServlet.java:324)
         at com.bea.netuix.servlets.manager.UIServlet.processControlTree(UIServlet.java:220)
         at com.bea.netuix.servlets.manager.PortalServlet.doPost(PortalServlet.java:820)
         at com.bea.netuix.servlets.manager.PortalServlet.doGet(PortalServlet.java:671)
         at com.bea.netuix.servlets.manager.UIServlet.service(UIServlet.java:147)
         at com.cd.servlets.portal.MainPortalServlet.service(MainPortalServlet.java:116)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
         at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
         at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:28)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
         at com.bea.p13n.servlets.PortalServletFilter.doFilter(PortalServletFilter.java:293)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
         at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:7053)
         at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
         at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
         at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3902)
         at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2773)
         at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)
    "ExecuteThread: '18' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x0090cec0 nid=0x39 runnable [0x602fd000..0x602ffc28]
         at java.util.LinkedHashMap.transfer(LinkedHashMap.java:224)
         at java.util.HashMap.resize(HashMap.java:452)
         at java.util.LinkedHashMap.addEntry(LinkedHashMap.java:399)
         at java.util.HashMap.put(HashMap.java:392)
         at com.bea.wlw.netui.script.el.util.ParseUtils.parse(ParseUtils.java:100)
         - locked <0x8d3a3f18> (a com.bea.wlw.netui.script.el.util.ParseUtils$Cache)
         at com.bea.wlw.netui.script.el.ExpressionEvaluatorImpl.containsExpression(ExpressionEvaluatorImpl.java:239)
         at com.bea.wlw.netui.pageflow.ProcessPopulate.populate(ProcessPopulate.java:295)
         at com.bea.wlw.netui.pageflow.PageFlowRequestProcessor.processPopulate(PageFlowRequestProcessor.java:402)
         at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:254)
         at com.bea.wlw.netui.pageflow.PageFlowRequestProcessor.process(PageFlowRequestProcessor.java:691)
         at com.bea.wlw.netui.pageflow.AutoRegisterActionServlet.process(AutoRegisterActionServlet.java:527)
         at com.bea.wlw.netui.pageflow.PageFlowActionServlet.process(PageFlowActionServlet.java:152)
         at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:507)
         at com.bea.wlw.netui.pageflow.PageFlowUtils.strutsLookup(PageFlowUtils.java:1786)
         at com.bea.portlet.adapter.scopedcontent.ScopedContentCommonSupport.executeAction(ScopedContentCommonSupport.java:561)
         at com.bea.portlet.adapter.scopedcontent.ScopedContentCommonSupport.processActionInternal(ScopedContentCommonSupport.java:121)
         at com.bea.portlet.adapter.scopedcontent.PageFlowStubImpl.processAction(PageFlowStubImpl.java:98)
         at com.bea.netuix.servlets.controls.content.NetuiContent.raiseScopedAction(NetuiContent.java:154)
         at com.bea.netuix.servlets.controls.content.NetuiContent.raiseScopedAction(NetuiContent.java:113)
         at com.bea.netuix.servlets.controls.content.NetuiContent.handlePostbackData(NetuiContent.java:228)
         at com.bea.netuix.nf.ControlLifecycle$3.visit(ControlLifecycle.java:171)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:356)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:126)
         at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:105)
         at com.bea.netuix.nf.Lifecycle.runInbound(Lifecycle.java:173)
         at com.bea.netuix.nf.Lifecycle.run(Lifecycle.java:137)
         at com.bea.netuix.servlets.manager.UIServlet.runLifecycle(UIServlet.java:324)
         at com.bea.netuix.servlets.manager.UIServlet.processControlTree(UIServlet.java:220)
         at com.bea.netuix.servlets.manager.PortalServlet.doPost(PortalServlet.java:820)
         at com.bea.netuix.servlets.manager.UIServlet.service(UIServlet.java:150)
         at com.cd.servlets.portal.MainPortalServlet.service(MainPortalServlet.java:96)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
         at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
         at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:28)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
         at com.bea.p13n.servlets.PortalServletFilter.doFilter(PortalServletFilter.java:293)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
         at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:7053)
         at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
         at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
         at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3902)
         at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2773)
         at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)

    Hi user9503787,
    Can you please confirm if the Weblogic instance / java process you took the Thread Dump from was showing high CPU utilization?
    Your Thread Dump data is quite interesting and a perfect example of a non Thread safe HashMap combined with a read/write race condition:
    Thread #23 (hanging for 600 seconds+) is showing a potential infinite looping condition in at java.util.HashMap.getEntry(HashMap.java:361)
    Thread #22 (hanging for 600 seconds+) is showing a potential infinite looping condition while performing a resize() following a put() operation
    Weblogic Portal has some known issues which regards to usage of non Thread safe HashHap data structures; which under load can cause infinite looping and sharp surge of CPU (due to corruption of the internal HashMap index preventing exiting the internal loop); ultimately hanging the whole JVM.
    We had the same issue for one of our Weblogic Portal 10.0 production environment and Oracle provided a patch (via Collections.synchronizedMap() ).
    Please open a SR and Oracle, share these 2 Threads and discuss with the engineer about a patch / know issues regarding non Thread safe HashMap and infinite looping.
    Regards,
    P-H
    http://javaeesupportpatterns.blogspot.com/

  • Stuck threads reading blob column from db table

    WLS 10.3.5, JDK 1.6u29, Oracle 11g RAC, ojdbc6 latest driver
    We're having problems with stuck threads trying to read a blob column from a DB table. The query to extract the blob is a simple select, without any locking such as "for update" clauses or whatever. The blob's size is <= 100k.
    The thread dump shows the following stack trace:
    +"[STUCK] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" RUNNABLE native+
    +     java.net.SocketInputStream.socketRead0(Native Method)+
    +     java.net.SocketInputStream.read(SocketInputStream.java:129)+
    +     oracle.net.ns.Packet.receive(Packet.java:300)+
    +     oracle.net.ns.DataPacket.receive(DataPacket.java:106)+
    +     oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:315)+
    +     oracle.net.ns.NetInputStream.read(NetInputStream.java:260)+
    +     oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:105)+
    +     oracle.jdbc.driver.T4CMAREngine.getNBytes(T4CMAREngine.java:1517)+
    +     oracle.jdbc.driver.T4C8TTILobd.unmarshalLobData(T4C8TTILobd.java:476)+
    +     oracle.jdbc.driver.T4C8TTILob.readLOBD(T4C8TTILob.java:770)+
    +     oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:361)+
    +     oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)+
    +     oracle.jdbc.driver.T4C8TTILob.read(T4C8TTILob.java:146)+
    +     oracle.jdbc.driver.T4CConnection.getBytes(T4CConnection.java:2392)+
    +     oracle.sql.BLOB.getBytes(BLOB.java:348)+
    +     oracle.sql.BLOB.getBytes(BLOB.java:222)+
    +     weblogic.jdbc.wrapper.Blob_oracle_sql_BLOB.getBytes(Unknown Source)+
    +     com.ibatis.sqlmap.engine.type.BlobTypeHandlerCallback.getResult(BlobTypeHandlerCallback.java:33)+
    +     com.ibatis.sqlmap.engine.type.CustomTypeHandler.getResult(CustomTypeHandler.java:52)+
    +     com.ibatis.sqlmap.engine.mapping.result.ResultMap.getPrimitiveResultMappingValue(ResultMap.java:619)+
    +     com.ibatis.sqlmap.engine.mapping.result.ResultMap.getResults(ResultMap.java:345)+
    +     com.ibatis.sqlmap.engine.execution.SqlExecutor.handleResults(SqlExecutor.java:384)+
    +     com.ibatis.sqlmap.engine.execution.SqlExecutor.handleMultipleResults(SqlExecutor.java:300)+
    +     com.ibatis.sqlmap.engine.execution.SqlExecutor.executeQuery(SqlExecutor.java:189)+
    +     com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.sqlExecuteQuery(MappedStatement.java:221)+
    +     com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.executeQueryWithCallback(MappedStatement.java:189)+
    +     com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.executeQueryForList(MappedStatement.java:139)+
    +     com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryForList(SqlMapExecutorDelegate.java:567)+
    +     com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryForList(SqlMapExecutorDelegate.java:541)+
    +     com.ibatis.sqlmap.engine.impl.SqlMapSessionImpl.queryForList(SqlMapSessionImpl.java:118)+
    +     com.ibatis.sqlmap.engine.impl.SqlMapClientImpl.queryForList(SqlMapClientImpl.java:94)+
    +     com.ibatis.dao.client.template.SqlMapDaoTemplate.queryForList(SqlMapDaoTemplate.java:282)"[STUCK] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" RUNNABLE native+
    +     java.net.SocketInputStream.socketRead0(Native Method)+
    +     java.net.SocketInputStream.read(SocketInputStream.java:129)+
    +     oracle.net.ns.Packet.receive(Packet.java:300)+
    +     oracle.net.ns.DataPacket.receive(DataPacket.java:106)+
    +     oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:315)+
    +     oracle.net.ns.NetInputStream.read(NetInputStream.java:260)+
    +     oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:105)+
    +     oracle.jdbc.driver.T4CMAREngine.getNBytes(T4CMAREngine.java:1517)+
    +     oracle.jdbc.driver.T4C8TTILobd.unmarshalLobData(T4C8TTILobd.java:476)+
    +     oracle.jdbc.driver.T4C8TTILob.readLOBD(T4C8TTILob.java:770)+
    +     oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:361)+
    +     oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)+
    +     oracle.jdbc.driver.T4C8TTILob.read(T4C8TTILob.java:146)+
    +     oracle.jdbc.driver.T4CConnection.getBytes(T4CConnection.java:2392)+
    +     oracle.sql.BLOB.getBytes(BLOB.java:348)+
    +     oracle.sql.BLOB.getBytes(BLOB.java:222)+
    +     weblogic.jdbc.wrapper.Blob_oracle_sql_BLOB.getBytes(Unknown Source)+
    +     com.ibatis.sqlmap.engine.type.BlobTypeHandlerCallback.getResult(BlobTypeHandlerCallback.java:33)+
    +     com.ibatis.sqlmap.engine.type.CustomTypeHandler.getResult(CustomTypeHandler.java:52)+
    +     com.ibatis.sqlmap.engine.mapping.result.ResultMap.getPrimitiveResultMappingValue(ResultMap.java:619)+
    +     com.ibatis.sqlmap.engine.mapping.result.ResultMap.getResults(ResultMap.java:345)+
    +     com.ibatis.sqlmap.engine.execution.SqlExecutor.handleResults(SqlExecutor.java:384)+
    +     com.ibatis.sqlmap.engine.execution.SqlExecutor.handleMultipleResults(SqlExecutor.java:300)+
    +     com.ibatis.sqlmap.engine.execution.SqlExecutor.executeQuery(SqlExecutor.java:189)+
    +     com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.sqlExecuteQuery(MappedStatement.java:221)+
         com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.executeQueryWithCallback(MappedStatement.java:189)
    +     com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.executeQueryForList(MappedStatement.java:139)+
    +     com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryForList(SqlMapExecutorDelegate.java:567)+
    +     com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryForList(SqlMapExecutorDelegate.java:541)+
    +     com.ibatis.sqlmap.engine.impl.SqlMapSessionImpl.queryForList(SqlMapSessionImpl.java:118)+
    +     com.ibatis.sqlmap.engine.impl.SqlMapClientImpl.queryForList(SqlMapClientImpl.java:94)+
    +     com.ibatis.dao.client.template.SqlMapDaoTemplate.queryForList(SqlMapDaoTemplate.java:282)+
    Some threads eventually end (after 1-2 hours), most of them remain there for days.
    Any hint would be quite useful, thanks.

    Threads are executing the actual allocated request from the Weblogic Kernel. Most of the problems happen when the Thread execution is reaching the application or business layer.
    At this point your application Java code module is sending or receiving data from external sources such as a an Oracle database for example. Any problem with such external system will cause the Thread to hang and wait for data to come back.
    Other situations can occur such as internal deadlock, infinite looping, heavy IO contention on your server etc.
    Doesn't loo like a driver issue.
    http://docs.oracle.com/cd/E21764_01/doc.1111/e14770/weblogic_server_issues.htm#autoId2
    Check at the Database end
    Cheers ...

  • ST3B14 : xdo  ReportService.getReportDefinition gives stuck thread

    We use the following code in List LOV source VVO to get the template names associated with the report:
    private static List<String> mTemplates = null;
    private static ReportService mReportService = null;
    private static int mCurrentRow = 0;
    private void initReportService() { 
    try { 
    mReportService = PPBIPUtil.getReportService();
    } catch (Throwable t) {
    t.printStackTrace();
    protected void executeQueryForCollection(Object qc, Object[] params,
    int noUserParams) {
    System.out.println("Nitesh : Entering method executeQueryForCollection");
    String reportPath =
    "/Financials/Receivables/BFBPrintSummary.xdo";
    try {
    ReportDefinition reportDefinition =
    mReportService.getReportDefinition(reportPath);
    System.out.println("Nitesh : Getting the templates");
    mTemplates = reportDefinition.getTemplateIds().getItem();
    System.out.println("Nitesh : mTemplates.size = "+ mTemplates.size());
    I have a local ST3B14 deployment pointing to the st3b14 debug db and I see that when the xdo is not uploaded to the BIP server, the parameterVO UI comes up fine, obviously the template LOV is blank as expected as it cannot find the report. However, when I upload the report the highlighted code :
    ReportDefinition reportDefinition =
    mReportService.getReportDefinition(reportPath);
    gives a stuck thread and execution hangs at this point. Here is what I see in the Adminserver logs:
    oracle.j2ee.ws.client.jaxws.WsClientProxyInvocationHandler.invoke(WsClientProxyInvocationHandler.java:142)
    $Proxy237.getReportDefinition(Unknown Source)
    oracle.xdo.service.client.ReportService.getReportDefinition(ReportService.java:307)
    oracle.apps.financials.receivables.transactions.balanceForwardBills.publicEssModel.view.BFBSummaryReportTemplateVVOImpl.executeQueryForCollection(BFBSummaryReportTemplateVVOImpl.java:55)
    oracle.jbo.server.ViewRowSetImpl.execute(ViewRowSetImpl.java:1036)
    ^-- Holding lock: oracle.jbo.JboSyncLock@75080cf[thin lock]
    oracle.jbo.server.ViewRowSetImpl.execute(ViewRowSetImpl.java:1021)
    oracle.jbo.server.ViewRowSetIteratorImpl.ensureRefreshed(ViewRowSetIteratorImpl.java:2738)
    ^-- Holding lock: oracle.jbo.JboSyncLock@75080cf[thin lock]
    oracle.jbo.server.ViewRowSetIteratorImpl.ensureRefreshed(ViewRowSetIteratorImpl.java:2730)
    oracle.jbo.server.ViewRowSetIteratorImpl.getAllRowsInRangeInternal(ViewRowSetIteratorImpl.java:2107)
    ^-- Holding lock: oracle.jbo.JboSyncLock@75080cf[thin lock]
    oracle.jbo.server.ViewRowSetIteratorImpl.getAllRowsInRange(ViewRowSetIteratorImpl.java:2165)
    Ess server logs
    oracle.jdbc.driver.T4CPreparedStatement.fetch(T4CPreparedStatement.java:995)
    oracle.jdbc.driver.OracleResultSetImpl.close_or_fetch_from_next(OracleResultSetImpl.java:244)
    ^-- Holding lock: oracle.jdbc.driver.T4CConnection@13be4a9[thin lock]
    oracle.jdbc.driver.OracleResultSetImpl.next(OracleResultSetImpl.java:187)
    ^-- Holding lock: oracle.jdbc.driver.T4CConnection@13be4a9[thin lock]
    weblogic.jdbc.wrapper.ResultSet_oracle_jdbc_driver_OracleResultSetImpl.next(Unknown Source)
    Has somebody encountered this? Also, where I can find the XDO source code jar/library that I can import in Jdev to debug further.
    thanks
    Nitesh
    Edited by: user595705 on Nov 2, 2010 3:57 PM

    Hi,
    Any updates on this?How did u solve it?I am encountering a similar issue on my junits.
    Thanks

  • WorkManager entry for max-stuck-thread-time doesnot override the default

    Hi All,
    I am using WLS 10.0 and have configured the following workmanager entry in the config.xml:
    <work-manager>
    <name>myworkmanager</name>
    <target>AdminServer</target>
    <ignore-stuck-threads>false</ignore-stuck-threads>
    <work-manager-shutdown-trigger>
    <max-stuck-thread-time>60</max-stuck-thread-time>
    <stuck-thread-count>5</stuck-thread-count>
    </work-manager-shutdown-trigger>
    </work-manager>
    I want that the <BEA-000337> message should be logged after the work-manager-shutdown-trigger's max-stuck-thread-time exceeds (after 60 seconds).
    If more than 5 threads of applications which are dispatched to this workmanager becomes stuck for more than 60 sec. an additional request to this applications are responded with http code 503.
    However, no <BEA-000337> message is written in to the server’s log file.
    This message is only written to the log file, when the stuck-thread-max-time (e.g. set to 150 secs) element of the servers configuration is exceeded.
    Sample:
    ####<09.01.2009 13:31:58 IST> <Error> <WebLogicServer> <appcashd>
    <AdminServerrefs1m1vm1> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default
    (self-tuning)'> <<WLS Kernel>> <> <> <1231504318065>
    <BEA-000337> <[STUCK] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "151" seconds working on the request "Http Request: /testservlet/sleep", which is more than the configured time (StuckThreadMaxTime) of "150" seconds.
    Stack trace:
    java.lang.Thread.sleep(Native Method)
    at.compny.testservlet.SleepServlet.doGet(SleepServlet.java:61)
    I believe that the <BEA-000337> message should already be logged after the work-manager-shutdown-trigger's max-stuck-thread-time exceeds (ergo after 60 seconds).
    I would like to know if this is the default behavior or we are encountering a bug.
    Any pointers would be appreciated

    Hi,
    what I was thinking about is that the work manager indeed works since the next request after the specifies number of requests is returned with a 503 error.
    This makes me believe that it works !
    However, the behavior of the server regarding the <stuck-thread-max-time> is not overridden.
    I have specified this value as 60 seconds in the work manager.
    However, this message is only written to the log file, when the stuck-thread-max-time (e.g. set to 150 secs) element of the servers configuration is exceeded.
    Any Idea if this is the default behavior or if I am encountering a bug

  • WorkManager and ignore-stuck-thread setting

    In WL 10.3, using an application scoped work manager with the following .war deployment descriptors:
    web.xml:
    <resource-ref>
    <res-ref-name>wm/WorkManagerABC</res-ref-name>
    <res-type>commonj.work.WorkManager</res-type>
    <res-auth>Container</res-auth>
    <res-sharing-scope>Shareable</res-sharing-scope>
    </resource-ref>
    weblogic.xml:
    <work-manager>
    <name>WorkManagerABC</name>
    <fair-share-request-class>
    <name>abcClass</name>
    <fair-share>1000</fair-share>
    </fair-share-request-class>
    <ignore-stuck-threads>true</ignore-stuck-threads>
    <min-threads-constraint>
    <name>MinThreadsCountTen</name>
    <count>10</count>
    </min-threads-constraint>
    <max-threads-constraint>
    <name>MaxThreadsCountTwenty</name>
    <count>20</count>
    </max-threads-constraint>
    </work-manager>
    <wl-dispatch-policy>WorkManagerABC</wl-dispatch-policy>
    ( This particular app is meant to be a socket listener using javax.nio ). This results in WL seeing "stuck threads" after the default 600 seconds. In our clusters, our clusters are configured that 3 stuck threads cause a server restart by nm.
    So I created the same objects at the domain level ( workmanager, min/max/fair-share ), with the workmanager set to ignore stuck threads.
    web.xml
    <resource-ref>
    <res-ref-name>WorkManagerABC</res-ref-name>
    <res-type>commonj.work.WorkManager</res-type>
    <res-auth>Container</res-auth>
    <res-sharing-scope>Shareable</res-sharing-scope>
    </resource-ref>
    weblogic.xml
    <weblogic-version>10.3</weblogic-version>
    <!-- no work manager defined -->
    <wl-dispatch-policy>WorkManagerEni</wl-dispatch-policy>
    The app deploys and simple web pages are processed within the WorkManagerABC wm. Although WL still marks 4 threads as hoggers, none of them are stuck. So no restart, and the server Health stays OK.
    So a workManager at the server level obeys the "Ignore Stuck Threads" while an application scoped workmanager does not.

    I wasn't too disappointed that the application-scoped workmanager didn't obey the stuck threads; I'd rather have the work manager configurable at the server console level without having to involve customized deployment plans to override hard-coded thread restraint levels in the app's deployment descriptors.
    I think having it defined within the server console will make it simpler to manage the different performances in production versus qa, etc., than having it defined within the app and then adjusting it in each environment.

  • Need some help with stucked threads

    Hi, I am running an ADF application on Weblogic 10.3.4. I get a lot of stucked threads and can't figure out the reason.
    Here is my stack dump:
    <31.mar.2011 kl 08.36 CEST> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '21' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "625" seconds working on the request "weblogic.servlet.internal.ServletRequestImpl@237aaa4c[
    POST /faces/hovedside?_adf.ctrl-state=13zq3miluz_4 HTTP/1.1
    Connection: keep-alive
    Referer: https://please.help.me/?_afrLoop=8410499084123&_afrWindowMode=0&_afrWindowId=null
    Content-Length: 356
    Adf-Ads-Page-Id: 3
    Origin: https://please.help.me
    User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.204 Safari/534.16
    Content-Type: application/x-www-form-urlencoded; charset=UTF-8
    Adf-Rich-Message: true
    Accept: */*
    Accept-Encoding: gzip,deflate,sdch
    Accept-Language: nb-NO,nb;q=0.8,no;q=0.6,nn;q=0.4,en-US;q=0.2,en;q=0.2
    Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
    Cookie: JSESSIONID=1dQ7NJpXZhYGM11Sf2ppDz4LZwQM5FpL6yxDrj0cSlMTsTdWvPPN!-222523808
    ]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
    Thread-133 "[STUCK] ExecuteThread: '21' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, suspended, priority=1, DAEMON> {
    java.util.ArrayList.clone(ArrayList.java:250)
    oracle.jbo.uicli.binding.JUCtrlHierNodeBinding.getChildren(JUCtrlHierNodeBinding.java:719)
    oracle.jbo.uicli.binding.JUCtrlHierNodeBinding.findChildNode(JUCtrlHierNodeBinding.java:844)
    oracle.jbo.uicli.binding.JUCtrlHierBinding.bringNodeToRangeKeyPath(JUCtrlHierBinding.java:772)
    oracle.adfinternal.view.faces.model.binding.FacesCtrlHierBinding.bringNodeToRangeKeyPath(FacesCtrlHierBinding.java:101)
    oracle.adfinternal.view.faces.model.binding.RowDataManager.setRowKey(RowDataManager.java:119)
    oracle.adfinternal.view.faces.model.binding.FacesCtrlHierBinding$FacesModel.setRowKey(FacesCtrlHierBinding.java:633)
    org.apache.myfaces.trinidad.model.RowKeySetTreeImpl$PathIterator.nextModelKey(RowKeySetTreeImpl.java:745)
    org.apache.myfaces.trinidad.model.RowKeySetTreeImpl$PathIterator.nextItem(RowKeySetTreeImpl.java:740)
    org.apache.myfaces.trinidad.model.RowKeySetTreeImpl$PathIterator.next(RowKeySetTreeImpl.java:721)
    oracle.adfinternal.view.faces.renderkit.rich.TreeRendererUtils.initModel(TreeRendererUtils.java:288)
    oracle.adfinternal.view.faces.renderkit.rich.TreeRendererUtils.wrapModel(TreeRendererUtils.java:527)
    oracle.adfinternal.view.faces.renderkit.rich.TreeRendererUtils.getCollectionObject(TreeRendererUtils.java:340)
    oracle.adfinternal.view.faces.renderkit.rich.TreeTableRenderer.getCollectionObject(TreeTableRenderer.java:762)
    oracle.adfinternal.view.faces.renderkit.rich.TableRenderer.storeHeightAutosized(TableRenderer.java:2365)
    oracle.adfinternal.view.faces.renderkit.rich.TableRenderer.encodeAll(TableRenderer.java:482)
    oracle.adfinternal.view.faces.renderkit.rich.TreeTableRenderer.encodeAll(TreeTableRenderer.java:585)
    oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1252)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:326)
    org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:758)
    org.apache.myfaces.trinidad.component.UIXCollection.encodeEnd(UIXCollection.java:527)
    javax.faces.component.UIComponent.encodeAll(UIComponent.java:920)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:398)
    oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2608)
    oracle.adfinternal.view.faces.renderkit.rich.PanelGroupLayoutRenderer._encodeChild(PanelGroupLayoutRenderer.java:398)
    oracle.adfinternal.view.faces.renderkit.rich.PanelGroupLayoutRenderer.access$300(PanelGroupLayoutRenderer.java:30)
    oracle.adfinternal.view.faces.renderkit.rich.PanelGroupLayoutRenderer$EncoderCallback.processComponent(PanelGroupLayoutRenderer.java:582)
    oracle.adfinternal.view.faces.renderkit.rich.PanelGroupLayoutRenderer$EncoderCallback.processComponent(PanelGroupLayoutRenderer.java:573)
    org.apache.myfaces.trinidad.component.UIXComponent.processFlattenedChildren(UIXComponent.java:154)
    org.apache.myfaces.trinidad.component.UIXComponent.processFlattenedChildren(UIXComponent.java:285)
    org.apache.myfaces.trinidad.component.UIXComponent.encodeFlattenedChildren(UIXComponent.java:252)
    oracle.adfinternal.view.faces.renderkit.rich.PanelGroupLayoutRenderer.encodeAll(PanelGroupLayoutRenderer.java:236)
    oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1252)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:326)
    org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:758)
    javax.faces.component.UIComponent.encodeAll(UIComponent.java:920)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:398)
    oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2608)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeAllChildren(CoreRenderer.java:407)
    oracle.adfinternal.view.faces.renderkit.rich.PanelHeaderRenderer.renderChildrenAfterHelpAndInfo(PanelHeaderRenderer.java:471)
    oracle.adfinternal.view.faces.renderkit.rich.PanelHeaderRenderer._renderContentCell(PanelHeaderRenderer.java:939)
    oracle.adfinternal.view.faces.renderkit.rich.PanelHeaderRenderer.renderContentRow(PanelHeaderRenderer.java:425)
    oracle.adfinternal.view.faces.renderkit.rich.PanelHeaderRenderer.encodeAll(PanelHeaderRenderer.java:160)
    oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1252)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:326)
    org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:758)
    javax.faces.component.UIComponent.encodeAll(UIComponent.java:920)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:398)
    javax.faces.component.UIComponent.encodeAll(UIComponent.java:920)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:398)
    oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2608)
    oracle.adfinternal.view.faces.renderkit.rich.PanelGroupLayoutRenderer._encodeChild(PanelGroupLayoutRenderer.java:398)
    oracle.adfinternal.view.faces.renderkit.rich.PanelGroupLayoutRenderer.access$300(PanelGroupLayoutRenderer.java:30)
    oracle.adfinternal.view.faces.renderkit.rich.PanelGroupLayoutRenderer$EncoderCallback.processComponent(PanelGroupLayoutRenderer.java:582)
    oracle.adfinternal.view.faces.renderkit.rich.PanelGroupLayoutRenderer$EncoderCallback.processComponent(PanelGroupLayoutRenderer.java:573)
    org.apache.myfaces.trinidad.component.UIXComponent.processFlattenedChildren(UIXComponent.java:154)
    org.apache.myfaces.trinidad.component.UIXComponent.processFlattenedChildren(UIXComponent.java:285)
    org.apache.myfaces.trinidad.component.UIXComponent.encodeFlattenedChildren(UIXComponent.java:252)
    oracle.adfinternal.view.faces.renderkit.rich.PanelGroupLayoutRenderer.encodeAll(PanelGroupLayoutRenderer.java:236)
    oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1252)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:326)
    org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:758)
    javax.faces.component.UIComponent.encodeAll(UIComponent.java:920)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:398)
    oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2608)
    oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer.encodeCenterFacet(PanelStretchLayoutRenderer.java:769)
    oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer._encodeCenterPane(PanelStretchLayoutRenderer.java:1055)
    oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer._encodeMiddlePanes(PanelStretchLayoutRenderer.java:345)
    oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer.encodeAll(PanelStretchLayoutRenderer.java:122)
    oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1252)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:326)
    org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:758)
    javax.faces.component.UIComponent.encodeAll(UIComponent.java:920)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:398)
    oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2608)
    oracle.adf.view.rich.render.RichRenderer.encodeStretchedChild(RichRenderer.java:1998)
    oracle.adfinternal.view.faces.renderkit.rich.RegionRenderer.access$400(RegionRenderer.java:46)
    oracle.adfinternal.view.faces.renderkit.rich.RegionRenderer$ChildEncoderCallback.processComponent(RegionRenderer.java:562)
    oracle.adfinternal.view.faces.renderkit.rich.RegionRenderer$ChildEncoderCallback.processComponent(RegionRenderer.java:553)
    org.apache.myfaces.trinidad.component.UIXComponent.processFlattenedChildren(UIXComponent.java:154)
    org.apache.myfaces.trinidad.component.UIXComponent.processFlattenedChildren(UIXComponent.java:285)
    org.apache.myfaces.trinidad.component.UIXComponent.encodeFlattenedChildren(UIXComponent.java:252)
    oracle.adfinternal.view.faces.renderkit.rich.RegionRenderer._encodeChildren(RegionRenderer.java:229)
    oracle.adfinternal.view.faces.renderkit.rich.RegionRenderer.encodeAll(RegionRenderer.java:144)
    oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1252)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:326)
    org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:758)
    oracle.adf.view.rich.component.fragment.UIXRegion.encodeEnd(UIXRegion.java:292)
    javax.faces.component.UIComponent.encodeAll(UIComponent.java:920)
    oracle.adfinternal.view.faces.taglib.region.IncludeTag$FacetWrapper.encodeAll(IncludeTag.java:511)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:398)
    oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2608)
    oracle.adfinternal.view.faces.renderkit.rich.DecorativeBoxRenderer.encodeFacet(DecorativeBoxRenderer.java:361)
    oracle.adfinternal.view.faces.renderkit.rich.DecorativeBoxRenderer._encodeCenterPane(DecorativeBoxRenderer.java:529)
    oracle.adfinternal.view.faces.renderkit.rich.DecorativeBoxRenderer.encodeAll(DecorativeBoxRenderer.java:167)
    oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1252)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:326)
    org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:758)
    javax.faces.component.UIComponent.encodeAll(UIComponent.java:920)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:398)
    oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2608)
    oracle.adf.view.rich.render.RichRenderer.encodeStretchedChild(RichRenderer.java:1998)
    oracle.adfinternal.view.faces.renderkit.rich.PanelSplitterRenderer._renderPane(PanelSplitterRenderer.java:1267)
    oracle.adfinternal.view.faces.renderkit.rich.PanelSplitterRenderer.encodeAll(PanelSplitterRenderer.java:88)
    oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1252)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:326)
    org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:758)
    javax.faces.component.UIComponent.encodeAll(UIComponent.java:920)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:398)
    oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2608)
    oracle.adf.view.rich.render.RichRenderer.encodeStretchedChild(RichRenderer.java:1998)
    oracle.adfinternal.view.faces.renderkit.rich.PanelSplitterRenderer._renderPane(PanelSplitterRenderer.java:1267)
    oracle.adfinternal.view.faces.renderkit.rich.PanelSplitterRenderer.encodeAll(PanelSplitterRenderer.java:88)
    oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1252)
    oracle.adfinternal.view.faces.renderkit.rich.PanelSplitterRenderer.encodeAll(PanelSplitterRenderer.java:88)
    oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1252)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:326)
    org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:758)
    javax.faces.component.UIComponent.encodeAll(UIComponent.java:920)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:398)
    oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2608)
    oracle.adfinternal.view.faces.renderkit.rich.DecorativeBoxRenderer.encodeFacet(DecorativeBoxRenderer.java:361)
    oracle.adfinternal.view.faces.renderkit.rich.DecorativeBoxRenderer._encodeCenterPane(DecorativeBoxRenderer.java:529)
    oracle.adfinternal.view.faces.renderkit.rich.DecorativeBoxRenderer.encodeAll(DecorativeBoxRenderer.java:167)
    oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1252)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:326)
    org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:758)
    javax.faces.component.UIComponent.encodeAll(UIComponent.java:920)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:398)
    oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2608)
    oracle.adf.view.rich.render.RichRenderer.encodeStretchedChild(RichRenderer.java:1998)
    oracle.adfinternal.view.faces.renderkit.rich.PanelSplitterRenderer._renderPane(PanelSplitterRenderer.java:1267)
    oracle.adfinternal.view.faces.renderkit.rich.PanelSplitterRenderer.encodeAll(PanelSplitterRenderer.java:88)
    oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1252)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:326)
    org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:758)
    javax.faces.component.UIComponent.encodeAll(UIComponent.java:920)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:398)
    oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2608)
    oracle.adf.view.rich.render.RichRenderer.encodeStretchedChild(RichRenderer.java:1998)
    oracle.adfinternal.view.faces.renderkit.rich.PanelSplitterRenderer._renderPane(PanelSplitterRenderer.java:1267)
    oracle.adfinternal.view.faces.renderkit.rich.PanelSplitterRenderer.encodeAll(PanelSplitterRenderer.java:88)
    oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1252)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:326)
    org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:758)
    javax.faces.component.UIComponent.encodeAll(UIComponent.java:920)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:398)
    oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2608)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeAllChildren(CoreRenderer.java:407)
    oracle.adfinternal.view.faces.renderkit.rich.PageTemplateRenderer.encodeAll(PageTemplateRenderer.java:69)
    oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1252)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:326)
    org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:758)
    oracle.adf.view.rich.component.fragment.ContextSwitchingComponent.encodeEnd(ContextSwitchingComponent.java:155)
    javax.faces.component.UIComponent.encodeAll(UIComponent.java:920)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:398)
    oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2608)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeAllChildren(CoreRenderer.java:407)
    oracle.adfinternal.view.faces.renderkit.rich.FormRenderer.encodeAll(FormRenderer.java:158)
    oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1252)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:326)
    org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:758)
    javax.faces.component.UIComponent.encodeAll(UIComponent.java:920)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:398)
    oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2608)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeAllChildren(CoreRenderer.java:407)
    oracle.adfinternal.view.faces.renderkit.rich.DocumentRenderer.encodeAll(DocumentRenderer.java:725)
    oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1252)
    org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:326)
    org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:758)
    javax.faces.component.UIComponent.encodeAll(UIComponent.java:920)
    javax.faces.component.UIComponent.encodeAll(UIComponent.java:920)
    com.sun.faces.application.ViewHandlerImpl.doRenderView(ViewHandlerImpl.java:256)
    com.sun.faces.application.ViewHandlerImpl.renderView(ViewHandlerImpl.java:131)
    javax.faces.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:189)
    org.apache.myfaces.trinidadinternal.application.ViewHandlerImpl.renderView(ViewHandlerImpl.java:155)
    oracle.adfinternal.view.faces.lifecycle.LifecycleImpl._renderResponse(LifecycleImpl.java:773)
    oracle.adfinternal.view.faces.lifecycle.LifecycleImpl._executePhase(LifecycleImpl.java:252)
    oracle.adfinternal.view.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:207)
    javax.faces.webapp.FacesServlet.service(FacesServlet.java:246)
    oracle.adfinternal.view.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:207)
    javax.faces.webapp.FacesServlet.service(FacesServlet.java:246)
    weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
    weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
    weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:206)
    weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    oracle.adf.model.servlet.ADFBindingFilter.doFilter(ADFBindingFilter.java:147)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    oracle.adfinternal.view.faces.webapp.rich.RegistrationFilter.doFilter(RegistrationFilter.java:69)
    org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:437)
    oracle.adfinternal.view.faces.activedata.AdsFilter.doFilter(AdsFilter.java:44)
    org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:437)
    org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl._doFilterImpl(TrinidadFilterImpl.java:231)
    org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl.doFilter(TrinidadFilterImpl.java:133)
    org.apache.myfaces.trinidad.webapp.TrinidadFilter.doFilter(TrinidadFilter.java:91)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    oracle.adf.library.webapp.LibraryFilter.doFilter(LibraryFilter.java:163)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:111)
    oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:299)
    oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:405)
    oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:94)
    oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:98)
    oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:70)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:86)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:25)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3687)
    weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3681)
    weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:308)
    weblogic.security.service.SecurityManager.runAs(SecurityManager.java:116)
    weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2213)
    weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2135)
    weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1420)
    weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
    weblogic.work.ExecuteThread.run(ExecuteThread.java:168)
    Please help.
    Thomas

    Hi Thomas,
    Your Thread is showing a processor computing operation ArrayList.clone() triggered from the Oracle JBO framework / API.
    Few questions for you:
    -     Are you using Solaris OS? If yes can you please capture the PRSTAT at the same time as your JVM Thread Dump using command:
    prstat -L 0 1 | grep <java pid>;kill -3 <java pid>
    -     How many threads are showing this condition?
    -     Was CPU utilization of the affected Java process very high on the server at that time?
    -     What is the utilization of your Java Heap at the time of the Thread Dump capture?
    This Thread could be stuck in an infinite loop since it is stuck for at least 600 seconds (10 minutes). It would probably help to generate a few more snapshots earlier and understand if Thread is hanging in the same code portion (possibility Oracle JBO) for every snapshot or not.
    This could also be data related (too many data rows) returned and causing high processing time for the clone operation from the Myfaces / JSF table that you are trying to render.
    Regards,
    P-H
    http://javaeesupportpatterns.blogspot.com/

  • What causes BUFFER GETS and PHYSICAL READS in INSERT operation to be high?

    Hi All,
    Am performing a huge number of INSERTs to a newly installed Oracle XE 10.2.0.1.0 on Windows. There is no SELECT statement running, but just INSERTs one after the other of 550,000 in count. When I monitor the SESSION I/O from Home > Administration > Database Monitor > Sessions, I see the following stats:
    BUFFER GETS = 1,550,560
    CONSISTENT GETS = 512,036
    PHYSICAL READS = 3,834
    BLOCK CHANGES = 1,034,232
    The presence of 2 stats confuses. Though the operation is just INSERT in database for this session, why should there be BUFFER GETS of this magnitude and why should there by PHYSICAL READS. Aren't these parameters for read operations? The BLOCK CHANGES value is clear as there are huge writes and the writes change these many blocks. Can any kind soul explain me what causes there parameters to show high value?
    The total columns in the display table are as follows (from the link mentioned above)
    1. Status
    2. SID
    3. Database Users
    4. Command
    5. Time
    6. Block Gets
    7. Consistent Gets
    8. Physical Reads
    9. Block Changes
    10. Consistent Changes
    What does CONSISTENT GETS and CONSISTENT CHANGES mean in a typical INSERT operation? And does someone know which all tables are involved in getting these values?
    Thank,
    ...

    Flake wrote:
    Hans, gracias.
    The table just have 2 columns, both of which are varchar2 (500). No constraints, no indexes, neither foreign key references are in place. The total size of RAM in system is 1GB, and yes, there are other GUI's going on like Firefox browser, notepad and command terminals.
    But, what does these other applications have to do with Oracle BUFFER GETS, PHYSICAL READS etc.? Awaiting your reply.Total RAM is 1GB. If you let XE decide how much RAM is to be allocated to buffers, on startup that needs to be shared with any/all other applications. Let's say that leaves us with, say 400M for the SGA + PGA.
    PGA is used for internal stuff, such as sorting, which is also used in determing the layout of secondary facets such as indexes and uniqueness. Total PGA usage varies in size based on the number of connections and required operations.
    And then there's the SGA. That needs to cover the space requirement for the data dictionary, any/all stored procedures and SQL statements being run, user security and so on. As well as the buffer blocks which represent the tablespace of the database. Since it is rare that the entire tablespace will fit into memory, stuff needs to be swapped in and out.
    So - put too much space pressure on the poor operating system before starting the database, and the SGA may be squeezed. Put that space pressure on the system and you may enbd up with swapping or paging.
    This is one of the reasons Oracle professionals will argue for dedicated machines to handle Oracle software.

  • Weblogic 10.3.4 stuck thread after attempting to passify ApplicationModule

    Using ADF 11.1.1.4
    Is there any reason that Weblogic should ever end up with a stuck thread when attempting to passify an application module. See the stack trace below.
    There is no indication of what business components if any were responsible. It goes right up to the panel stretch layout and jsp template.
    ]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
      oracle.jbo.common.Base64$Base64OutputStream.getDeflatedBytes(Base64.java:135)
      oracle.jbo.common.Base64$Base64OutputStream.toString(Base64.java:116)
      oracle.jbo.common.VariableValueManagerImpl.writeVarsToXML(VariableValueManagerImpl.java:819)
      oracle.jbo.server.ViewObjectImpl.doPassivateSettings(ViewObjectImpl.java:19100)
      oracle.jbo.server.ApplicationModuleImpl.passivateVOs(ApplicationModuleImpl.java:7712)
      oracle.jbo.server.ApplicationModuleImpl.doPassivateState(ApplicationModuleImpl.java:7522)
      oracle.jbo.server.ApplicationModuleImpl.doPassivateAMState(ApplicationModuleImpl.java:7509)
      oracle.jbo.server.Serializer.passivate(Serializer.java:213)
      oracle.jbo.server.DBSerializer.passivateRootAM(DBSerializer.java:290)
      oracle.jbo.server.DBSerializer.passivateRootAM(DBSerializer.java:267)
      oracle.jbo.server.ApplicationModuleImpl.passivateStateInternal(ApplicationModuleImpl.java:5913)
      oracle.jbo.server.ApplicationModuleImpl.passivateState(ApplicationModuleImpl.java:5783)
      oracle.jbo.server.ApplicationPoolMessageHandler.doPoolMessage(ApplicationPoolMessageHandler.java:381)
      oracle.jbo.server.ApplicationModuleImpl.doPoolMessage(ApplicationModuleImpl.java:8933)
      oracle.jbo.common.ampool.ApplicationPoolImpl.sendPoolMessage(ApplicationPoolImpl.java:4496)
      oracle.jbo.common.ampool.ApplicationPoolImpl.manageReferencingState(ApplicationPoolImpl.java:1445)
      oracle.jbo.common.ampool.ApplicationPoolImpl.recycleReferencedInstance(ApplicationPoolImpl.java:2544)
      oracle.jbo.common.ampool.ApplicationPoolImpl.prepareApplicationModule(ApplicationPoolImpl.java:2365)
      oracle.jbo.common.ampool.ApplicationPoolImpl.doCheckout(ApplicationPoolImpl.java:2270)
      oracle.jbo.common.ampool.ApplicationPoolImpl.useApplicationModule(ApplicationPoolImpl.java:3168)
      oracle.jbo.common.ampool.SessionCookieImpl.useApplicationModule(SessionCookieImpl.java:460)
      oracle.jbo.http.HttpSessionCookieImpl.useApplicationModule(HttpSessionCookieImpl.java:234)
      oracle.jbo.common.ampool.SessionCookieImpl.useApplicationModule(SessionCookieImpl.java:431)
      oracle.jbo.common.ampool.SessionCookieImpl.useApplicationModule(SessionCookieImpl.java:426)
      oracle.adf.model.bc4j.DCJboDataControl.initializeApplicationModule(DCJboDataControl.java:516)
      oracle.adf.model.bc4j.DCJboDataControl.getApplicationModule(DCJboDataControl.java:862)
      oracle.adf.model.binding.DCDataControl.setErrorHandler(DCDataControl.java:483)
      oracle.jbo.uicli.binding.JUApplication.setErrorHandler(JUApplication.java:261)
      oracle.adf.model.BindingContext.put(BindingContext.java:1300)
      oracle.adf.model.binding.DCDataControlReference.getDataControl(DCDataControlReference.java:174)
      oracle.adf.model.BindingContext.instantiateDataControl(BindingContext.java:1024)
      oracle.adf.model.dcframe.DataControlFrameImpl.doFindDataControl(DataControlFrameImpl.java:1282)
      oracle.adf.model.dcframe.DataControlFrameImpl.internalFindDataControl(DataControlFrameImpl.java:1186)
      oracle.adf.model.dcframe.DataControlFrameImpl.findDataControl(DataControlFrameImpl.java:1146)
      oracle.adf.model.BindingContext.internalFindDataControl(BindingContext.java:1149)
      oracle.adf.model.BindingContext.get(BindingContext.java:1102)
      oracle.adf.model.binding.DCParameter.evaluateValue(DCParameter.java:82)
      oracle.adf.model.binding.DCParameter.getValue(DCParameter.java:111)
      oracle.adf.model.binding.DCBindingContainer.getChildByName(DCBindingContainer.java:2708)
      oracle.adf.model.binding.DCBindingContainer.internalGet(DCBindingContainer.java:2756)
      oracle.adf.model.binding.DCExecutableBinding.get(DCExecutableBinding.java:115)
      oracle.adf.model.binding.DCUtil.findSpelObject(DCUtil.java:328)
      oracle.adf.model.binding.DCBindingContainer.evaluateParameterWithElCheck(DCBindingContainer.java:1460)
      oracle.adf.model.binding.DCBindingContainer.findDataControl(DCBindingContainer.java:1590)
      oracle.adf.model.binding.DCIteratorBinding.initDataControl(DCIteratorBinding.java:2470)
      oracle.adf.model.binding.DCIteratorBinding.getDataControl(DCIteratorBinding.java:2414)
      oracle.adf.model.binding.DCIteratorBinding.refresh(DCIteratorBinding.java:4313)
      oracle.adf.model.binding.DCExecutableBinding.refreshIfNeeded(DCExecutableBinding.java:341)
      oracle.adf.model.binding.DCIteratorBinding.getAllRowsInRange(DCIteratorBinding.java:2337)
      sun.reflect.GeneratedMethodAccessor437.invoke(Unknown Source)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      java.lang.reflect.Method.invoke(Method.java:597)
      oracle.jbo.common.JboBeanUtils.getProperty(JboBeanUtils.java:113)
      oracle.jbo.common.JboAbstractMap.internalGet(JboAbstractMap.java:97)
      oracle.adf.model.binding.DCExecutableBinding.internalGet(DCExecutableBinding.java:169)
      oracle.adf.model.binding.DCIteratorBinding.internalGet(DCIteratorBinding.java:4558)
      oracle.adf.model.binding.DCExecutableBinding.get(DCExecutableBinding.java:115)
      javax.el.MapELResolver.getValue(MapELResolver.java:164)
      com.sun.faces.el.DemuxCompositeELResolver._getValue(DemuxCompositeELResolver.java:173)
      com.sun.faces.el.DemuxCompositeELResolver.getValue(DemuxCompositeELResolver.java:200)
      com.sun.el.parser.AstValue.getValue(Unknown Source)
      com.sun.el.ValueExpressionImpl.getValue(Unknown Source)
      org.apache.myfaces.trinidad.bean.FacesBeanImpl.getProperty(FacesBeanImpl.java:68)
      org.apache.myfaces.trinidad.component.UIXComponentBase.getProperty(UIXComponentBase.java:1194)
      org.apache.myfaces.trinidad.component.UIXIterator.getValue(UIXIterator.java:706)
      org.apache.myfaces.trinidad.component.UIXCollection._flushCachedModel(UIXCollection.java:1386)
      org.apache.myfaces.trinidad.component.UIXCollection.__processFlattenedChildrenBegin(UIXCollection.java:1369)
      org.apache.myfaces.trinidad.component.UIXIterator.processFlattenedChildren(UIXIterator.java:156)
      org.apache.myfaces.trinidad.component.UIXComponent.processFlattenedChildren(UIXComponent.java:160)
      org.apache.myfaces.trinidad.component.UIXComponent.processFlattenedChildren(UIXComponent.java:290)
      org.apache.myfaces.trinidad.component.UIXComponent.processFlattenedChildren(UIXComponent.java:213)
      oracle.adfinternal.view.faces.renderkit.rich.PanelGroupLayoutRenderer.encodeAll(PanelGroupLayoutRenderer.java:315)
      oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1396)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:335)
      org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:767)
      javax.faces.component.UIComponent.encodeAll(UIComponent.java:937)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:399)
      oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2633)
      oracle.adfinternal.view.faces.renderkit.rich.ShowDetailItemRenderer.access$500(ShowDetailItemRenderer.java:31)
      oracle.adfinternal.view.faces.renderkit.rich.ShowDetailItemRenderer$ChildEncoderCallback.processComponent(ShowDetailItemRenderer.java:531)
      oracle.adfinternal.view.faces.renderkit.rich.ShowDetailItemRenderer$ChildEncoderCallback.processComponent(ShowDetailItemRenderer.java:504)
      org.apache.myfaces.trinidad.component.UIXComponent.processFlattenedChildren(UIXComponent.java:170)
      org.apache.myfaces.trinidad.component.UIXComponent.processFlattenedChildren(UIXComponent.java:290)
      org.apache.myfaces.trinidad.component.UIXComponent.encodeFlattenedChildren(UIXComponent.java:255)
      oracle.adfinternal.view.faces.renderkit.rich.ShowDetailItemRenderer._encodeChildren(ShowDetailItemRenderer.java:446)
      oracle.adfinternal.view.faces.renderkit.rich.ShowDetailItemRenderer.encodeAll(ShowDetailItemRenderer.java:115)
      oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1396)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:335)
      org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:767)
      javax.faces.component.UIComponent.encodeAll(UIComponent.java:937)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:399)
      oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2633)
      oracle.adf.view.rich.render.RichRenderer.encodeStretchedChild(RichRenderer.java:2004)
      oracle.adfinternal.view.faces.renderkit.rich.PanelAccordionRenderer._encodeNonScreenReaderDetailItem(PanelAccordionRenderer.java:1422)
      oracle.adfinternal.view.faces.renderkit.rich.PanelAccordionRenderer._encodeDetailItem(PanelAccordionRenderer.java:1483)
      oracle.adfinternal.view.faces.renderkit.rich.PanelAccordionRenderer.access$4800(PanelAccordionRenderer.java:52)
      oracle.adfinternal.view.faces.renderkit.rich.PanelAccordionRenderer$PanelEncoderCallback.processComponent(PanelAccordionRenderer.java:2503)
      oracle.adfinternal.view.faces.renderkit.rich.PanelAccordionRenderer$PanelEncoderCallback.processComponent(PanelAccordionRenderer.java:2447)
      org.apache.myfaces.trinidad.component.UIXComponent.processFlattenedChildren(UIXComponent.java:170)
      org.apache.myfaces.trinidad.component.UIXComponent.processFlattenedChildren(UIXComponent.java:290)
      org.apache.myfaces.trinidad.component.UIXComponent.encodeFlattenedChildren(UIXComponent.java:255)
      oracle.adfinternal.view.faces.renderkit.rich.PanelAccordionRenderer.encodeAll(PanelAccordionRenderer.java:456)
      oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1396)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:335)
      org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:767)
      javax.faces.component.UIComponent.encodeAll(UIComponent.java:937)
      oracle.adfinternal.view.faces.taglib.region.IncludeTag$FacetWrapper.encodeAll(IncludeTag.java:536)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:399)
      oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2633)
      oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer.encodeCenterFacet(PanelStretchLayoutRenderer.java:769)
      oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer._encodeCenterPane(PanelStretchLayoutRenderer.java:1140)
      oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer._encodeMiddlePanes(PanelStretchLayoutRenderer.java:348)
      oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer.encodeAll(PanelStretchLayoutRenderer.java:313)
      oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1396)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:335)
      org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:767)
      javax.faces.component.UIComponent.encodeAll(UIComponent.java:937)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:399)
      oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2633)
      oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer.encodeStartFacet(PanelStretchLayoutRenderer.java:805)
      oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer._encodeVerticalPane(PanelStretchLayoutRenderer.java:1369)
      oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer._encodeMiddlePanes(PanelStretchLayoutRenderer.java:347)
      oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer.encodeAll(PanelStretchLayoutRenderer.java:313)
      oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1396)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:335)
      org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:767)
      javax.faces.component.UIComponent.encodeAll(UIComponent.java:937)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:399)
      oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2633)
      oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer.encodeCenterFacet(PanelStretchLayoutRenderer.java:769)
      oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer._encodeCenterPane(PanelStretchLayoutRenderer.java:1140)
      oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer._encodeMiddlePanes(PanelStretchLayoutRenderer.java:348)
      oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer.encodeAll(PanelStretchLayoutRenderer.java:313)
      oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1396)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:335)
      org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:767)
      javax.faces.component.UIComponent.encodeAll(UIComponent.java:937)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:399)
      oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2633)
      oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer.encodeCenterFacet(PanelStretchLayoutRenderer.java:769)
      oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer._encodeCenterPane(PanelStretchLayoutRenderer.java:1140)
      oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer._encodeMiddlePanes(PanelStretchLayoutRenderer.java:348)
      oracle.adfinternal.view.faces.renderkit.rich.PanelStretchLayoutRenderer.encodeAll(PanelStretchLayoutRenderer.java:313)
      oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1396)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:335)
      org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:767)
      javax.faces.component.UIComponent.encodeAll(UIComponent.java:937)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:399)
      oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2633)
      oracle.adf.view.rich.render.RichRenderer.encodeStretchedChild(RichRenderer.java:2004)
      oracle.adfinternal.view.faces.renderkit.rich.PanelSplitterRenderer._renderPane(PanelSplitterRenderer.java:1353)
      oracle.adfinternal.view.faces.renderkit.rich.PanelSplitterRenderer.encodeAll(PanelSplitterRenderer.java:259)
      oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1396)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:335)
      org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:767)
      javax.faces.component.UIComponent.encodeAll(UIComponent.java:937)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:399)
      oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2633)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeAllChildren(CoreRenderer.java:415)
      oracle.adfinternal.view.faces.renderkit.rich.PageTemplateRenderer.encodeAll(PageTemplateRenderer.java:69)
      oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1396)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:335)
      org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:767)
      oracle.adf.view.rich.component.fragment.ContextSwitchingComponent.encodeEnd(ContextSwitchingComponent.java:155)
      javax.faces.component.UIComponent.encodeAll(UIComponent.java:937)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:399)
      oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2633)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeAllChildren(CoreRenderer.java:415)
      oracle.adfinternal.view.faces.renderkit.rich.FormRenderer.encodeAll(FormRenderer.java:220)
      oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1396)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:335)
      org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:767)
      javax.faces.component.UIComponent.encodeAll(UIComponent.java:937)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeChild(CoreRenderer.java:399)
      oracle.adf.view.rich.render.RichRenderer.encodeChild(RichRenderer.java:2633)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeAllChildren(CoreRenderer.java:415)
      oracle.adfinternal.view.faces.renderkit.rich.DocumentRenderer.encodeAll(DocumentRenderer.java:1181)
      oracle.adf.view.rich.render.RichRenderer.encodeAll(RichRenderer.java:1396)
      org.apache.myfaces.trinidad.render.CoreRenderer.encodeEnd(CoreRenderer.java:335)
      org.apache.myfaces.trinidad.component.UIXComponentBase.encodeEnd(UIXComponentBase.java:767)
      javax.faces.component.UIComponent.encodeAll(UIComponent.java:937)
      javax.faces.component.UIComponent.encodeAll(UIComponent.java:933)
      com.sun.faces.application.ViewHandlerImpl.doRenderView(ViewHandlerImpl.java:266)
      com.sun.faces.application.ViewHandlerImpl.renderView(ViewHandlerImpl.java:197)
      javax.faces.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:189)
      org.apache.myfaces.trinidadinternal.application.ViewHandlerImpl.renderView(ViewHandlerImpl.java:193)
      oracle.adfinternal.view.faces.lifecycle.LifecycleImpl._renderResponse(LifecycleImpl.java:777)
      oracle.adfinternal.view.faces.lifecycle.LifecycleImpl._executePhase(LifecycleImpl.java:293)
      oracle.adfinternal.view.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:213)
      javax.faces.webapp.FacesServlet.service(FacesServlet.java:266)
      weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
      weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
      weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:300)
      weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
      weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
      oracle.adf.model.servlet.ADFBindingFilter.doFilter(ADFBindingFilter.java:205)
      weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
      oracle.adfinternal.view.faces.webapp.rich.RegistrationFilter.doFilter(RegistrationFilter.java:106)
      org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:446)
      oracle.adfinternal.view.faces.activedata.AdsFilter.doFilter(AdsFilter.java:60)
      org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:446)
      org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl._doFilterImpl(TrinidadFilterImpl.java:271)
      org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl.doFilter(TrinidadFilterImpl.java:177)
      org.apache.myfaces.trinidad.webapp.TrinidadFilter.doFilter(TrinidadFilter.java:92)
      weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
      oracle.adf.library.webapp.LibraryFilter.doFilter(LibraryFilter.java:175)
      weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
      oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:111)
      java.security.AccessController.doPrivileged(Native Method)
      oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:313)
      oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:413)
      oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:94)
      oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:161)
      oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:71)
      weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
      oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:136)
      weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
      weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:27)
      weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
      weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3715)
      weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3681)
      weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
      weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
      weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2277)
      weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2183)
      weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1454)
      weblogic.work.ExecuteThread.execute(ExecuteThread.java:207)
      weblogic.work.ExecuteThread.run(ExecuteThread.java:176)
    SUBSYSTEM = WebLogicServer USERID = <WLS Kernel> SEVERITY = Error THREAD = [ACTIVE] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)' MSGID = BEA-000337 MACHINE = app-sahe.dmz TXID =  CONTEXTID = ed3140ecffa37389:3c925abd:13f6031c9fe:-8000-0000000000015dff TIMESTAMP = 1371715760129 
    WatchAlarmType: AutomaticReset
    WatchAlarmResetPeriod: 600000

    Was their any changes in any of configuration files or something else. If not, then try bouncing the system once again and see if its working.

Maybe you are looking for

  • Usage of LDB

    I want to try a report where I can use LDB.So tried as follows Report zgldb. Nodes:pap. When I just compile it says 'PAP is not a node of Logical database _S'.. what does this mean.How to use LDB in a report? Thank you

  • ITunes locking up iMac

    So, I've been fighting with my iMac for about a month now. Issue is that iTunes and/or the Finder locks up. Applications that I already have open seem to continue to function but if I leave my iMac on and come back to it, iTunes either won't open (ic

  • How to join MS SQL Server 2012 to Dreamweaver CS6

    Hello everyone. Here's my question: Is it possible to link a Microsoft SQL Server 2012 Database to Dreamweaver CS6, instead of the default MySQL Database? I don't particullarily like MySQL because it's interface isn't familiar to me. Instead, I use s

  • HT201272 An error keeps popping up whenever I try to download

    An error keeps popping up whenever I try to download some of my music in my iCloud and also when I attempt to download old Purchases of Albums and TV Shows. What should I do?

  • Add search criteria field to ERPQOrder

    I would like to add additional search criteria fields to the Dynamic Query Object ERPQOrder. I want to enhance the class CL_CRM_QORDER_RUN_ERPIL for the purpose. Is there a way to accomplish this requirement?