JDBC read stuck if RAC node goes down

We did several tests with Java applications against our RAC DB and face a hanging application if we power off the RAC node that executes the current (long) running query.
We can see that the application receives HA-events via UCP:
2015-01-22 13:02:11 | r-thread-1 | WARN  | o.ucp.jdbc.oracle.ONSDatabaseFailoverEvent    | NO timezone in HA event
However, the application started a query before and the query is not aborted with an exception. A Thread dump after about 7 minutes shows that the application is hanging in a socket read call:
"pool-1-thread-1" #32 prio=5 os_prio=0 tid=0x00007fedf45b2000 nid=0xbc4 runnable [0x00007fee00cd3000]
   java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:150)
    at java.net.SocketInputStream.read(SocketInputStream.java:121)
    at oracle.net.ns.Packet.receive(Packet.java:283)
    at oracle.net.ns.DataPacket.receive(DataPacket.java:103)
    at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:230)
    at oracle.net.ns.NetInputStream.read(NetInputStream.java:175)
    at oracle.net.ns.NetInputStream.read(NetInputStream.java:100)
    at oracle.net.ns.NetInputStream.read(NetInputStream.java:85)
    at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:123)
    at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:79)
    at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1122)
    at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1099)
    at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:288)
    at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:191)
    at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:523)
    at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:207)
    at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:863)
    at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1153)
    at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1275)
    at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3576)
    at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3620)
    - locked <0x00000000c0ddcb20> (a oracle.jdbc.driver.T4CConnection)
    at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1491)
    at org.springframework.jdbc.core.JdbcTemplate$1.doInPreparedStatement(JdbcTemplate.java:703)
The expected behaviour would be that a running query is aborted with an exception. (BTW: This happens if the service is taken down with "shutdown immediate". All ok for this case.)
We consider to implement custom ONS listeners [1], but we actually expect that UCP would handle such situations or lets us register strategies/callbacks for certain events.
Our config:
Oracle Enterprise 11.2.0.4.0 with RAC
ons.jar 12.1.0.1
ojdbc6.jar 11.2.0.2
ucp.jar 12.1.0.1
Server JRE 1.8.0_25
Any hints appreciated.
[1] http://docs.oracle.com/cd/E11882_01/java.112/e16548/apxracfan.htm#JJDBC28945

You're concept isn't right:
http://docs.oracle.com/cd/E11882_01/server.112/e25494/restart.htm#ADMIN13178
Overview of Fast Application Notification
FAN is a notification mechanism that Oracle Restart can use to notify other processes about configuration changes that include service status changes, such as UP or DOWN events. FAN provides the ability to immediately terminate inflight transaction when an instance or server fails. Integrated Oracle clients receive the events and respond. Applications can respond either by propagating the error to the user or by resubmitting the transactions and masking the error from the application user. When a DOWN event occurs, integrated clients immediately clean up connections to the terminated database. When an UP event occurs, the clients create new connections to the new primary database instance.
Also, take a look at these docs: http://docs.oracle.com/cd/E11882_01/java.112/e12265/rac.htm#JJUCP08100 ; and https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=890204623685515&id=566573.1&_afrWindowMode=0&_adf.ctrl-s…
And make a test, execute  a query that took about 1 minute and after you executed, just power down the node where it is executing, to see if it will retrieve the results.
Regards.

Similar Messages

  • Host server live migration causing Guest Cluster node goes down

    Hi 
    I have two node Hyper host cluster , Im using converged network for Host management,Live migartion and cluster network. And Separate NICs for ISCSI multi-pathing. When I live migrate the Guest node from one host to another , within guest cluster the node
    is going down.  I have increased clusterthroshold and clusterdelay values.  Guest nodes are connecting to ISCSI network directly from ISCSI initiator on Server 2012. 
    The converged networks for management ,cluster and live migration networks are built on top of a NIC Team with switch Independent mode and load balancing as Hyper V port. 
    I have VMQ enabled on Converged fabric  and jumbo frames enabled on ISCSI. 
    Can Anyone guess why would live migration cause failure on the guest node. 
    thanks
    mumtaz 

    Repost here: http://social.technet.microsoft.com/Forums/en-US/winserverhyperv/threads
    in the Hyper-V forum.  You'll get a lot more help there.
    This forum is for Virtual Server 2005.

  • Archive REDO When One RAC Node is Down

    I have a question about how redo log get archived when one of the instances in a two node RAC cluster is down (not open, not mounted).
    For example, let's assume instance1 was shutdown and only instance 2 is running.
    I have 6 redo logs:
    <font face="courier">
    SQL> SELECT GROUP#, THREAD#, SEQUENCE#, STATUS FROM V$LOG;
    GROUP# THREAD# SEQUENCE# STATUS
    1 1 3390 INACTIVE
    2 1 3389 INACTIVE
    3 1 3391 ACTIVE
    5 2 3886 INACTIVE
    4 2 3887 INACTIVE
    6 2 3888 CURRENT
    </font>
    If I run the following statement what will happen?
    <font face="courier">
    SQL> ALTER SYSTEM ARCHIVE LOG CURRENT;
    </font>
    The documentation says +Specify CURRENT to manually archive the current redo log file group of the specified thread, forcing a log switch. If you omit the THREAD parameter, then Oracle Database archives all redo log file groups from all enabled threads, including logs previous to current logs. You can specify CURRENT only when the database is open.+
    Would Oracle archive sequence# 3391 from thread 1 even though the instance is not open?

    When your instance are not working, it means that it doesn't have any CURRENT redo log. So when you issue switching logfile - it launches archiving only current redo log, for working instance only. That 3391 redo log is not current, it has Active status, that means that it could be needed for recovery purpose, but it had to be archived earlier.

  • SC 3.1 and Oracle 10g RAC: instance goes down when rebooting other node

    I have Sun cluster 3.1 with Oracle 10gR2 on Solaris 10 Sparc. Thanks to this forum now that my cluster seems fine with a database running. However I still have one problem: when I reboot node1, the instance on node2 also disappears. The instance on the node2 will make itself alive once node1 comes back. This happens also for the instance on node1 if I reboot node2.
    The interconnect cables are direct cross-over cable.
    Any input is appreciated,
    Luke

    Although I am not TIm I can anticipate his answer, as he gave it 3 topics back in this forum. You cannot mount UFS on top of shared SVM. It does not work as you can see with your own configuration. The only shared filesystem that works for RAC is shared QFS. The doc
    http://docs.sun.com/app/docs/doc/819-0583/6n30h62v7?a=view
    has all the details.
    If you need a shared filesystem for your binaries or whatever, you have to use UFS/PxFS but that sits on top of normal SVM and not shared SVM.
    Hartmut

  • ORA-12514 on R12 EBS Apps server when 1 DB RAC node crashed/down

    Just now Production 11.2.0.2 RAC DB on windows 2008 server Node1 crashed. While on Node 2 all services are up and running including database. But from EBS R12.1.2 application server when connecting as username/password from sql*plus is throwing ORA-12514 error.
    While It is connecting if I give username/password@TNSNAME but not without @TNSNAME. Due to this none of the application services are starting.
    Please help/advise. Thank you.
    Following is the tnsnames.ora,
    # This file is automatically generated by AutoConfig.  It will be read and
    # overwritten.  If you were instructed to edit this file, or if you are not
    # able to use the settings created by AutoConfig, refer to Metalink Note
    # 387859.1 for assistance.
    #$Header: NetServiceHandler.java 120.19.12010000.6 2010/03/09 08:11:36 jmajumde ship $
    ORCL=
            (DESCRIPTION=
                    (ADDRESS=(PROTOCOL=tcp)(HOST=orcldbscan)(PORT=1521))
                (CONNECT_DATA=
                    (SERVICE_NAME=ORCL)
                    (INSTANCE_NAME=ORCL1)
    ORCL1=
            (DESCRIPTION=
                    (ADDRESS=(PROTOCOL=tcp)(HOST=orcldbscan)(PORT=1521))
                (CONNECT_DATA=
                    (SERVICE_NAME=ORCL)
                    (INSTANCE_NAME=ORCL1)
    ORCL1_FO=
            (DESCRIPTION=
                    (ADDRESS=(PROTOCOL=tcp)(HOST=orcldbscan)(PORT=1521))
                (CONNECT_DATA=
                    (SERVICE_NAME=ORCL)
                    (INSTANCE_NAME=ORCL1)
    ORCL_FO=
            (DESCRIPTION=
                    (ADDRESS=(PROTOCOL=tcp)(HOST=orcldbscan)(PORT=1521))
                (CONNECT_DATA=
                    (SERVICE_NAME=ORCL)
                    (INSTANCE_NAME=ORCL1)
    ORCL1=
            (DESCRIPTION=
                    (ADDRESS=(PROTOCOL=tcp)(HOST=orcldbscan)(PORT=1521))
                (CONNECT_DATA=
                    (SERVICE_NAME=ORCL)
                    (INSTANCE_NAME=ORCL1)
    ORCL1_FO=
            (DESCRIPTION=
                    (ADDRESS=(PROTOCOL=tcp)(HOST=orcldbscan)(PORT=1521))
                (CONNECT_DATA=
                    (SERVICE_NAME=ORCL)
                    (INSTANCE_NAME=ORCL1)
    ORCL2=
            (DESCRIPTION=
                    (ADDRESS=(PROTOCOL=tcp)(HOST=orcldb2-vip.sa.company.net)(PORT=1521))
                (CONNECT_DATA=
                    (SERVICE_NAME=ORCL)
                    (INSTANCE_NAME=ORCL2)
    ORCL2_FO=
            (DESCRIPTION=
                    (ADDRESS=(PROTOCOL=tcp)(HOST=orcldb2-vip.sa.company.net)(PORT=1521))
                (CONNECT_DATA=
                    (SERVICE_NAME=ORCL)
                    (INSTANCE_NAME=ORCL2)
    ORCL_BALANCE=
            (DESCRIPTION=
                (ADDRESS_LIST=
                    (LOAD_BALANCE=YES)
                    (FAILOVER=YES)
                    (ADDRESS=(PROTOCOL=tcp)(HOST=orcldb1-vip.sa.company.net)(PORT=1521))
                    (ADDRESS=(PROTOCOL=tcp)(HOST=orcldb2-vip.sa.company.net)(PORT=1521))
                (CONNECT_DATA=
                    (SERVICE_NAME=ORCL)
    FNDFS_orclAPPL=
            (DESCRIPTION=
                    (ADDRESS=(PROTOCOL=tcp)(HOST=orclAPPL.sa.company.net)(PORT=1626))
                (CONNECT_DATA=
                    (SID=FNDFS)
    FNDFS_orclAPPL.sa.company.net=
            (DESCRIPTION=
                    (ADDRESS=(PROTOCOL=tcp)(HOST=orclAPPL.sa.company.net)(PORT=1626))
                (CONNECT_DATA=
                    (SID=FNDFS)
    FNDFS_ORCL_orclAPPL=
            (DESCRIPTION=
                    (ADDRESS=(PROTOCOL=tcp)(HOST=orclAPPL.sa.company.net)(PORT=1626))
                (CONNECT_DATA=
                    (SID=FNDFS)
    FNDFS_ORCL_orclAPPL.sa.company.net=
            (DESCRIPTION=
                    (ADDRESS=(PROTOCOL=tcp)(HOST=orclAPPL.sa.company.net)(PORT=1626))
                (CONNECT_DATA=
                    (SID=FNDFS)
    FNDSM_orclAPPL_ORCL=
            (DESCRIPTION=
                    (ADDRESS=(PROTOCOL=tcp)(HOST=orclAPPL.sa.company.net)(PORT=1626))
                (CONNECT_DATA=
                    (SID=FNDSM)
    FNDSM_orclAPPL.sa.company.net_ORCL=
            (DESCRIPTION=
                    (ADDRESS=(PROTOCOL=tcp)(HOST=orclAPPL.sa.company.net)(PORT=1626))
                (CONNECT_DATA=
                    (SID=FNDSM)
    FNDFS_APPLTOP_orclappl=
            (DESCRIPTION=
                (ADDRESS_LIST=
                    (ADDRESS=(PROTOCOL=tcp)(HOST=orclAPPL.sa.company.net)(PORT=1626))
                (CONNECT_DATA=
                    (SID=FNDFS)
    IFILE=E:\ORACLE\ORCL\INST\APPS\ORCL_orclappl\ora\10.1.2\network\admin\ORCL_orclappl_ifile.ora

    A database/OS crash shouldn't have such impact on your configuration. Please review the following docs and verify your setup.
    Using Oracle 11g Release 2 Real Application Clusters with Oracle E-Business Suite Release 12 (Doc ID 823587.1)
    Configuring and Managing E-Business Application Tier for RAC (Doc ID 1311528.1)
    Thanks,
    Hussein

  • Getting Stuck threads ,application is going down

    Hi All ,
    Need your expertise on the below issue . from the past 2 days we are getting stuck thread issues .  can you please provide insight  into the below issue  please ?
    ===== FULL THREAD DUMP ===============
      Wed Jan 8 09:28:23 2014
      Oracle JRockit(R) R28.2.3-13-149708-1.6.0_31-20120327-1523-linux-x86_64
      "Main Thread" id=1 idx=0x4 tid=24390 prio=5 alive, waiting, native_blocked
      -- Waiting for notification on: weblogic/t3/srvr/T3Srvr@0xf18da770[fat lock]
      at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
      at java/lang/Object.wait(J)V(Native Method)
      at java/lang/Object.wait(Object.java:485)
      at weblogic/t3/srvr/T3Srvr.waitForDeath(T3Srvr.java:981)
      ^-- Lock released while waiting: weblogic/t3/srvr/T3Srvr@0xf18da770[fat lock]
      at weblogic/t3/srvr/T3Srvr.run(T3Srvr.java:490)
      at weblogic/Server.main(Server.java:71)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "(Signal Handler)" id=2 idx=0x8 tid=24391 prio=5 alive, native_blocked, daemon
      "(OC Main Thread)" id=3 idx=0xc tid=24392 prio=5 alive, native_waiting, daemon
      "(GC Worker Thread 1)" id=? idx=0x10 tid=24393 prio=5 alive, daemon
      "(GC Worker Thread 2)" id=? idx=0x14 tid=24394 prio=5 alive, daemon
      "(GC Worker Thread 3)" id=? idx=0x18 tid=24395 prio=5 alive, daemon
      "(GC Worker Thread 4)" id=? idx=0x1c tid=24396 prio=5 alive, daemon
      "(GC Worker Thread 5)" id=? idx=0x20 tid=24397 prio=5 alive, daemon
      "(GC Worker Thread 6)" id=? idx=0x24 tid=24398 prio=5 alive, daemon
      "(GC Worker Thread 7)" id=? idx=0x28 tid=24399 prio=5 alive, daemon
      "(GC Worker Thread 8)" id=? idx=0x2c tid=24400 prio=5 alive, daemon
      "(GC Worker Thread 9)" id=? idx=0x30 tid=24401 prio=5 alive, daemon
      "(GC Worker Thread 10)" id=? idx=0x34 tid=24402 prio=5 alive, daemon
      "(GC Worker Thread 11)" id=? idx=0x38 tid=24403 prio=5 alive, daemon
      "(GC Worker Thread 12)" id=? idx=0x3c tid=24404 prio=5 alive, daemon
      "(GC Worker Thread 13)" id=? idx=0x40 tid=24405 prio=5 alive, daemon
      "(GC Worker Thread 14)" id=? idx=0x44 tid=24406 prio=5 alive, daemon
      "(GC Worker Thread 15)" id=? idx=0x48 tid=24407 prio=5 alive, daemon
      "(GC Worker Thread 16)" id=? idx=0x4c tid=24408 prio=5 alive, daemon
      "(GC Worker Thread 17)" id=? idx=0x50 tid=24409 prio=5 alive, daemon
      "(GC Worker Thread 18)" id=? idx=0x54 tid=24410 prio=5 alive, daemon
      "(Code Generation Thread 1)" id=4 idx=0x58 tid=24411 prio=5 alive, native_waiting, daemon
      "(Code Optimization Thread 1)" id=5 idx=0x5c tid=24412 prio=5 alive, native_waiting, daemon
      "(VM Periodic Task)" id=6 idx=0x60 tid=24413 prio=10 alive, native_blocked, daemon
      "Finalizer" id=7 idx=0x64 tid=24414 prio=8 alive, native_waiting, daemon
      at jrockit/memory/Finalizer.waitForFinalizees(J[Ljava/lang/Object;)I(Native Method)
      at jrockit/memory/Finalizer.access$700(Finalizer.java:12)
      at jrockit/memory/Finalizer$4.run(Finalizer.java:189)
      at java/lang/Thread.run(Thread.java:662)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "Reference Handler" id=8 idx=0x68 tid=24415 prio=10 alive, native_waiting, daemon
      at java/lang/ref/Reference.waitForActivatedQueue(J)Ljava/lang/ref/Reference;(Native Method)
      at java/lang/ref/Reference.access$100(Reference.java:11)
      at java/lang/ref/Reference$ReferenceHandler.run(Reference.java:82)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "(Sensor Event Thread)" id=9 idx=0x6c tid=24416 prio=5 alive, native_blocked, daemon
      "VM JFR Buffer Thread" id=10 idx=0x70 tid=24417 prio=5 alive, in native, daemon
      "Timer-0" id=13 idx=0x74 tid=24418 prio=5 alive, waiting, native_blocked, daemon
      -- Waiting for notification on: java/util/TaskQueue@0xf190d120[fat lock]
      at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
      at java/lang/Object.wait(J)V(Native Method)
      at java/lang/Object.wait(Object.java:485)
      at java/util/TimerThread.mainLoop(Timer.java:483)
      ^-- Lock released while waiting: java/util/TaskQueue@0xf190d120[fat lock]
      at java/util/TimerThread.run(Timer.java:462)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "Timer-1" id=14 idx=0x78 tid=24419 prio=5 alive, waiting, native_blocked, daemon
      -- Waiting for notification on: java/util/TaskQueue@0xf190d188[fat lock]
      at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
      at java/lang/Object.wait(J)V(Native Method)
      at java/util/TimerThread.mainLoop(Timer.java:509)
      ^-- Lock released while waiting: java/util/TaskQueue@0xf190d188[fat lock]
      at java/util/TimerThread.run(Timer.java:462)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "[STUCK] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" id=15 idx=0x7c tid=24420 prio=1 alive, blocked, native_blocked, daemon
      -- Blocked trying to get lock: oracle/jdbc/driver/T4CConnection@0xe4b45e08[thin lock]
      at jrockit/vm/Threads.sleep(I)V(Native Method)
      at jrockit/vm/Locks.waitForThinRelease(Locks.java:955)[optimized]
      at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1083)
      at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1005)
      at jrockit/vm/Locks.monitorEnter(Locks.java:2179)
      at oracle/jdbc/driver/PhysicalConnection.rollback(PhysicalConnection.java:3896)
      at weblogic/jdbc/wrapper/Connection.forcedCleanup(Connection.java:153)
      at weblogic/common/resourcepool/ResourcePoolImpl.timeoutInactiveResources(ResourcePoolImpl.java:1953)
      at weblogic/common/resourcepool/ResourcePoolImpl.access$1500(ResourcePoolImpl.java:41)
      at weblogic/common/resourcepool/ResourcePoolImpl$ResourcePoolMaintanenceTask.timerExpired(ResourcePoolImpl.java:2675)
      at weblogic/timers/internal/TimerImpl.run(TimerImpl.java:273)
      at weblogic/work/SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528)
      at weblogic/work/ExecuteThread.execute(ExecuteThread.java:209)
      at weblogic/work/ExecuteThread.run(ExecuteThread.java:178)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "JFR request timer" id=16 idx=0x80 tid=24421 prio=5 alive, waiting, native_blocked, daemon
      -- Waiting for notification on: java/util/TaskQueue@0xf190d208[fat lock]
      at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
      at java/lang/Object.wait(J)V(Native Method)
      at java/lang/Object.wait(Object.java:485)
      at java/util/TimerThread.mainLoop(Timer.java:483)
      ^-- Lock released while waiting: java/util/TaskQueue@0xf190d208[fat lock]
      at java/util/TimerThread.run(Timer.java:462)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "weblogic.time.TimeEventGenerator" id=18 idx=0x84 tid=24422 prio=9 alive, waiting, native_blocked, daemon
      -- Waiting for notification on: weblogic/time/common/internal/TimeTable@0xf190e690[fat lock]
      at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
      at java/lang/Object.wait(J)V(Native Method)
      at weblogic/time/common/internal/TimeTable.snooze(TimeTable.java:286)
      ^-- Lock released while waiting: weblogic/time/common/internal/TimeTable@0xf190e690[fat lock]
      at weblogic/time/common/internal/TimeEventGenerator.run(TimeEventGenerator.java:117)
      at java/lang/Thread.run(Thread.java:662)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "JMAPI event thread" id=19 idx=0x88 tid=24423 prio=5 alive, in native, daemon
      "weblogic.timers.TimerThread" id=20 idx=0x8c tid=24424 prio=9 alive, waiting, native_blocked, daemon
      -- Waiting for notification on: weblogic/timers/internal/TimerThread@0xf190d310[fat lock]
      at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
      at java/lang/Object.wait(J)V(Native Method)
      at weblogic/timers/internal/TimerThread$Thread.run(TimerThread.java:262)
      ^-- Lock released while waiting: weblogic/timers/internal/TimerThread@0xf190d310[fat lock]
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" id=21 idx=0x90 tid=24425 prio=5 alive, waiting, native_blocked, daemon
      -- Waiting for notification on: weblogic/work/ExecuteThread@0xf190dfe8[fat lock]
      at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
      at java/lang/Object.wait(J)V(Native Method)
      at java/lang/Object.wait(Object.java:485)
      at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:162)
      ^-- Lock released while waiting: weblogic/work/ExecuteThread@0xf190dfe8[fat lock]
      at weblogic/work/ExecuteThread.run(ExecuteThread.java:183)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" id=24 idx=0x94 tid=24426 prio=5 alive, blocked, native_blocked, daemon
      -- Blocked trying to get lock: java/lang/String@0xf18a6480[fat lock]
      at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)
      at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1411)[optimized]
      at jrockit/vm/Locks.lockFat(Locks.java:1512)[optimized]
      at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1054)[optimized]
      at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1005)[optimized]
      at jrockit/vm/Locks.monitorEnter(Locks.java:2179)[optimized]
      at weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:153)
      at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)
      at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)
      at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)
      at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" id=25 idx=0x98 tid=24427 prio=5 alive, in native, daemon
      at jrockit/ext/epoll/EPoll.epollWait0(ILjava/nio/ByteBuffer;II)I(Native Method)
      at jrockit/ext/epoll/EPoll.epollWait(EPoll.java:115)
      at weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:156)
      ^-- Holding lock: java/lang/String@0xf18a6480[fat lock]
      at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)
      at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)
      at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)
      at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" id=26 idx=0x9c tid=24428 prio=5 alive, blocked, native_blocked, daemon
      -- Blocked trying to get lock: java/lang/String@0xf18a6480[fat lock]
      at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)
      at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1411)[optimized]
      at jrockit/vm/Locks.lockFat(Locks.java:1512)[optimized]
      at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1054)[optimized]
      at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1005)[optimized]
      at jrockit/vm/Locks.monitorEnter(Locks.java:2179)[optimized]
      at weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:153)
      at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)
      at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)
      at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)
      at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "ExecuteThread: '3' for queue: 'weblogic.socket.Muxer'" id=27 idx=0xa0 tid=24429 prio=5 alive, blocked, native_blocked, daemon
      -- Blocked trying to get lock: java/lang/String@0xf18a6480[fat lock]
      at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)
      at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1411)[optimized]
      at jrockit/vm/Locks.lockFat(Locks.java:1512)[optimized]
      at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1054)[optimized]
      at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1005)[optimized]
      at jrockit/vm/Locks.monitorEnter(Locks.java:2179)[optimized]
      at weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:153)
      at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)
      at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)
      at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)
      at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "VDE Transaction Processor Thread" id=30 idx=0xa4 tid=24430 prio=2 alive, waiting, native_blocked, daemon
      -- Waiting for notification on: com/octetstring/vde/backend/standard/TransactionProcessor@0xf190e0d0[fat lock]
      at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
      at java/lang/Object.wait(J)V(Native Method)
      at java/lang/Object.wait(Object.java:485)
      at com/octetstring/vde/backend/standard/TransactionProcessor.waitChange(TransactionProcessor.java:367)
      ^-- Lock released while waiting: com/octetstring/vde/backend/standard/TransactionProcessor@0xf190e0d0[fat lock]
      at com/octetstring/vde/backend/standard/TransactionProcessor.run(TransactionProcessor.java:212)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "Query_Execution_78" id=248 idx=0xa8 tid=28978 prio=5 alive, parked, native_blocked, daemon
      -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0xe2f645d8
      at jrockit/vm/Locks.park0(J)V(Native Method)
      at jrockit/vm/Locks.park(Locks.java:2230)
      at sun/misc/Unsafe.park(ZJ)V(Native Method)
      at java/util/concurrent/locks/LockSupport.park(LockSupport.java:156)
      at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
      at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
      at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
      at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
      at java/lang/Thread.run(Thread.java:662)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "DoSManager" id=33 idx=0xac tid=24432 prio=6 alive, sleeping, native_waiting, daemon
      at java/lang/Thread.sleep(J)V(Native Method)
      at com/octetstring/vde/DoSManager.run(DoSManager.java:433)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "Thread-12" id=34 idx=0xb0 tid=24433 prio=5 alive, parked, native_blocked
      -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0xf48abb88
      at jrockit/vm/Locks.park0(J)V(Native Method)
      at jrockit/vm/Locks.park(Locks.java:2230)
      at sun/misc/Unsafe.park(ZJ)V(Native Method)
      at java/util/concurrent/locks/LockSupport.park(LockSupport.java:156)
      at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
      at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
      at weblogic/utils/concurrent/JDK15ConcurrentBlockingQueue.take(JDK15ConcurrentBlockingQueue.java:89)
      at weblogic/store/internal/PersistentStoreImpl.getOutstandingWork(PersistentStoreImpl.java:672)
      at weblogic/store/internal/PersistentStoreImpl.run(PersistentStoreImpl.java:721)
      at java/lang/Thread.run(Thread.java:662)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "Timer-2" id=35 idx=0xb4 tid=24434 prio=5 alive, waiting, native_blocked, daemon
      -- Waiting for notification on: java/util/TaskQueue@0xf5cf4fc0[fat lock]
      at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
      at java/lang/Object.wait(J)V(Native Method)
      at java/util/TimerThread.mainLoop(Timer.java:509)
      ^-- Lock released while waiting: java/util/TaskQueue@0xf5cf4fc0[fat lock]
      at java/util/TimerThread.run(Timer.java:462)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "Timer-3" id=37 idx=0xb8 tid=24435 prio=5 alive, waiting, native_blocked, daemon
      -- Waiting for notification on: java/util/TaskQueue@0xf5c14d00[fat lock]
      at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
      at java/lang/Object.wait(J)V(Native Method)
      at java/util/TimerThread.mainLoop(Timer.java:509)
      ^-- Lock released while waiting: java/util/TaskQueue@0xf5c14d00[fat lock]
      at java/util/TimerThread.run(Timer.java:462)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "DBQueueList" id=36 idx=0xbc tid=24436 prio=5 alive, waiting, native_blocked, daemon
      -- Waiting for notification on: java/lang/Object@0xf5c24d00[fat lock]
      at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
      at java/lang/Object.wait(J)V(Native Method)
      at com/demantra/applicationServer/metaDataObjects/dbQueue/DBQueueList.run(DBQueueList.java:256)
      ^-- Lock released while waiting: java/lang/Object@0xf5c24d00[fat lock]
      at java/lang/Thread.run(Thread.java:662)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "DemantraWorkflowLogCleaner" id=38 idx=0xc0 tid=24437 prio=5 alive, waiting, native_blocked, daemon
      -- Waiting for notification on: com/demantra/workflow/util/WorkflowServer$LogCleaner@0xe4abf950[fat lock]
      at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
      at java/lang/Object.wait(J)V(Native Method)
      at com/demantra/workflow/util/WorkflowServer$LogCleaner.run(WorkflowServer.java:1379)
      ^-- Lock released while waiting: com/demantra/workflow/util/WorkflowServer$LogCleaner@0xe4abf950[fat lock]
      at java/lang/Thread.run(Thread.java:662)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "WFProcess_0" id=39 idx=0xc4 tid=24438 prio=5 alive, in native, daemon
      at jrockit/net/SocketNativeIO.readBytesPinned(Ljava/io/FileDescriptor;[BIII)I(Native Method)
      at jrockit/net/SocketNativeIO.socketRead(SocketNativeIO.java:32)
      at java/net/SocketInputStream.socketRead0(Ljava/io/FileDescriptor;[BIII)I(SocketInputStream.java)
      at java/net/SocketInputStream.read(SocketInputStream.java:129)
      at oracle/net/ns/Packet.receive(Packet.java:293)
      at oracle/net/ns/DataPacket.receive(DataPacket.java:104)
      at oracle/net/ns/NetInputStream.getNextPacket(NetInputStream.java:315)[optimized]
      at oracle/net/ns/NetInputStream.read(NetInputStream.java:260)
      at oracle/net/ns/NetInputStream.read(NetInputStream.java:185)
      at oracle/net/ns/NetInputStream.read(NetInputStream.java:102)
      at oracle/jdbc/driver/T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124)
      at oracle/jdbc/driver/T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80)
      at oracle/jdbc/driver/T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1136)
      at oracle/jdbc/driver/T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1113)
      at oracle/jdbc/driver/T4CTTIfun.receive(T4CTTIfun.java:288)
      at oracle/jdbc/driver/T4CTTIfun.doRPC(T4CTTIfun.java:191)
      at oracle/jdbc/driver/T4C8Oall.doOALL(T4C8Oall.java:523)
      at oracle/jdbc/driver/T4CStatement.doOall8(T4CStatement.java:193)
      at oracle/jdbc/driver/T4CStatement.executeForRows(T4CStatement.java:998)
      at oracle/jdbc/driver/OracleStatement.doExecuteWithTimeout(OracleStatement.java:1315)
      at oracle/jdbc/driver/OracleStatement.executeInternal(OracleStatement.java:1890)
      at oracle/jdbc/driver/OracleStatement.execute(OracleStatement.java:1855)
      ^-- Holding lock: oracle/jdbc/driver/T4CConnection@0xe4b45e08[thin lock]
      at oracle/jdbc/driver/OracleStatementWrapper.execute(OracleStatementWrapper.java:304)
      at weblogic/jdbc/wrapper/Statement.execute(Statement.java:466)
      at org/apache/commons/dbcp/DelegatingStatement.execute(DelegatingStatement.java:264)
      at com/demantra/applicationServer/services/DBServicesCommon.executeSelect(DBServicesCommon.java:1464)
      at com/demantra/applicationServer/services/DBServices.executeSelect(DBServices.java:663)
      at com/demantra/applicationServer/services/DBServicesCommon.createMaterializedViewAs(DBServicesCommon.java:1143)
      at com/demantra/applicationServer/services/DBServices.createMaterializedViewAs(DBServices.java:429)
      at com/demantra/applicationServer/services/queryrun/ExportDataHandler.exportData(ExportDataHandler.java:53)
      at com/demantra/applicationServer/services/queryrun/ExportRunner.exportData(ExportRunner.java:62)
      at com/demantra/applicationServer/metaDataObjects/transfer/profiles/QueryProfile.exportProfile(QueryProfile.java:1002)
      ^-- Holding lock: com/demantra/applicationServer/metaDataObjects/transfer/profiles/QueryProfile@0xe4aa5cd8[biased lock]
      at com/demantra/applicationServer/metaDataObjects/transfer/profiles/IntegrationProfile.exportData(IntegrationProfile.java:331)
      at com/demantra/applicationServer/metaDataObjects/transfer/Transfer.exportData(Transfer.java:370)
      at com/demantra/workflow/step/TransferStep.doExecute(TransferStep.java:266)
      at com/demantra/workflow/step/LinkedStep.execute(LinkedStep.java:914)
      at com/demantra/workflow/process/WFProcess.mainLoop(WFProcess.java:1259)
      at com/demantra/workflow/process/WFProcess.run(WFProcess.java:654)
      at java/lang/Thread.run(Thread.java:662)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "WFProcess_1" id=40 idx=0xc8 tid=24439 prio=5 alive, sleeping, native_waiting, daemon
      at java/lang/Thread.sleep(J)V(Native Method)
      at com/demantra/workflow/process/WFProcess.checkStepEndConditions(WFProcess.java:1405)
      at com/demantra/workflow/process/WFProcess.mainLoop(WFProcess.java:1284)
      at com/demantra/workflow/process/WFProcess.run(WFProcess.java:654)
      at java/lang/Thread.run(Thread.java:662)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "WFProcess_2" id=41 idx=0xcc tid=24440 prio=5 alive, sleeping, native_waiting, daemon
      at java/lang/Thread.sleep(J)V(Native Method)
      at com/demantra/workflow/process/WFProcess.checkStepEndConditions(WFProcess.java:1405)
      at com/demantra/workflow/process/WFProcess.mainLoop(WFProcess.java:1284)
      at com/demantra/workflow/process/WFProcess.run(WFProcess.java:654)
      at java/lang/Thread.run(Thread.java:662)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "WFProcess_3" id=42 idx=0xd0 tid=24441 prio=5 alive, sleeping, native_waiting, daemon
      at java/lang/Thread.sleep(J)V(Native Method)
      at com/demantra/workflow/process/WFProcess.checkStepEndConditions(WFProcess.java:1405)
      at com/demantra/workflow/process/WFProcess.mainLoop(WFProcess.java:1284)
      at com/demantra/workflow/process/WFProcess.run(WFProcess.java:654)
      at java/lang/Thread.run(Thread.java:662)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "WFProcess_4" id=43 idx=0xd4 tid=24442 prio=5 alive, sleeping, native_waiting, daemon
      at java/lang/Thread.sleep(J)V(Native Method)
      at com/demantra/workflow/process/WFProcess.checkStepEndConditions(WFProcess.java:1405)
      at com/demantra/workflow/process/WFProcess.mainLoop(WFProcess.java:1284)
      at com/demantra/workflow/process/WFProcess.run(WFProcess.java:654)
      at java/lang/Thread.run(Thread.java:662)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "WFProcess_5" id=44 idx=0xd8 tid=24443 prio=5 alive, sleeping, native_waiting, daemon
      at java/lang/Thread.sleep(J)V(Native Method)
      at com/demantra/workflow/process/WFProcess.checkStepEndConditions(WFProcess.java:1405)
      at com/demantra/workflow/process/WFProcess.mainLoop(WFProcess.java:1284)
      at com/demantra/workflow/process/WFProcess.run(WFProcess.java:654)
      at java/lang/Thread.run(Thread.java:662)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "WFProcess_6" id=45 idx=0xdc tid=24444 prio=5 alive, sleeping, native_waiting, daemon
      at java/lang/Thread.sleep(J)V(Native Method)
      at com/demantra/workflow/process/WFProcess.checkStepEndConditions(WFProcess.java:1405)
      at com/demantra/workflow/process/WFProcess.mainLoop(WFProcess.java:1284)
      at com/demantra/workflow/process/WFProcess.run(WFProcess.java:654)
      at java/lang/Thread.run(Thread.java:662)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "WFProcess_7" id=46 idx=0xe0 tid=24445 prio=5 alive, sleeping, native_waiting, daemon
      at java/lang/Thread.sleep(J)V(Native Method)
      at com/demantra/workflow/process/WFProcess.checkStepEndConditions(WFProcess.java:1405)
      at com/demantra/workflow/process/WFProcess.mainLoop(WFProcess.java:1284)
      at com/demantra/workflow/process/WFProcess.run(WFProcess.java:654)
      at java/lang/Thread.run(Thread.java:662)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "Query_Execution_86" id=256 idx=0xe4 tid=28986 prio=5 alive, parked, native_blocked, daemon
      -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0xe2f645d8
      at jrockit/vm/Locks.park0(J)V(Native Method)
      at jrockit/vm/Locks.park(Locks.java:2230)
      at sun/misc/Unsafe.park(ZJ)V(Native Method)
      at java/util/concurrent/locks/LockSupport.park(LockSupport.java:156)
      at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
      at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
      at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
      at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
      at java/lang/Thread.run(Thread.java:662)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "Query_Execution_80" id=250 idx=0xe8 tid=28980 prio=5 alive, parked, native_blocked, daemon
      -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0xe2f645d8
      at jrockit/vm/Locks.park0(J)V(Native Method)
      at jrockit/vm/Locks.park(Locks.java:2230)
      at sun/misc/Unsafe.park(ZJ)V(Native Method)
      at java/util/concurrent/locks/LockSupport.park(LockSupport.java:156)
      at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
      at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
      at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
      at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
      at java/lang/Thread.run(Thread.java:662)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "Query_Execution_85" id=255 idx=0xec tid=28985 prio=5 alive, parked, native_blocked, daemon
      -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0xe2f645d8
      at jrockit/vm/Locks.park0(J)V(Native Method)
      at jrockit/vm/Locks.park(Locks.java:2230)
      at sun/misc/Unsafe.park(ZJ)V(Native Method)
      at java/util/concurrent/locks/LockSupport.park(LockSupport.java:156)
      at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
      at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
      at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
      at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
      at java/lang/Thread.run(Thread.java:662)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "Query_Execution_84" id=254 idx=0xf0 tid=28984 prio=5 alive, parked, native_blocked, daemon
      -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0xe2f645d8
      at jrockit/vm/Locks.park0(J)V(Native Method)
      at jrockit/vm/Locks.park(Locks.java:2230)
      at sun/misc/Unsafe.park(ZJ)V(Native Method)
      at java/util/concurrent/locks/LockSupport.park(LockSupport.java:156)
      at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
      at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
      at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
      at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
      at java/lang/Thread.run(Thread.java:662)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "Query_Execution_87" id=257 idx=0xf4 tid=28987 prio=5 alive, parked, native_blocked, daemon
      -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0xe2f645d8
      at jrockit/vm/Locks.park0(J)V(Native Method)
      at jrockit/vm/Locks.park(Locks.java:2230)
      at sun/misc/Unsafe.park(ZJ)V(Native Method)
      at java/util/concurrent/locks/LockSupport.park(LockSupport.java:156)
      at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
      at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
      at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
      at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
      at java/lang/Thread.run(Thread.java:662)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "Query_Execution_81" id=251 idx=0xf8 tid=28981 prio=5 alive, parked, native_blocked, daemon
      -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0xe2f645d8
      at jrockit/vm/Locks.park0(J)V(Native Method)
      at jrockit/vm/Locks.park(Locks.java:2230)
      at sun/misc/Unsafe.park(ZJ)V(Native Method)
      at java/util/concurrent/locks/LockSupport.park(LockSupport.java:156)
      at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
      at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
      at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
      at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
      at java/lang/Thread.run(Thread.java:662)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "Query_Execution_83" id=253 idx=0xfc tid=28983 prio=5 alive, parked, native_blocked, daemon
      -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0xe2f645d8
      at jrockit/vm/Locks.park0(J)V(Native Method)
      at jrockit/vm/Locks.park(Locks.java:2230)
      at sun/misc/Unsafe.park(ZJ)V(Native Method)
      at java/util/concurrent/locks/LockSupport.park(LockSupport.java:156)
      at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
      at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
      at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
      at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
      at java/lang/Thread.run(Thread.java:662)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "WFProcess_8" id=54 idx=0x100 tid=24453 prio=5 alive, sleeping, native_waiting, daemon
      at java/lang/Thread.sleep(J)V(Native Method)
      at com/demantra/workflow/process/WFProcess.checkStepEndConditions(WFProcess.java:1405)
      at com/demantra/workflow/process/WFProcess.mainLoop(WFProcess.java:1284)
      at com/demantra/workflow/process/WFProcess.run(WFProcess.java:654)
      at java/lang/Thread.run(Thread.java:662)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "Query_Execution_82" id=252 idx=0x104 tid=28982 prio=5 alive, parked, native_blocked, daemon
      -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0xe2f645d8
      at jrockit/vm/Locks.park0(J)V(Native Method)
      at jrockit/vm/Locks.park(Locks.java:2230)
      at sun/misc/Unsafe.park(ZJ)V(Native Method)
      at java/util/concurrent/locks/LockSupport.park(LockSupport.java:156)
      at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
      at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
      at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
      at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
      at java/lang/Thread.run(Thread.java:662)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "Query_Execution_79" id=249 idx=0x108 tid=28979 prio=5 alive, parked, native_blocked, daemon
      -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0xe2f645d8
      at jrockit/vm/Locks.park0(J)V(Native Method)
      at jrockit/vm/Locks.park(Locks.java:2230)
      at sun/misc/Unsafe.park(ZJ)V(Native Method)
      at java/util/concurrent/locks/LockSupport.park(LockSupport.java:156)
      at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
      at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
      at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
      at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
      at java/lang/Thread.run(Thread.java:662)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "Query_Execution_92" id=262 idx=0x10c tid=28992 prio=5 alive, parked, native_blocked, daemon
      -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0xe2f645d8
      at jrockit/vm/Locks.park0(J)V(Native Method)
      at jrockit/vm/Locks.park(Locks.java:2230)
      at sun/misc/Unsafe.park(ZJ)V(Native Method)
      at java/util/concurrent/locks/LockSupport.park(LockSupport.java:156)
      at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
      at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
      at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
      at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
      at java/lang/Thread.run(Thread.java:662)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "Thread-17" id=58 idx=0x110 tid=24457 prio=5 alive, parked, native_blocked
      -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0xe4a33f00
      at jrockit/vm/Locks.park0(J)V(Native Method)
      at jrockit/vm/Locks.park(Locks.java:2230)
      at sun/misc/Unsafe.park(ZJ)V(Native Method)
      at java/util/concurrent/locks/LockSupport.park(LockSupport.java:156)
      at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
      at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
      at weblogic/utils/concurrent/JDK15ConcurrentBlockingQueue.take(JDK15ConcurrentBlockingQueue.java:89)
      at weblogic/store/internal/PersistentStoreImpl.getOutstandingWork(PersistentStoreImpl.java:672)
      at weblogic/store/internal/PersistentStoreImpl.run(PersistentStoreImpl.java:721)
      at java/lang/Thread.run(Thread.java:662)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "DynamicListenThread[Default]" id=60 idx=0x114 tid=24458 prio=9 alive, in native, daemon
      at java/net/PlainSocketImpl.socketAccept(Ljava/net/SocketImpl;)V(Native Method)
      at java/net/PlainSocketImpl.accept(PlainSocketImpl.java:408)
      ^-- Holding lock: java/net/SocksSocketImpl@0xe4aec048[biased lock]
      at java/net/ServerSocket.implAccept(ServerSocket.java:462)
      at java/net/ServerSocket.accept(ServerSocket.java:430)
      at weblogic/socket/WeblogicServerSocket.accept(WeblogicServerSocket.java:38)
      at weblogic/server/channels/DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:535)
      at weblogic/server/channels/DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:417)
      at weblogic/server/channels/DynamicListenThread.run(DynamicListenThread.java:173)
      at java/lang/Thread.run(Thread.java:662)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "[STANDBY] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'" id=61 idx=0x118 tid=24459 prio=5 alive, waiting, native_blocked, daemon
      -- Waiting for notification on: weblogic/work/ExecuteThread@0xe4a52f68[fat lock]
      at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
      at java/lang/Object.wait(J)V(Native Method)
      at java/lang/Object.wait(Object.java:485)
      at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:162)
      ^-- Lock released while waiting: weblogic/work/ExecuteThread@0xe4a52f68[fat lock]
      at weblogic/work/ExecuteThread.run(ExecuteThread.java:183)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" id=62 idx=0x11c tid=24460 prio=5 alive, waiting, native_blocked, daemon
      -- Waiting for notification on: weblogic/work/ExecuteThread@0xe4afa538[fat lock]
      at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
      at java/lang/Object.wait(J)V(Native Method)
      at java/lang/Object.wait(Object.java:485)
      at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:162)
      ^-- Lock released while waiting: weblogic/work/ExecuteThread@0xe4afa538[fat lock]
      at weblogic/work/ExecuteThread.run(ExecuteThread.java:183)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "[ACTIVE] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'" id=63 idx=0x120 tid=24461 prio=5 alive, waiting, native_blocked, daemon
      -- Waiting for notification on: weblogic/work/ExecuteThread@0xe4a626a8[fat lock]
      at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
      at java/lang/Object.wait(J)V(Native Method)
      at java/lang/Object.wait(Object.java:485)
      at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:162)
      ^-- Lock released while waiting: weblogic/work/ExecuteThread@0xe4a626a8[fat lock]
      at weblogic/work/ExecuteThread.run(ExecuteThread.java:183)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "Query_Execution_90" id=260 idx=0x124 tid=28990 prio=5 alive, parked, native_blocked, daemon
      -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0xe2f645d8
      at jrockit/vm/Locks.park0(J)V(Native Method)
      at jrockit/vm/Locks.park(Locks.java:2230)
      at sun/misc/Unsafe.park(ZJ)V(Native Method)
      at java/util/concurrent/locks/LockSupport.park(LockSupport.java:156)
      at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
      at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
      at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
      at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
      at java/lang/Thread.run(Thread.java:662)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "Query_Execution_88" id=258 idx=0x128 tid=28988 prio=5 alive, parked, native_blocked, daemon
      -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0xe2f645d8
      at jrockit/vm/Locks.park0(J)V(Native Method)
      at jrockit/vm/Locks.park(Locks.java:2230)
      at sun/misc/Unsafe.park(ZJ)V(Native Method)
      at java/util/concurrent/locks/LockSupport.park(LockSupport.java:156)
      at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
      at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
      at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
      at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
      at java/lang/Thread.run(Thread.java:662)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "Query_Execution_89" id=259 idx=0x12c tid=28989 prio=5 alive, parked, native_blocked, daemon
      -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0xe2f645d8
      at jrockit/vm/Locks.park0(J)V(Native Method)
      at jrockit/vm/Locks.park(Locks.java:2230)
      at sun/misc/Unsafe.park(ZJ)V(Native Method)
      at java/util/concurrent/locks/LockSupport.park(LockSupport.java:156)
      at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
      at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
      at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
      at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
      at java/lang/Thread.run(Thread.java:662)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "Query_Execution_91" id=261 idx=0x130 tid=28991 prio=5 alive, parked, native_blocked, daemon
      -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0xe2f645d8
      at jrockit/vm/Locks.park0(J)V(Native Method)
      at jrockit/vm/Locks.park(Locks.java:2230)
      at sun/misc/Unsafe.park(ZJ)V(Native Method)
      at java/util/concurrent/locks/LockSupport.park(LockSupport.java:156)
      at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
      at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
      at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
      at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
      at java/lang/Thread.run(Thread.java:662)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "Query_Execution_260" id=585 idx=0x134 tid=29439 prio=5 alive, parked, native_blocked, daemon
      -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0xe2f645d8
      at jrockit/vm/Locks.park0(J)V(Native Method)
      at jrockit/vm/Locks.park(Locks.java:2230)
      at sun/misc/Unsafe.park(ZJ)V(Native Method)
      at java/util/concurrent/locks/LockSupport.park(LockSupport.java:156)
      at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
      at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
      at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
      at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
      at java/lang/Thread.run(Thread.java:662)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "Timer-6" id=69 idx=0x138 tid=26756 prio=5 alive, waiting, native_blocked, daemon
      -- Waiting for notification on: java/util/TaskQueue@0xe17170f0[fat lock]
      at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
      at java/lang/Object.wait(J)V(Native Method)
      at java/util/TimerThread.mainLoop(Timer.java:509)
      ^-- Lock released while waiting: java/util/TaskQueue@0xe17170f0[fat lock]
      at java/util/TimerThread.run(Timer.java:462)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "Event_Manager_67" id=923 idx=0x13c tid=29870 prio=5 alive, parked, native_blocked, daemon
      -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0xf4b91b40
      at jrockit/vm/Locks.park0(J)V(Native Method)
      at jrockit/vm/Locks.park(Locks.java:2230)
      at sun/misc/Unsafe.park(ZJ)V(Native Method)
      at java/util/concurrent/locks/LockSupport.park(LockSupport.java:156)
      at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
      at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
      at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
      at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
      at java/lang/Thread.run(Thread.java:662)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "Task_Loader_Thread" id=71 idx=0x140 tid=27086 prio=5 alive, sleeping, native_waiting, daemon
      at java/lang/Thread.sleep(J)V(Native Method)
      at com/demantra/common/task/TaskLoader.run(TaskLoader.java:133)
      at java/lang/Thread.run(Thread.java:662)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "Query_Execution_268" id=608 idx=0x144 tid=29473 prio=5 alive, parked, native_blocked, daemon
      -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0xe2f645d8
      at jrockit/vm/Locks.park0(J)V(Native Method)
      at jrockit/vm/Locks.park(Locks.java:2230)
      at sun/misc/Unsafe.park(ZJ)V(Native Method)
      at java/util/concurrent/locks/LockSupport.park(LockSupport.java:156)
      at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
      at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
      at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
      at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
      at java/lang/Thread.run(Thread.java:662)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'" id=73 idx=0x148 tid=28139 prio=5 alive, waiting, native_blocked, daemon
      -- Waiting for notification on: weblogic/work/ExecuteThread@0xe2ffb420[fat lock]
      at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
      at java/lang/Object.wait(J)V(Native Method)
      at java/lang/Object.wait(Object.java:485)
      at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:162)
      ^-- Lock released while waiting: weblogic/work/ExecuteThread@0xe2ffb420[fat lock]
      at weblogic/work/ExecuteThread.run(ExecuteThread.java:183)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "[ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'" id=74 idx=0x14c tid=28140 prio=5 alive, waiting, native_blocked, daemon
      -- Waiting for notification on: weblogic/work/ExecuteThread@0xe2feb970[fat lock]
      at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
      at java/lang/Object.wait(J)V(Native Method)
      at java/lang/Object.wait(Object.java:485)
      at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:162)
      ^-- Lock released while waiting: weblogic/work/ExecuteThread@0xe2feb970[fat lock]
      at weblogic/work/ExecuteThread.run(ExecuteThread.java:183)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "[ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'" id=75 idx=0x150 tid=28141 prio=5 alive, waiting, native_blocked, daemon
      -- Waiting for notification on: weblogic/work/ExecuteThread@0xe300ab10[fat lock]
      at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
      at java/lang/Object.wait(J)V(Native Method)
      at java/lang/Object.wait(Object.java:485)
      at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:162)
      ^-- Lock released while waiting: weblogic/work/ExecuteThread@0xe300ab10[fat lock]
      at weblogic/work/ExecuteThread.run(ExecuteThread.java:183)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "[ACTIVE] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'" id=76 idx=0x154 tid=28142 prio=5 alive, waiting, native_blocked, daemon
      -- Waiting for notification on: weblogic/work/ExecuteThread@0xe2f55dc0[fat lock]
      at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
      at java/lang/Object.wait(J)V(Native Method)
      at java/lang/Object.wait(Object.java:485)
      at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:162)
      ^-- Lock released while waiting: weblogic/work/ExecuteThread@0xe2f55dc0[fat lock]
      at weblogic/work/ExecuteThread.run(ExecuteThread.java:183)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "[ACTIVE] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)'" id=77 idx=0x158 tid=28143 prio=5 alive, waiting, native_blocked, daemon
      -- Waiting for notification on: weblogic/work/ExecuteThread@0xe2f63970[fat lock]
      at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
      at java/lang/Object.wait(J)V(Native Method)
      at java/lang/Object.wait(Object.java:485)
      at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:162)
      ^-- Lock released while waiting: weblogic/work/ExecuteThread@0xe2f63970[fat lock]
      at weblogic/work/ExecuteThread.run(ExecuteThread.java:183)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "[ACTIVE] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'" id=78 idx=0x15c tid=28144 prio=5 alive, waiting, native_blocked, daemon
      -- Waiting for notification on: weblogic/work/ExecuteThread@0xe2f73640[fat lock]
      at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
      at java/lang/Object.wait(J)V(Native Method)
      at java/lang/Object.wait(Object.java:485)
      at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:162)
      ^-- Lock released while waiting: weblogic/work/ExecuteThread@0xe2f73640[fat lock]
      at weblogic/work/ExecuteThread.run(ExecuteThread.java:183)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "[ACTIVE] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)'" id=79 idx=0x160 tid=28145 prio=5 alive, waiting, native_blocked, daemon
      -- Waiting for notification on: weblogic/work/ExecuteThread@0xe2fec498[fat lock]
      at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
      at java/lang/Object.wait(J)V(Native Method)
      at java/lang/Object.wait(Object.java:485)
      at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:162)
      ^-- Lock released while waiting: weblogic/work/ExecuteThread@0xe2fec498[fat lock]
      at weblogic/work/ExecuteThread.run(ExecuteThread.java:183)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "[ACTIVE] ExecuteThread: '12' for queue: 'weblogic.kernel.Default (self-tuning)'" id=80 idx=0x164 tid=28146 prio=5 alive, waiting, native_blocked, daemon
      -- Waiting for notification on: weblogic/work/ExecuteThread@0xe2f82d20[fat lock]
      at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
      at java/lang/Object.wait(J)V(Native Method)
      at java/lang/Object.wait(Object.java:485)
      at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:162)
      ^-- Lock released while waiting: weblogic/work/ExecuteThread@0xe2f82d20[fat lock]
      at weblogic/work/ExecuteThread.run(ExecuteThread.java:183)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
      "[ACTIVE] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)'" id=81 idx=0x168 tid=28147 prio=5 alive, waiting, native_blocked, daemon
      -- Waiting for notification on: weblogic/work/ExecuteThread@0xe2fa0458[fat lock]
      at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
      at java/lang/Object.wait(J)V(Native Method)
      at java/lang/Object.wait(Object.java:485)
      at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:162)
      ^-- Lock released while waiting: weblogic/work/ExecuteThread@0xe2fa0458[fat lock]

    Check -Xmx and -Xms start parameters for java in opmn.xml for the container you are workingk, as well in the Console you can change this in the Properties for the container.
    Greetings

  • One RAC node is down give the following error when starting the database!

    wHEN TRYING TO START THE DATABASE ON RAC ENVIORNMENT
    SQL> connect sys as sysdba
    Enter password:
    Connected to an idle instance.
    SQL> startup
    ORA-27102: out of memory
    HPUX-ia64 Error: 12: Not enough space
    SQL> exit
    Disconnected
    When we r trying to start the database it said
    $ bdf
    Filesystem kbytes used avail %used Mounted on
    /dev/vg00/lvol3 2097152 240944 1841896 12% /
    /dev/vg00/lvol1 344064 115376 226944 34% /stand
    /dev/vg00/lvol8 10485760 9370960 1106232 89% /var
    /dev/vg00/lvol7 4866048 2557680 2290400 53% /usr
    /dev/vg00/u02 10485760 3502229 6547116 35% /u02
    /dev/vg00/u01 10485760 10476596 9164 100% /u01
    /dev/vg00/lvol4 2097152 601872 1483944 29% /tmp
    /dev/vg00/lvol6 4194304 3231000 955792 77% /opt
    /dev/vg00/lvol5 524288 311520 211136 60% /home
    WHERE /U01 WAS 100%. Now i emptied the space in /u01 to
    $ bdf
    Filesystem kbytes used avail %used Mounted on
    /dev/vg00/lvol3 2097152 240944 1841896 12% /
    /dev/vg00/lvol1 344064 115376 226944 34% /stand
    /dev/vg00/lvol8 10485760 9370960 1106232 89% /var
    /dev/vg00/lvol7 4866048 2557680 2290400 53% /usr
    /dev/vg00/u02 10485760 3502229 6547116 35% /u02
    /dev/vg00/u01 10485760 9508934 930943 91% /u01
    /dev/vg00/lvol4 2097152 601872 1483944 29% /tmp
    /dev/vg00/lvol6 4194304 3231000 955792 77% /opt
    /dev/vg00/lvol5 524288 311520 211136 60% /home
    When trying to start the db again its giving the following error...
    SQL> connect sys as sysdba
    Enter password:
    Connected to an idle instance.
    SQL> startup
    ORA-27102: out of memory
    HPUX-ia64 Error: 12: Not enough space
    SQL> exit
    Disconnected
    here i changed the sga_target and now it says
    ORACLE instance started.
    Total System Global Area 436207616 bytes
    Fixed Size 1297912 bytes
    Variable Size 148648456 bytes
    Database Buffers 285212672 bytes
    Redo Buffers 1048576 bytes
    ORA-01105: mount is incompatible with mounts by other instances
    ORA-19808: recovery destination parameter mismatch
    What could be the issue..
    Ur help would be highly appreciated...

    Hello
    SQL> startup
    ORA-27102: out of memory
    HPUX-ia64 Error: 12: Not enough space
    SQL> exiterror is not related to space on your mount point. it is related to memory.
    if you are getting this error means chekc at the OS level whether something is consuming more memory due to which it is not allowing oracle to allocate sga.
    Check top/sar/glance to see who is consuming more memory
    Total System Global Area 436207616 bytes
    Fixed Size 1297912 bytes
    Variable Size 148648456 bytes
    Database Buffers 285212672 bytes
    Redo Buffers 1048576 bytes
    ORA-01105: mount is incompatible with mounts by other instances
    ORA-19808: recovery destination parameter mismatchit is not the best practice to maintain differnet parameters for each instance in RAC env. also check db_recovery_file_dest and db_recovery_file_dest_size is same on all node. it should be same i.e it should be a shared location.
    Anil Malkai

  • JMS Uniform Distribute Queue Unit Of Order, problem when one node goes down

    Hi ,
    I have the following code which post a message (with Unit of Order set ) to a Uniform Distribute Queue in a cluster with two member servers (server1 and server2).
    --UDQ is targeted to a subdeployment that is mapped to two JMS servers pointing to each member servers
    --Connection Factory is using default targeting ( i tried mapping to Sub deployment also)
    javax.naming.InitialContext serverContext = new javax.naming.InitialContext();
    javax.jms.QueueConnectionFactory qConnFactory = (javax.jms.QueueConnectionFactory)serverContext.lookup(jmsQConnFactoryName);
    javax.jms.QueueConnection qConn = (javax.jms.QueueConnection)qConnFactory.createConnection();
    javax.jms.QueueSession qSession = qConn.createQueueSession(false, Session.AUTO_ACKNOWLEDGE);
    javax.jms.Queue q = ( javax.jms.Queue)serverContext.lookup(jmsQName);
    weblogic.jms.extensions.WLMessageProducer qSender = (weblogic.jms.extensions.WLMessageProducer) qSession.createProducer(q);
    qSender.setUnitOfOrder("MyUnitOfOrder");
    javax.jms.ObjectMessage message = qSession.createObjectMessage();
    HashMap<String, Object> map = new HashMap<String, Object>();
    map.put("something", "SomeObject");
    message.setObject(map);
    qSender.send(message);
    } catch (Exception e) {           
    Steps followed:
    1. Post a message from "server1"
    2. Message picked up by "server2"
    3. Everything fine
    4. Shutdown "server2"
    5. Post a message from "server1"
    6. ERROR: "hashed member of MyAppJMSModule!MyDistributedQ is MyAppJMSModule!MyJMSServer-2@MyDistributedQ which is not available"
    WebLogic version : 10.3.5
    Is there a way (other than configuring Path Service ) to make this code work "with unit of order" for a UDQ even if some member servers go down ?
    Thanks very much for your time.

    If you want to avoid use of the Path Service, then the alternative is to make the destination members highly available. This will help ensure that the host member for a particular UOO is up.
    One approach to HA is to configure "service migration". For more information see the Automatic Service Migration white-paper at
    http://www.oracle.com/technology/products/weblogic/pdf/weblogic-automatic-service-migration-whitepaper.pdf
    In addition, I recommend referencing Best Practices for JMS Beginners and Advanced Users
    http://docs.oracle.com/cd/E17904_01/web.1111/e13738/best_practice.htm#JMSAD455 to help with WL configuration in general.
    Hope this helps,
    Tom

  • JDBC URL for Two Oracle Nodes with different SID

    Hi All,
    I have two DB nodes which are running independently (Mean no RAC), I want to configure JDBC URL for these nodes that if one node goes down my application automatically connected to second one.
    Both nodes have different HOST and SID but the tables i am going to access have same structure. we are using this DB for some logging purpose that's why we didn't choose RAC option.
    Regards,
    imran

    For RAC database, I used:
    jdbc:oracle:thin:@(DESCRIPTION=(LOAD_BALANCE=on)(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=hostname)(PORT=port))(ADDRESS=(PROTOCOL=TCP)(HOST=hostname)(PORT=port)))(CONNECT_DATA=(SERVICE_NAME=servicename)))
    For non RAC database you could ommit LOAD_BALANCE setting, and set correct hostname & port for both nodes. We connected using common service name. If your databases have different service names, then the SID will have to be part of ADDRESS somehow - maybe (SID=sid).
    I have to mention though that unless the main reason behind your decision was lower cost without RAC (and you don't need to store any information apart from logs), it was a wrong decision. If you decide to store any additional information in the database for whatever reason (new requirements), you are in trouble.

  • Oracle Forms 10g runtime handling during RAC node failover.

    Hi,
    Forms version 10g R2 (10.1.2.0.2)
    Oracle DB version 10g R2 RAC with 3 nodes.
    If the RAC DB node that the user is connected to goes down, the user gets FRM-40733 and ORA-03114 error messages and the client forms application gets locked down/ goes in a loop with the error messages. The user has to close the browser to get out of the loop. I understand that this is the expected behaviour, but I'm wondering whether we can trap the error ORA-03114 and fire the "key-exit" trigger to get out of the application.
    Have any one implemented a clean way to exit the Forms application when the RAC DB node goes down..?
    I'm looking for some suggestions or an elegant way to handle the above failure.
    Thank you in advance.
    Sudhakar

    Glen,
    I haven't solve this one yet.I have been playing around with the following:
    In my environment, I am still using 6i (not web) forms/reports.
    My clients are XP, NT, 2000.
    I have the forms/report runtime installed on their PCs.
    Their TNSNAMES.ORA will be pointing to PRIMARY ( PDB)
    If a SWITCHOVER or FAILOVER happens to the physical standby (SDB), I want a trigger to kick a batch file that will manipulate the TNSNAME.ORA on each clients stations.
    On the standby
    CREATE OR REPLACE TRIGGER change_tns
    AFTER DB_ROLE_CHANGE ON DATABASE
    DECLARE
    role VARCHAR2(30);
    dbname varchar2(100);
    BEGIN
    SELECT
    DATABASE_ROLE,
    DB_UNIQUE_NAME
    INTO
    role,
    dbname
    FROM
    V$DATABASE;
    IF role = 'PRIMARY' and dbname='SDB' THEN
    dbms_scheduler.create_job(
    job_name=>'move_sqlnet',
    job_type=>'executable',
    job_action=>'c:\temp\movetns.cmd',
    enabled=TRUE
    ELSE
    -- if the standby >was< PRIMARY,
    -- but the primary comes BACK on line,
    -- need to reverse the step above.
    END IF;
    END;
    As for the movetns.cmd
    something like
    rem -- attach to the workstation,
    net use m: \\station name\share name
    rem -- stdb_tnsname.ora would be pointing to STANDBY
    copy stdb_tnsname.ora m:\orant\net80\tnsname.ora
    net use m: /delete
    rem -- need to do that for all workstations..
    As you can see, there could be lots of problems with this procedure.
    Client doesn't know about the failover, starts a RE-BOOT on the pc, therefore, the new tnsnames.ora will not get to client.. what to do for that client? Do i re-run the batch ...every hour?
    tell me if you come up with an answer..
    p-

  • JDBC URL for Two Oracle Nodes

    Hi All,
    I have two DB nodes which are running independently (Mean no RAC), I want to configure JDBC URL for these nodes that if one node goes down my application automatically connected to second one.
    Both nodes have different HOST and SID.
    Regards,
    imran

    Hi jwenting,
    I tried following URL in PL/SQL developer and my fail over is working fine with separate nodes, Only thing i did was i keep SID of both nodes with same name.
    ERS=(DESCRIPTION=(FAILOVER=on)(LOAD_BALANCE=off)(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=172.21.5.130)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=172.21.5.133)(PORT=1521)))(CONNECT_DATA=(SID=orcl)))
    But strange when i connected this through JDBC URL like following it gives me error of invalid URL
    jdbc:oracle:thin:@(DESCRIPTION=(FAILOVER=on)(LOAD_BALANCE=off)(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=172.21.5.130)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=172.21.5.133)(PORT=1521)))(CONNECT_DATA=(SID=orcl)))
    Regards,
    imran

  • RAC Node eviction question...

    Say we have 3 node RAC cluster on OEL5.3. What happens if one node evicted out of it? I know other two instance will do dynamic remastering... and something more.
    I want to know eachand every steps in detail. What really happens when one node goes down in RAC environment.
    Experts please comment.
    Many Thanks.

    I want to know each and every steps in detail. Assume you know "each and every steps in detail." what will you do differently based upon this information?
    Handle:      vh_dba
    Status Level:      Newbie (30)
    Registered:      Jan 10, 2010
    Total Posts:      38
    Total Questions:      16 (15 unresolved)
    So many questions with only a single answer.
    :-(

  • Listeners are going down frequently..

    Hi,
    I have two node cluster databases.in that cluster 5 databases are running. We have configuared 2 listeners for each database and registered in OCR.But frequently some listeners are going down or UNKNOWN state from first node only. everything is running fine from second node. How to troubleshoot this?
    Thanks in advance

    user2017273 wrote:
    I have two node cluster databases.in that cluster 5 databases are running. We have configuared 2 listeners for each database and registered in OCR.As I understand, you are running 5 RAC databases on the same cluster (meaning 5 RAC instances per node), with each instance having 2 listeners, resulting in 10 listeners per node?
    What on earth for? Why? What problem/issue are you trying to address that way?
    But frequently some listeners are going down or UNKNOWN state from first node only. everything is running fine from second node. How to troubleshoot this? RAC is only as robust as
    a) the architecture it is implemented on (server h/w, Interconnect, etc)
    b) how well it adheres to Oracle concepts and fundamentals
    5 RAC instances on the same 2 nodes, using 10 listeners per node? That is anything but adhering to Oracle fundamentals. More like total madness IMO.
    Cannot recall ever having a listener (stand-alone or on any of our RACs) simply going down and not working as designed. So I doubt strongly that your are dealing with an Oracle issue here. Seems to be a case of an unnecessary self inflicted wound.

  • Failover did not happen when one node went down!!! PLEASE HELP

    Hi gurus,
    Yesterday one disaster struck my RAC database. We have two node cluster and it is 10.2.0.2, both of them located in different sites, yesterday suddenly power went down and the one of the network switch went down and got destructed, node one of RAC database was connected to that switch, but the failover did not happen to the node two as this should be the case when one node goes down the other should be available for all the node one sessions/connections.
    when I tried to ping/telnet the node 1, it was not happening because the switch was down, the network guyz connected the cables to other switch available. When I connected to the node 1, it was showing "Oracle is not available" message.
    And when I tried the other node, it was the same case but I did not see any error in alert log file. Then my TL restarted both the nodes and then the database was available.
    I am very confused that how the failover did not happen and how the database went down, PLEASE suggest something to how to identifiy what was happened. Thanks & Regards

    Thanks for your reply,
    after the network switch was replaced we connected to both the nodes and found that the instances are down with no reason given in the Alertlog file. We just restarted both the instances and then the database was up and the clients connected to both the instances with equal sessions on both the instances. I want to know that whether the failover can be done at the application side or it should be done on the database side i,e; in tnsnames.ora file with the required parameters? as in our scenario there is no failover configuration in the tnsnames.ora file.
    Thanks & Regards

  • Sound volume goes down spontaneously

    For the last few months I have had a problem on both my macbook pro and my mac mini that they both seem to get stuck with the volume going down to zero and the volume change indicator the speaker with the little boxes below it going down to muted volume and when you try to increase the volume you get one bar of volume and the second you let go of the key it instantly goes back down. I have tried on both changing out the speakers for the mac mini and used headphones on the MBP but nothing really helps. I did a reinstall of OSX on the mac mini to see if this fixes anything. I have tried just about everything I can think of and I am trying to figure out how I can get it so that this doesn't happen.

    I have tried just about everything I can think of and I am trying to figure out how I can get it so that this doesn't happen.
    Try trashing the "com.apple.systemsound.plist" file then restart your computer.

Maybe you are looking for