Oracle occupy 100% CPU

Hi,
While running program from JDBC, Oracle process occupy 100% CPU. We have oracle 11gR2.we have 14GB RAM.Memory target is set to 4GB.can you please guide me how to troubleshoot.

user12116651 wrote:
While running program from JDBC, Oracle process occupy 100% CPU. We have oracle 11gR2.we have 14GB RAM.Memory target is set to 4GB.can you please guide me how to troubleshoot.Identify the physical o/s process consuming CPU via a command like top.
If this is an Oracle process, identify the session and process in the Oracle instance using the v$process and v$session views - the former contains the o/s process id and the latter is joined to get the Oracle session id and basic session details.
Next use the v$sess* views to analyse the wait states, events and statistics of that session. You can also determine the current SQL cursor being executed and use DBMS_XPLAN.display_cursor() to get its execution plan.
Details on the virtual performance v$ views are in the Oracle® Database Reference guide.

Similar Messages

  • Oracle uses 100% CPU

    Hi,
    I have installed Oracle 10g R2 on 64bit linux. It was working fine, until last week. I got alert that oracle uses 100% CPU.
    Some lines from alert:
    4894 /opt/oracle/product/10.2.0/db/perl/bin/perl /opt/oracle/product/10.2.0/db/sysman/admin/scripts/db/dbresp.pl 56.3 12978 7720 22860 oracle
    9244 /opt/oracle/product/10.2.0/db/perl/bin/perl /opt/oracle/product/10.2.0/db/sysman/admin/scripts/db/dbresp.pl 50.4 10357 7720 22860 oracle
    11798 /opt/oracle/product/10.2.0/db/perl/bin/perl /opt/oracle/product/10.2.0/db/sysman/admin/scripts/db/dbresp.pl 41.1 4429 7720 22860 oracle
    11740 /opt/oracle/product/10.2.0/db/perl/bin/perl /opt/oracle/product/10.2.0/db/sysman/admin/scripts/db/dbresp.pl 28.3 45350 7724 22864 oracle
    1516 /opt/oracle/product/10.2.0/db/perl/bin/perl /opt/oracle/product/10.2.0/db/sysman/admin/scripts/db/dbresp.pl 24.2 29968 7728 22868 oracle
    17006 /opt/oracle/product/10.2.0/db/perl/bin/perl /opt/oracle/product/10.2.0/db/sysman/admin/scripts/db/dbresp.pl 21.8 21913 7724 22864 oracle
    19449 /opt/oracle/product/10.2.0/db/perl/bin/perl /opt/oracle/product/10.2.0/db/sysman/admin/scripts/db/dbresp.pl 21.5 20865 7720 22860 oracle
    24877 /opt/oracle/product/10.2.0/db/perl/bin/perl /opt/oracle/product/10.2.0/db/sysman/admin/scripts/db/dbresp.pl 21.0 18708 7728 22868 oracle
    26566 /opt/oracle/product/10.2.0/db/perl/bin/perl /opt/oracle/product/10.2.0/db/sysman/admin/scripts/db/dbresp.pl 18.4 5727 7724 22864 oracle
    1788 /opt/oracle/product/10.2.0/db/perl/bin/perl /opt/oracle/product/10.2.0/db/sysman/admin/scripts/db/dbresp.pl 17.1 3319 7724 22864 oracle
    When I log in to EM, I can see section
    Host CPU
    It is 100%.
    I have only instance named "parama". and it uses 1% CPU ant the picture shows that 99% CPU is used By Other.
    What is Other? Why does it use that much CPU?
    I had another oracle instance few months ago, but I is not used now (it is not even started).
    How do I restore normal system performance?
    Can I kill these processes?

    Hello,
    I have a similar problem. I see 2 perl processes related to Oracle that are taking up 99% of 2 cpus. These 2 perl processes have been running for days and keeping the load average high (above 2) on a mostly idle system.
    The 2 perl processes are running:
    /oracle/app/product/10.2.0/db_1/perl/bin/perl /oracle/app/product/10.2.0/db_1/sysman/admin/scripts/db/dbarchfull.pl
    and
    /oracle/app/product/10.2.0/db_1/perl/bin/perl /oracle/app/product/10.2.0/db_1/sysman/admin/scripts/db/dbresp.pl
    Has anyone seen this problem? Can I kill the perl processes?
    --Dan                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       

  • "du process" pops up and occupy 100% CPU

    I didn't solve the issue. I've got a macbook pro 2010, partitioned with bootcamp and with os 10.6.8. "Du process" appears and disappears, and when pops up occupy 100% CPU. This process isn't present in my other macbooks with the same os. Is it possibile and useful (to get 100% cpu availability) to cancel it? How?
    Thank you in advance for your help!

    Safari third party plug-ins and extensions can increase CPU usage.
    From the Safari menu bar click Safari > Preferences then select the Extensions tab. Turn that OFF, quit and relaunch Safari to test.
    If that helped, turn one extension on then quit and relaunch Safari to test until you find the incompatible extension then click uninstall.
    If it's not an extensions issue, try troubleshooting third party plug-ins.
    Back to Safari > Preferences. This time select the Security tab. Deselect:  Allow plug-ins. Quit and relaunch Safari to test.
    If that made a difference, instructions for troubleshooting plugins here.

  • Oracle taking 100% CPU usage

    Hi,
    I have a situation. I am trying to execute a big query by involving 3 tables (joins). The query is taking about 20 minutes time to execute. While the query is executing the CPU usage for ORACLE is 100%. Please put somelight on this as of to why this is happening? Do I need to tweak my query a bit more. This is happening only recently as there are more number of records. ORACLE is installed on a Unix machine
    Thanks in advance.

    The First Rule Of Performance Troubleshooting: IDENTIFY THE PERFORMANCE PROBLEM.
    All you have done thus far is identifying a symptom. High CPU utilisation. This in itself is totally meaningless - nothing can be inferred from this symptom without additional troubleshooting.
    This means determining WHAT is happening when that query/process runs. What is it doing? (look at wait states and event state summaries of the process)
    WHY is it doing that? Look at the execution plan.
    With performance tuning, one needs to deal with facts. So you need to pop the hood and get your hands dirty so see what is actually happening under the hood.

  • T41---proc​ess rundll32.e​xe occupied 100% cpu usage and caused slow performanc​e.

    hi i have a big problem whit the performance of my laptop. I bought a extra 4gb ram for the second slot that not exist; and still continuos getting my cpu usage at 100% whit 2 programs running like office and google crhome.
    i have this specf of my latop:
    - procesor: amd c-60 apu  whit radeon hd graphics 1,00 ghz
    - ram: 4gz 
    system type: 64-bit  OS
    winows 7 ultimate
    please tell me if i have to install another OS, made changes in my configuration or just buy another laptop.
    tks

    Rundll32.exe Using High CPU on Windows 8.1
    I use a Lenovo T440s laptop for my service calls and when I need to do work remotely. My laptop runs Windows 8.1 Professional. I love this laptop and it has proved to be tough as well as extremely fast. I noticed a couple weeks ago that my battery was going dead after about an hour of usage. I also noticed that my cooling fan in the laptop was running louder than normal. I decided to open Task Manger on my laptop to see what the issue was. I soon noticed that a process called rundll32.exe was using around 25 percent of my CPU. I killed the process and everything returned to normal.
    A few days later, my battery was going dead very rapidly. My cooling fan was on almost constantly and was blowing warm air. I opened Task Manger and sure enough, rundll32.exe was using 25% of my CPU again. I killed the process and everything was fine. It wasn’t long and my laptop was doing the exact same thing as I described before. I decided it was time to get to the bottom of this issue. Here is how I troubleshot what program was using the rundll32.exe and in the process was killing my battery.
    Step One: I downloaded Process Explorer. This is an excellent tool. It is similar to Task Manager but goes into way more depth on the processes and dlls that are running. I extracted the download and then ran Process Explorer as an administrator.
    Step Two: One of the first things I noticed was rundll32.exe was using 23 percent of my CPU again. 
    I right-clicked on the rundll32.exe process and selected properties. I clicked on the Image tab and at once saw the culprit. A utility to monitor and gauge my battery usage, called PwrMgrBkGndMonitor, was draining my battery instead of saving it.
    Step Three: After a little digging around in my installed programs I located a program called Lenovo Settings Dependency Package. PwrMgrBkGndMonitor was installed in the Lenovo Settings Dependency Package.
    I didn’t use this program and decided I didn’t need it. I uninstalled the program and restarted my laptop. I haven’t had a problem with the rundll32.exe using up my CPU since I uninstalled Lenovo Settings Dependency Package.
    If you are experiencing the same symptoms has stated above and you are running a Lenovo laptop with Windows 8.1, this may be your issue. However, before you uninstall the program I would recommend using Process Explorer. Process Explorer will help you determine the exact program that is giving you issues before blindly uninstalling and killing services. I hope my troubleshooting steps can help you find program is making rundll32.exe use high CPU usage. Let me know in the comments if this helped you!
    Source: techspeeder.com

  • Oracle.exe consuming 100% CPU on windows and database hang

    Hi all,
    every time my oracle database is hanging when the application run, the problem is the oracle.exe consum 100% CPU but not memory and the server hang and the dabase is going to inaccessible, we need to restart oracle instance service or server to bring the databas eback to normal but it's not permanent because the problem occurs once the application turn on.
    Checking the log file i found the below error every time:
    My database version is 9.2.0.7.0
    OS: Windows 2003 Server Standard Edition Service Pack 2
    RAM: 3,5Gb
    CPU: Inte Xeon 3.20 GHz
    ORA-00600: internal error code, arguments: [kghuclientasp_03], [0xBFEADCE0], [0], [0], [0], [], [], []
    ORA-29913: error in executing ODCIEXTTABLEFETCH callout
    ORA-29400: data cartridge error
    KUP-04050: error while attempting to allocate 163500 bytes of memory
    ORA-06512: at "SYS.ORACLE_LOADER", line 14
    ORA-06512: at line 1
    Fri Mar 05 05:35:15 2010
    Errors in file e:\oracle\admin\optprod\udump\optprod_ora_5876.trc:
    ORA-00603: ORACLE server session terminated by fatal error
    ORA-04030: out of process memory when trying to allocate 8389132 bytes (pga heap,redo read buffer)
    ORA-04030: out of process memory when trying to allocate 8389132 bytes (pga heap,redo read buffer)
    ORA-04030: out of process memory when trying to allocate 8180 bytes (callheap,kcbtmal allocation)
    Thank you
    Lucienot.

    Is this a new application on this database?
    Has it run well in the past?
    I have had this happen before on a 32bit Windows server. Our problem was a poorly written procedure that kept pegging the cpu to 100%. You should be able to figure out what SQL is being used that is causing this problem, it will be the Top Working SQL most likely.
    I also had this problem on a Logical Standby server which was trying to apply SQL to the SYS.AUD$ table. As soon as SQL Apply was started, the CPU went to 100%. Once I truncated that table, the cpu usage went back to normal. Not sure what you are using to monitor your database but if you can, try to find out what SQL is running when your CPU goes to 100%.

  • Oracle.exe(oralce 10.1)  on Window 2003 is using 100% CPU

    Oracle.exe(oralce 10.1) on Window 2003 is using 100% CPU.
    while using importing. CPU IS PENTIUM 4 AND RAM IS 2G
    any tuning ideas ?

    WEll I just did google for you:
    Take a look at:
    http://www.dba-village.com/village/dvp_forum.OpenThread?ThreadIdA=15940&DestinationA=RSS
    Hope this Helps
    Regards

  • Oracle on NT(taking 100% CPU)

    I have my Oracle running on NT.It is a Dell machine .The database isna test database.When the instance starts up it is taking 100% CPU.No activity possible on it.All querries hanging.Any suggestions ?

    Please look at patch name EM_NT_1224539.
    null

  • InterMedia search within report components causing Oracle to run at 100%cpu

    Hi,
    Running Win NT SP5
    Oracle 8.1.7.1
    IAS 1.0.2.2
    Portal 3.0.9.8.1
    I have a table with an intermedia index.
    My report query has :
    AND contains(i.description,:b_description) > 0
    This appears to be the reason why my application components sometimes run othertime doesnt. It also seems to have caused my CPU to go to 100% .. such that Oracle process taking 100% CPU.
    Take out this line.. and everything works fine!
    Anyone else experienced this, any fix available ?
    matt.

    Hi,
    Running Win NT SP5
    Oracle 8.1.7.1
    IAS 1.0.2.2
    Portal 3.0.9.8.1
    I have a table with an intermedia index.
    My report query has :
    AND contains(i.description,:b_description) > 0
    This appears to be the reason why my application components sometimes run othertime doesnt. It also seems to have caused my CPU to go to 100% .. such that Oracle process taking 100% CPU.
    Take out this line.. and everything works fine!
    Anyone else experienced this, any fix available ?
    matt.

  • Some Oracle processes are consuming 100% CPU

    Hi,
    Some of my oracle processes are consuming most amount of cpu continuously.What are the general steps to sort out this problem.I think the query is a tuned one.

    Hi,
    What is your veriosn of Oracle? what is the OS? which oracle process is consuming 100% CPU? how did u find out it is conuming?
    and you have mentiond query is tuned one... it does your query is taking time to execute?
    there is no clarity in your question...
    Regards,
    Vijayaraghavan K

  • Oracle.exe is consuming 100% cpu o windows

    Hi,
    We have 6 oracle 10g database on windows box and from task manager we are able to see oracle.exe is consuming 100% cpu
    Need to know this oracle.exe belongs to which database and can i get the sql for this process
    Thanks

    On my Vista PC, I find that the easiest way to identify which instance an oracle.exe is, is the task manager. On the Processes tab, right-click the oracle.exe and then click Go To Service(s) and that takes me to the OracleServiceORCL or whichever insance it is.
    I'm sure there is a better way that a Windows person could tell us about. Then having identified the instance, it is just normal tuning from there.

  • 100% CPU Usage Overhead running EM DBConsole 11g on OEL-5.2

    After upgrading to OEL-5.2 and relinking all Oracle binaries, my old Oracle 11g installation, installed several months before on OEL-5.1, has been working well, including Enterprise Manager Database Console working nicely as always with respectful performance. Unfortunatelly, it lasted just several days.
    Yesterday I decided to uninstall the 11g completely and perform new clean installation (software and database) with the same configuration options and settings as before, including EM dbconsole, all configured using dbca. After completing the installation (EM was started automatically by dbca), oracle continued to suck 80-85% CPU time. In further few minutes CPU utilization raised up to 99% due to only one (always the same PID) client process - "oracleorcl (LOCAL=NO)". For first ten minutes I didn't care too much since I always enable Automatic Management in dbca. But after two hours, I started to worry. The process was still running, consuming sustained 99% of CPU power. No other system activity, no database activity, no disks activity at all!
    I was really puzzled since I installed and reinstalled the 11g at least 20 times on OEL-5.0 and 5.1, experimenting with ASM, raw devices, loopback devices and various combinations of installation options, but never experienced such a behaviour. It took me 3 minutes to log in to EM dbconsole as it was almost unusable performing too slow. After three hours CPU temperature was nearly 60 degrees celsius. I decided to shutdown EM and after that everything became quiet. Oracle was running normally. Started EM again, the problem was back again. Tracing enabled, it filled a 350 MB trace file in just 20 minutes. Reinstalling the software and database once again didn't help. Whenever EM is up, the CPU usage overhead of 99% persists.
    Here is a cca 23 minutes session summary report taken from EM dbconsole's Performance page. The trace file is too big to list it here, but it shows the same.
            Host CPU:  100%
    Active Sessions:  100%The details for the Selected 5 Minute Interval (the last 5 min interval) are shown as follow:
        TOP SESSIONS:  SYSMAN, Program: OMS
            Activity:  100%  
         TOP MODULES:  OEM.CacheModeWaitPool, Service: orcl
            Activity:  100%          
          TOP CLIENT:  Unnamed
            Activity:  99.1%
         TOP ACTIONS:  Unnamed (OEM.CacheModeWaitPool) (orcl)
            Activity:  100%
         TOP OBJECTS: SYSMAN.MGMT_JOB_EXEC_SUMMARY (Table)
            Activity:  100%
          TOP PL/SQL:  SYSMAN.MGMT_JOB_ENGINE.INSERT_EXECUTION
       PL/SQL Source:  SYSMAN.MGMT_JOB_ENGINE
         Line Number:  7135
            Activity:  100%
             TOP SQL:  SELECT EXECUTION_ID, STATUS, STATUS_DETAIL FROM MGMT_JOB_EXEC_SUMMARY
    WHERE JOB_ID = :B3 AND TARGET_LIST_INDEX = :B2 AND EXPECTED_START_TIME = :B1;
            Activity:  100%
                                  STATISTICS SUMMARY
                                cca 23 minutes session
                            with no other system activity
                                            Per 
                           Total      Execution         Per Row
    Executions           105,103                 1       10,510.30
    Elapsed Time (sec)  1,358.95              0.01        135.90
    CPU Time (sec)      1,070.42             0.01        107.04
    Buffer Gets       85,585,518 814.30 8,558,551.80
    Disk Reads                 2            <0.01          0.20
    Direct Writes              0              0.00          0.00
    Rows                      10            <0.01             1
    Fetches              105,103             1.00     10,510.30
                       ----------------------------------------Wow!!! Note: no disk, no database activity !
    Has anyone experienced this or similar behaviour after clean 11g installation on OEL-5.2? If not, anyone has a clue what the hell is going on?
    Thanks in advance.

    Hi Tommy,
    I didn't want to experiment further with already working OEL-5.2, oracle and dbconsole on this machine, specially not after googling the problem and finding out that I am not alone in this world. There are another two threads on OTN forums (Database General) showing the same problem even on 2GB machines:
    DBConsole easting a CPU
    11g stuck. 50-100% CPU after fresh install
    So, I took another, a smaller free machine I've got at home (1GB RAM, 2.2MHz Pentium4, three 80GB disks), on which I used to experiment with new releases of software (this is the machine on which I installed 11g for the first time when it was released on OEL-5.0, and I can recall that everything was OK with EM). This is what I did:
    1. I installed OEL-5.0 on the machine, adjusted linux and kernel parameters, and performed full 11g installation. Database and EM dbconsole worked nice with acceptable performance. Without activity in the database, %CPU = zero !!! The whole system was perfectly quiet.
    2. Since everything was OK, I shutdown EM and oracle, and performed the full upgrade to OEL-5.2. When the upgrade finished, restarted the system, relinked all oracle binaries, and started oracle and EM dbconsole. Both worked perfectly again, just as before the upgrade. I repeated restarting the database and dbconsole several times, always with the same result - it really rocks. Without database activity, %CPU = zero%.
    3. Using dbca, I dropped the database and created the new one with the same configuration options. Wow! I'm again in trouble. A half an hour after the creation of the database, %CPU raised up to 99%. That's it.
    The crucial question here is: what is that in OEL-5.2, not existing in the 5.0, that causes dbca/em scripts to be embarrassed at the time of EM agent configuration?
    Here are the outputs you required picked 30 minutes after starting the database and EM dbconsole (sustained 99% CPU utilization). Note that this is just a 1GB machine.
    Kernel command line: ro root=LABEL=/ elevator=deadline rhgb quiet
    [root@localhost ~]# cat /proc/meminfo
    MemTotal:      1034576 kB
    MemFree:         27356 kB
    Buffers:          8388 kB
    Cached:         609660 kB
    SwapCached:      18628 kB
    Active:         675376 kB
    Inactive:       287072 kB
    HighTotal:      130304 kB
    HighFree:          260 kB
    LowTotal:       904272 kB
    LowFree:         27096 kB
    SwapTotal:     3148700 kB
    SwapFree:      2940636 kB
    Dirty:              72 kB
    Writeback:           0 kB
    AnonPages:      328700 kB
    Mapped:         271316 kB
    Slab:            21136 kB
    PageTables:      14196 kB
    NFS_Unstable:        0 kB
    Bounce:              0 kB
    CommitLimit:   3665988 kB
    Committed_AS:  1187464 kB
    VmallocTotal:   114680 kB
    VmallocUsed:      5860 kB
    VmallocChunk:   108476 kB
    HugePages_Total:     0
    HugePages_Free:      0
    HugePages_Rsvd:      0
    Hugepagesize:     4096 kB
    [root@localhost ~]# cat /proc/slabinfo
    slabinfo - version: 2.1
    # name            <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
    rpc_buffers            8      8   2048    2    1 : tunables   24   12    8 : slabdata      4      4      0
    rpc_tasks              8     15    256   15    1 : tunables  120   60    8 : slabdata      1      1      0
    rpc_inode_cache        6      7    512    7    1 : tunables   54   27    8 : slabdata      1      1      0
    ip_conntrack_expect    0      0     96   40    1 : tunables  120   60    8 : slabdata      0      0      0
    ip_conntrack          68     68    228   17    1 : tunables  120   60    8 : slabdata      4      4      0
    ip_fib_alias           7    113     32  113    1 : tunables  120   60    8 : slabdata      1      1      0
    ip_fib_hash            7    113     32  113    1 : tunables  120   60    8 : slabdata      1      1      0
    fib6_nodes            22    113     32  113    1 : tunables  120   60    8 : slabdata      1      1      0
    ip6_dst_cache         13     15    256   15    1 : tunables  120   60    8 : slabdata      1      1      0
    ndisc_cache            1     15    256   15    1 : tunables  120   60    8 : slabdata      1      1      0
    RAWv6                  4      5    768    5    1 : tunables   54   27    8 : slabdata      1      1      0
    UDPv6                  9     12    640    6    1 : tunables   54   27    8 : slabdata      2      2      0
    tw_sock_TCPv6          0      0    128   30    1 : tunables  120   60    8 : slabdata      0      0      0
    request_sock_TCPv6     0      0    128   30    1 : tunables  120   60    8 : slabdata      0      0      0
    TCPv6                  1      3   1280    3    1 : tunables   24   12    8 : slabdata      1      1      0
    jbd_1k                 0      0   1024    4    1 : tunables   54   27    8 : slabdata      0      0      0
    dm_mpath               0      0     28  127    1 : tunables  120   60    8 : slabdata      0      0      0
    dm_uevent              0      0   2460    3    2 : tunables   24   12    8 : slabdata      0      0      0
    dm_tio                 0      0     16  203    1 : tunables  120   60    8 : slabdata      0      0      0
    dm_io                  0      0     20  169    1 : tunables  120   60    8 : slabdata      0      0      0
    jbd_4k                 1      1   4096    1    1 : tunables   24   12    8 : slabdata      1      1      0
    scsi_cmd_cache        10     10    384   10    1 : tunables   54   27    8 : slabdata      1      1      0
    sgpool-128            36     36   2048    2    1 : tunables   24   12    8 : slabdata     18     18      0
    sgpool-64             33     36   1024    4    1 : tunables   54   27    8 : slabdata      9      9      0
    sgpool-32             34     40    512    8    1 : tunables   54   27    8 : slabdata      5      5      0
    sgpool-16             35     45    256   15    1 : tunables  120   60    8 : slabdata      3      3      0
    sgpool-8              60     60    128   30    1 : tunables  120   60    8 : slabdata      2      2      0
    scsi_io_context        0      0    104   37    1 : tunables  120   60    8 : slabdata      0      0      0
    ext3_inode_cache    4376   8216    492    8    1 : tunables   54   27    8 : slabdata   1027   1027      0
    ext3_xattr           165    234     48   78    1 : tunables  120   60    8 : slabdata      3      3      0
    journal_handle         8    169     20  169    1 : tunables  120   60    8 : slabdata      1      1      0
    journal_head         684   1008     52   72    1 : tunables  120   60    8 : slabdata     14     14      0
    revoke_table          18    254     12  254    1 : tunables  120   60    8 : slabdata      1      1      0
    revoke_record          0      0     16  203    1 : tunables  120   60    8 : slabdata      0      0      0
    uhci_urb_priv          0      0     28  127    1 : tunables  120   60    8 : slabdata      0      0      0
    UNIX                  56    112    512    7    1 : tunables   54   27    8 : slabdata     16     16      0
    flow_cache             0      0    128   30    1 : tunables  120   60    8 : slabdata      0      0      0
    cfq_ioc_pool           0      0     92   42    1 : tunables  120   60    8 : slabdata      0      0      0
    cfq_pool               0      0     96   40    1 : tunables  120   60    8 : slabdata      0      0      0
    crq_pool               0      0     44   84    1 : tunables  120   60    8 : slabdata      0      0      0
    deadline_drq         140    252     44   84    1 : tunables  120   60    8 : slabdata      3      3      0
    as_arq                 0      0     56   67    1 : tunables  120   60    8 : slabdata      0      0      0
    mqueue_inode_cache     1      6    640    6    1 : tunables   54   27    8 : slabdata      1      1      0
    isofs_inode_cache      0      0    368   10    1 : tunables   54   27    8 : slabdata      0      0      0
    hugetlbfs_inode_cache  1     11    340   11    1 : tunables   54   27    8 : slabdata      1      1      0
    ext2_inode_cache       0      0    476    8    1 : tunables   54   27    8 : slabdata      0      0      0
    ext2_xattr             0      0     48   78    1 : tunables  120   60    8 : slabdata      0      0      0
    dnotify_cache          2    169     20  169    1 : tunables  120   60    8 : slabdata      1      1      0
    dquot                  0      0    128   30    1 : tunables  120   60    8 : slabdata      0      0      0
    eventpoll_pwq          1    101     36  101    1 : tunables  120   60    8 : slabdata      1      1      0
    eventpoll_epi          1     30    128   30    1 : tunables  120   60    8 : slabdata      1      1      0
    inotify_event_cache    1    127     28  127    1 : tunables  120   60    8 : slabdata      1      1      0
    inotify_watch_cache   23     92     40   92    1 : tunables  120   60    8 : slabdata      1      1      0
    kioctx               135    135    256   15    1 : tunables  120   60    8 : slabdata      9      9      0
    kiocb                  0      0    128   30    1 : tunables  120   60    8 : slabdata      0      0      0
    fasync_cache           0      0     16  203    1 : tunables  120   60    8 : slabdata      0      0      0
    shmem_inode_cache    553    585    436    9    1 : tunables   54   27    8 : slabdata     65     65      0
    posix_timers_cache     0      0     88   44    1 : tunables  120   60    8 : slabdata      0      0      0
    uid_cache              5     59     64   59    1 : tunables  120   60    8 : slabdata      1      1      0
    ip_mrt_cache           0      0    128   30    1 : tunables  120   60    8 : slabdata      0      0      0
    tcp_bind_bucket       32    203     16  203    1 : tunables  120   60    8 : slabdata      1      1      0
    inet_peer_cache        1     59     64   59    1 : tunables  120   60    8 : slabdata      1      1      0
    secpath_cache          0      0     32  113    1 : tunables  120   60    8 : slabdata      0      0      0
    xfrm_dst_cache         0      0    384   10    1 : tunables   54   27    8 : slabdata      0      0      0
    ip_dst_cache           6     15    256   15    1 : tunables  120   60    8 : slabdata      1      1      0
    arp_cache              2     15    256   15    1 : tunables  120   60    8 : slabdata      1      1      0
    RAW                    2      7    512    7    1 : tunables   54   27    8 : slabdata      1      1      0
    UDP                    3      7    512    7    1 : tunables   54   27    8 : slabdata      1      1      0
    tw_sock_TCP            3     30    128   30    1 : tunables  120   60    8 : slabdata      1      1      0
    request_sock_TCP       4     30    128   30    1 : tunables  120   60    8 : slabdata      1      1      0
    TCP                   43     49   1152    7    2 : tunables   24   12    8 : slabdata      7      7      0
    blkdev_ioc             3    127     28  127    1 : tunables  120   60    8 : slabdata      1      1      0
    blkdev_queue          23     24    956    4    1 : tunables   54   27    8 : slabdata      6      6      0
    blkdev_requests      137    161    172   23    1 : tunables  120   60    8 : slabdata      7      7      0
    biovec-256             7      8   3072    2    2 : tunables   24   12    8 : slabdata      4      4      0
    biovec-128             7     10   1536    5    2 : tunables   24   12    8 : slabdata      2      2      0
    biovec-64              7     10    768    5    1 : tunables   54   27    8 : slabdata      2      2      0
    biovec-16              7     15    256   15    1 : tunables  120   60    8 : slabdata      1      1      0
    biovec-4               8     59     64   59    1 : tunables  120   60    8 : slabdata      1      1      0
    biovec-1             406    406     16  203    1 : tunables  120   60    8 : slabdata      2      2    300
    bio                  564    660    128   30    1 : tunables  120   60    8 : slabdata     21     22    204
    utrace_engine_cache    0      0     32  113    1 : tunables  120   60    8 : slabdata      0      0      0
    utrace_cache           0      0     32  113    1 : tunables  120   60    8 : slabdata      0      0      0
    sock_inode_cache     149    230    384   10    1 : tunables   54   27    8 : slabdata     23     23      0
    skbuff_fclone_cache   20     20    384   10    1 : tunables   54   27    8 : slabdata      2      2      0
    skbuff_head_cache     86    210    256   15    1 : tunables  120   60    8 : slabdata     14     14      0
    file_lock_cache       22     40     96   40    1 : tunables  120   60    8 : slabdata      1      1      0
    Acpi-Operand        1147   1196     40   92    1 : tunables  120   60    8 : slabdata     13     13      0
    Acpi-ParseExt          0      0     44   84    1 : tunables  120   60    8 : slabdata      0      0      0
    Acpi-Parse             0      0     28  127    1 : tunables  120   60    8 : slabdata      0      0      0
    Acpi-State             0      0     44   84    1 : tunables  120   60    8 : slabdata      0      0      0
    Acpi-Namespace       615    676     20  169    1 : tunables  120   60    8 : slabdata      4      4      0
    delayacct_cache      233    312     48   78    1 : tunables  120   60    8 : slabdata      4      4      0
    taskstats_cache       12     53     72   53    1 : tunables  120   60    8 : slabdata      1      1      0
    proc_inode_cache     622    693    356   11    1 : tunables   54   27    8 : slabdata     63     63      0
    sigqueue               8     27    144   27    1 : tunables  120   60    8 : slabdata      1      1      0
    radix_tree_node     6220   8134    276   14    1 : tunables   54   27    8 : slabdata    581    581      0
    bdev_cache            37     42    512    7    1 : tunables   54   27    8 : slabdata      6      6      0
    sysfs_dir_cache     4980   4992     48   78    1 : tunables  120   60    8 : slabdata     64     64      0
    mnt_cache             36     60    128   30    1 : tunables  120   60    8 : slabdata      2      2      0
    inode_cache         1113   1254    340   11    1 : tunables   54   27    8 : slabdata    114    114     81
    dentry_cache       11442  18560    136   29    1 : tunables  120   60    8 : slabdata    640    640    180
    filp                7607  10000    192   20    1 : tunables  120   60    8 : slabdata    500    500    120
    names_cache           19     19   4096    1    1 : tunables   24   12    8 : slabdata     19     19      0
    avc_node              14     72     52   72    1 : tunables  120   60    8 : slabdata      1      1      0
    selinux_inode_security 814   1170     48   78    1 : tunables  120   60    8 : slabdata     15     15      0
    key_jar               14     30    128   30    1 : tunables  120   60    8 : slabdata      1      1      0
    idr_layer_cache      170    203    136   29    1 : tunables  120   60    8 : slabdata      7      7      0
    buffer_head        38892  39024     52   72    1 : tunables  120   60    8 : slabdata    542    542      0
    mm_struct            108    135    448    9    1 : tunables   54   27    8 : slabdata     15     15      0
    vm_area_struct     11169  14904     84   46    1 : tunables  120   60    8 : slabdata    324    324    144
    fs_cache              82    177     64   59    1 : tunables  120   60    8 : slabdata      3      3      0
    files_cache          108    140    384   10    1 : tunables   54   27    8 : slabdata     14     14      0
    signal_cache         142    171    448    9    1 : tunables   54   27    8 : slabdata     19     19      0
    sighand_cache        127    135   1344    3    1 : tunables   24   12    8 : slabdata     45     45      0
    task_struct          184    246   1360    3    1 : tunables   24   12    8 : slabdata     82     82      0
    anon_vma            3313   5842     12  254    1 : tunables  120   60    8 : slabdata     23     23      0
    pgd                   84     84   4096    1    1 : tunables   24   12    8 : slabdata     84     84      0
    pid                  237    303     36  101    1 : tunables  120   60    8 : slabdata      3      3      0
    size-131072(DMA)       0      0 131072    1   32 : tunables    8    4    0 : slabdata      0      0      0
    size-131072            0      0 131072    1   32 : tunables    8    4    0 : slabdata      0      0      0
    size-65536(DMA)        0      0  65536    1   16 : tunables    8    4    0 : slabdata      0      0      0
    size-65536             2      2  65536    1   16 : tunables    8    4    0 : slabdata      2      2      0
    size-32768(DMA)        0      0  32768    1    8 : tunables    8    4    0 : slabdata      0      0      0
    size-32768             9      9  32768    1    8 : tunables    8    4    0 : slabdata      9      9      0
    size-16384(DMA)        0      0  16384    1    4 : tunables    8    4    0 : slabdata      0      0      0
    size-16384             6      6  16384    1    4 : tunables    8    4    0 : slabdata      6      6      0
    size-8192(DMA)         0      0   8192    1    2 : tunables    8    4    0 : slabdata      0      0      0
    size-8192              5      5   8192    1    2 : tunables    8    4    0 : slabdata      5      5      0
    size-4096(DMA)         0      0   4096    1    1 : tunables   24   12    8 : slabdata      0      0      0
    size-4096            205    205   4096    1    1 : tunables   24   12    8 : slabdata    205    205      0
    size-2048(DMA)         0      0   2048    2    1 : tunables   24   12    8 : slabdata      0      0      0
    size-2048            260    270   2048    2    1 : tunables   24   12    8 : slabdata    135    135      0
    size-1024(DMA)         0      0   1024    4    1 : tunables   54   27    8 : slabdata      0      0      0
    size-1024            204    204   1024    4    1 : tunables   54   27    8 : slabdata     51     51      0
    size-512(DMA)          0      0    512    8    1 : tunables   54   27    8 : slabdata      0      0      0
    size-512             367    464    512    8    1 : tunables   54   27    8 : slabdata     58     58      0
    size-256(DMA)          0      0    256   15    1 : tunables  120   60    8 : slabdata      0      0      0
    size-256             487    495    256   15    1 : tunables  120   60    8 : slabdata     33     33      0
    size-128(DMA)          0      0    128   30    1 : tunables  120   60    8 : slabdata      0      0      0
    size-128            2242   2490    128   30    1 : tunables  120   60    8 : slabdata     83     83      0
    size-64(DMA)           0      0     64   59    1 : tunables  120   60    8 : slabdata      0      0      0
    size-32(DMA)           0      0     32  113    1 : tunables  120   60    8 : slabdata      0      0      0
    size-64             1409   2950     64   59    1 : tunables  120   60    8 : slabdata     50     50      0
    size-32             3596   3842     32  113    1 : tunables  120   60    8 : slabdata     34     34      0
    kmem_cache           145    150    256   15    1 : tunables  120   60    8 : slabdata     10     10      0
    [root@localhost ~]# slabtop -d 5
    Active / Total Objects (% used)    : 97257 / 113249 (85.9%)
    Active / Total Slabs (% used)      : 4488 / 4488 (100.0%)
    Active / Total Caches (% used)     : 101 / 146 (69.2%)
    Active / Total Size (% used)       : 15076.34K / 17587.55K (85.7%)
    Minimum / Average / Maximum Object : 0.01K / 0.16K / 128.00K
      OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
    25776  25764  99%    0.05K    358       72      1432K buffer_head
    16146  15351  95%    0.08K    351       46      1404K vm_area_struct
    15138   7779  51%    0.13K    522       29      2088K dentry_cache
      9720   9106  93%    0.19K    486       20      1944K filp
      7714   7032  91%    0.27K    551       14      2204K radix_tree_node
      5070   5018  98%    0.05K     65       78       260K sysfs_dir_cache
      4826   4766  98%    0.01K     19      254        76K anon_vma
      4824   3406  70%    0.48K    603        8      2412K ext3_inode_cache
      3842   3691  96%    0.03K     34      113       136K size-32
      2190   2174  99%    0.12K     73       30       292K size-128
      1711   1364  79%    0.06K     29       59       116K size-64
      1210   1053  87%    0.33K    110       11       440K inode_cache
      1196   1147  95%    0.04K     13       92        52K Acpi-Operand
      1170    814  69%    0.05K     15       78        60K selinux_inode_security
       936    414  44%    0.05K     13       72        52K journal_head
       747    738  98%    0.43K     83        9       332K shmem_inode_cache
       693    617  89%    0.35K     63       11       252K proc_inode_cache
       676    615  90%    0.02K      4      169        16K Acpi-Namespace
       609    136  22%    0.02K      3      203        12K biovec-1
       495    493  99%    0.25K     33       15       132K size-256
       480    384  80%    0.12K     16       30        64K bio
       440    399  90%    0.50K     55        8       220K size-512
       312    206  66%    0.05K      4       78        16K delayacct_cache
       303    209  68%    0.04K      3      101        12K pid
       290    290 100%    0.38K     29       10       116K sock_inode_cache
    [root@localhost ~]# cat /etc/sysctl.conf
    # Kernel sysctl configuration file for Red Hat Linux
    # Controls IP packet forwarding
    net.ipv4.ip_forward=0
    # Controls source route verification
    net.ipv4.conf.default.rp_filter=1
    # Do not accept source routing
    net.ipv4.conf.default.accept_source_route=0
    # Oracle
    net.ipv4.ip_local_port_range=1024 65000
    net.core.rmem_default=4194304
    net.core.rmem_max=4194304
    net.core.wmem_default=262144
    net.core.wmem_max=262144
    net.ipv4.tcp_rmem=4096 65536 4194304
    net.ipv4.tcp_wmem=4096 65536 4194304
    # Keepalive Oracle
    net.ipv4.tcp_keepalive_time=3000
    net.ipv4.tcp_keepalive_intvl=30
    net.ipv4.tcp_keepalive_probes=15
    net.ipv4.tcp_retries2=3
    net.ipv4.tcp_syn_retries=2
    net.ipv4.tcp_sack=0
    net.ipv4.tcp_timestamps=0
    net.ipv4.tcp_window_scaling=0
    # Oracle
    fs.file-max = 6553600
    fs.aio-max-nr=3145728
    kernel.shmmni=4096
    kernel.sem=250 32000 100 142
    kernel.shmmax=2147483648
    kernel.shmall=3279547
    kernel.msgmnb=65536
    kernel.msgmni=2878
    kernel.msgmax=8192
    kernel.exec-shield=0
    # Controls the System Request debugging functionality of the kernel
    kernel.sysrq=1
    kernel.panic=60
    kernel.core_uses_pid=1
    [root@localhost ~]# free | grep Swap
    Swap:      3148700     319916    2828784
    [root@localhost ~]# cat /etc/fstab | grep "/dev/shm"
    tmpfs                   /dev/shm                tmpfs   size=1024M      0 0
    [root@localhost ~]# df | grep "/dev/shm"
    tmpfs                  1048576    452128    596448  44% /dev/shm
    NON-DEFAULT DB PARAMETERS:
    db_block_size        8192
    memory_target          633339904  /* automatic memory management */
    open_cursors         300
    processes            256
    disk_async_io        TRUE
    filesystemio_options SETALL

  • Sessions with Event "class slave wait" taking 100% CPU

    Hi experts,
    I have an issue in an 11.2.0.2 RAC database with 2 standby nodes that are driving me nuts.
    All started when I've been told there are 2 Oracle processes consuming too much CPU in the environment.
    I found 2 processes which takes 100% CPU each. These processes are Oracle processes into the DB and checking them out although they are registered in v$process and v$session, there is nothing related to them in v$bgprocess so I could not find out which oracle processes started these 2 OS processes.
    The view v$session shows them ACTIVE in the event "class slave wait". There aren't too much information about this event, at least I failed to find...
    Using dbms_monitor and dbms_system to create trace events did not create any tracefile. Only ORADEBUG was able to create events but with nearly no information inside the trace files.
    Questions I need to have answered or at least hints that I can follow to find the answer:
    1) what is causing the CPUs to be consumed at 100%? Which process?
    2) Why does these processes are using that much of CPU?
    3) What can be done to safely get over it?
    Honestly I don't know where else to look at except trying to get some help. Could someone give me a hand please?
    BR,
    Lauro Ojeda
    Edited by: LauroOjeda on 26/01/2013 06:44

    Hi Pal, thanks for your reply!
    Here are the answers:
    1. If it is RAC there is no such construct as a "standby node" and you say you have two of them. Please be specific ... is this RAC or Data Guard or a combination? Or do you have a three node cluster with all services pointing to only a single server wasting all of the resources of the other three?
    A: This is a combination of them. We have 2 nodes in a RAC environment shipping logs to two physical standby databases in another site.
    2. Two processes can not be at 100% of CPU any more than I can be 200% awake. Please show us how you arrived at this conclusion, on what hardware and operating system, and include a formatted (read the FAQ to learn how) extract showing what you are seeing.
    A: top in Linux shows two oracle processes consuming 100% (or nearly of it) of 1 CPU each. This is a 16 cores server (each primary node) so 2 of these are working on full capacity to service the described processes.
    3. What processes? Name them and again run a SQL statement and post the output so we can see what you are looking at.
    A: Like I said before I failed to find which background process they belongs to as there are no indications of them in v$bgprocess and in v$session/v$process either. I can see them in v$process and v$session but again, there are no indications of for which bg process they belong to.
    4. Is anything slow? Are there any problems with the system or are you only reacting to two numbers you think are too high?
    A: No, system is not slow because of that but the client wants to investigate and have it sorted as it is not normal.
    Additionally I found a bug in metalink which seems to be the culprit, but I'm unsure yet. This is Bug 12929268 : HIGH CPU ON ORA_O00N PROCESS
    Is any other information that I may provide you?
    Thanks for help!
    Lauro Ojeda

  • Oracle consumes 100% memory on Solaris 10

    Hi,
    Our database (Oracle 10g, R2) is running is running on Solaris, When I use the unix command prstat -a, it shows that, 100% Memomry utilized. Below are the details.
    PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
    12934 oracle 2573M 2559M sleep 59 0 0:00:00 0.1% oracle/1
    5914 sirsi 4912K 4664K sleep 59 0 0:00:29 0.0% prstat/1
    12937 oracle 4896K 4592K cpu3 49 0 0:00:00 0.0% prstat/1
    833 oracle 2572M 2558M sleep 59 0 0:01:05 0.0% oracle/1
    114 root 7464K 6632K sleep 59 0 0:01:20 0.0% picld/12
    829 oracle 2573M 2559M sleep 59 0 0:01:04 0.0% oracle/1
    823 oracle 2574M 2560M sleep 59 0 0:00:46 0.0% oracle/11
    811 oracle 2573M 2559M sleep 59 0 0:00:43 0.0% oracle/1
    146 root 2288K 1312K sleep 59 0 0:00:22 0.0% in.mpathd/1
    831 oracle 2576M 2562M sleep 59 0 0:00:24 0.0% oracle/1
    639 root 3664K 2392K sleep 59 0 0:00:00 0.0% snmpXdmid/2
    700 nobody 7520K 3752K sleep 59 0 0:00:00 0.0% httpd/1
    701 nobody 7520K 3752K sleep 59 0 0:00:00 0.0% httpd/1
    637 root 3080K 2048K sleep 59 0 0:00:00 0.0% dmispd/1
    472 root 5232K 2320K sleep 59 0 0:00:00 0.0% dtlogin/1
    720 root 2912K 2400K sleep 59 0 0:00:01 0.0% vold/5
    629 root 2376K 1664K sleep 59 0 0:00:00 0.0% snmpdx/1
    702 nobody 7520K 3736K sleep 59 0 0:00:00 0.0% httpd/1
    378 root 3928K 1784K sleep 59 0 0:00:00 0.0% sshd/1
    699 nobody 7520K 3704K sleep 59 0 0:00:00 0.0% httpd/1
    697 root 9384K 6520K sleep 59 0 0:00:01 0.0% snmpd/1
    695 root 7360K 5376K sleep 59 0 0:00:04 0.0% httpd/1
    375 root 12M 8088K sleep 59 0 0:00:01 0.0% fmd/15
    354 root 3728K 2040K sleep 59 0 0:00:00 0.0% syslogd/13
    415 root 2016K 1440K sleep 59 0 0:00:00 0.0% smcboot/1
    416 root 2008K 1016K sleep 59 0 0:00:00 0.0% smcboot/1
    338 root 4736K 1296K sleep 59 0 0:00:00 0.0% automountd/2
    340 root 5080K 2384K sleep 59 0 0:00:00 0.0% automountd/3
    263 daemon 2384K 1760K sleep 60 -20 0:00:00 0.0% lockd/2
    256 root 1280K 936K sleep 59 0 0:00:00 0.0% utmpd/1
    395 root 7592K 2560K sleep 59 0 0:00:02 0.0% sendmail/1
    273 root 2232K 1496K sleep 59 0 0:00:00 0.0% ttymon/1
    254 root 2072K 1224K sleep 59 0 0:00:00 0.0% sf880drd/1
    417 root 2008K 1016K sleep 59 0 0:00:00 0.0% smcboot/1
    272 root 5152K 4016K sleep 59 0 0:00:02 0.0% inetd/4
    206 root 1232K 536K sleep 59 0 0:00:00 0.0% efdaemon/1
    394 smmsp 7568K 1904K sleep 59 0 0:00:00 0.0% sendmail/1
    128 root 2904K 2056K sleep 59 0 0:00:00 0.0% devfsadm/6
    241 daemon 2640K 1528K sleep 59 0 0:00:00 0.0% rpcbind/1
    245 daemon 2672K 1992K sleep 59 0 0:00:00 0.0% statd/1
    251 root 2000K 1248K sleep 59 0 0:00:00 0.0% sac/1
    123 root 3992K 3008K sleep 59 0 0:00:07 0.0% nscd/26
    NPROC USERNAME SIZE RSS MEMORY TIME CPU
    24 oracle 48G 48G 100% 0:04:48 0.1%
    10 sirsi 1101M 35M 0.1% 0:00:32 0.0%
    37 root 148M 97M 0.2% 0:02:18 0.0%
    10 nobody 73M 36M 0.1% 0:00:00 0.0%
    1 smmsp 7568K 1904K 0.0% 0:00:00 0.0%
    4 daemon 12M 7920K 0.0% 0:00:00 0.0%
    Total: 86 processes, 260 lwps, load averages: 0.02, 0.02, 0.02
    Can anyone suggest why Oracle consumes 100% Memory? and how do we resolve this?.
    Regards,
    Sabdar Syed.

    Many Unix tools add to each dedicated server process memory the SGA size because under Unix each dedicated server is attaching the SGA shared memory segment to its process address space: so these Unix tools are not so reliable for Oracle.
    To check the Oracle memory usage, it is generally more recommendded to use the V$ views such as V$SGASTAT and V$PGASTAT.

  • Weblogic reaches 100% CPU utilisation

    I am using WebLogic Server Version 10.3.1.0. I got an application deployed in this server, which is developed with java1.6, ejb, hibernate. I start the server with 8GB of RAM assigned to it and after a day or two the weblogic server causes the use of the CPU to reach to 100% and as a result the server sort of doesn't response anymore. I will be creating 30-40 thread per managed server and one server has two managed server.  The volume will be around 30k per day. I dont know which is causing the 100% cpu utilisation and here below are the thread dumps which have taken at the time of high cpu in a server.  Please let me know where is the exact issue and where is the stuck thread (if any)? In which part the issue is occuring?
    10774:
    ===== FULL THREAD DUMP ===============
    Thu Jun 19 16:57:39 2014
    Oracle JRockit(R) R28.2.8-10-156881-1.6.0_51-20130611-1146-linux-x86_64
    "Main Thread" id=1 idx=0x4 tid=10776 prio=5 alive, waiting, native_blocked
        -- Waiting for notification on: weblogic/t3/srvr/T3Srvr@0xa4db4c58[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at java/lang/Object.wait(J)V(Native Method)
        at java/lang/Object.wait(Object.java:485)
        at weblogic/t3/srvr/T3Srvr.waitForDeath(T3Srvr.java:981)
        ^-- Lock released while waiting: weblogic/t3/srvr/T3Srvr@0xa4db4c58[fat lock]
        at weblogic/t3/srvr/T3Srvr.run(T3Srvr.java:490)
        at weblogic/Server.main(Server.java:71)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "(Signal Handler)" id=2 idx=0x8 tid=10777 prio=5 alive, native_blocked, daemon
    "(OC Main Thread)" id=3 idx=0xc tid=10778 prio=5 alive, native_waiting, daemon
    "(GC Worker Thread 1)" id=? idx=0x10 tid=10779 prio=5 alive, daemon
    "(GC Worker Thread 2)" id=? idx=0x14 tid=10780 prio=5 alive, daemon
    "(GC Worker Thread 3)" id=? idx=0x18 tid=10781 prio=5 alive, daemon
    "(GC Worker Thread 4)" id=? idx=0x1c tid=10782 prio=5 alive, daemon
    "(Code Generation Thread 1)" id=4 idx=0x20 tid=10783 prio=5 alive, native_waiting, daemon
    "(Code Optimization Thread 1)" id=5 idx=0x24 tid=10784 prio=5 alive, native_waiting, daemon
    "(VM Periodic Task)" id=6 idx=0x28 tid=10785 prio=10 alive, native_waiting, daemon
    "Finalizer" id=7 idx=0x2c tid=10786 prio=8 alive, native_waiting, daemon
        at jrockit/memory/Finalizer.waitForFinalizees(J[Ljava/lang/Object;)I(Native Method)
        at jrockit/memory/Finalizer.access$700(Finalizer.java:12)
        at jrockit/memory/Finalizer$4.run(Finalizer.java:201)
        at java/lang/Thread.run(Thread.java:662)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "Reference Handler" id=8 idx=0x30 tid=10787 prio=10 alive, native_waiting, daemon
        at java/lang/ref/Reference.waitForActivatedQueue(J)Ljava/lang/ref/Reference;(Native Method)
        at java/lang/ref/Reference.access$100(Reference.java:11)
        at java/lang/ref/Reference$ReferenceHandler.run(Reference.java:82)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "(Sensor Event Thread)" id=9 idx=0x34 tid=10788 prio=5 alive, native_blocked, daemon
    "VM JFR Buffer Thread" id=10 idx=0x38 tid=10789 prio=5 alive, in native, daemon
    "Timer-0" id=13 idx=0x3c tid=10793 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: java/util/TaskQueue@0xa4d5d260[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
        at java/lang/Object.wait(Object.java:485)[optimized]
        at java/util/TimerThread.mainLoop(Timer.java:483)
        ^-- Lock released while waiting: java/util/TaskQueue@0xa4d5d260[fat lock]
        at java/util/TimerThread.run(Timer.java:462)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '0' for queue: 'default'" id=14 idx=0x40 tid=10839 prio=5 alive, native_blocked, daemon
        at java/util/HashMap.get(HashMap.java:385)[optimized]
        at com/appl/vsms/pi/outbound/ControllerCache.getEntry(ControllerCache.java:48)[inlined]
        at com/appl/vsms/bean/pi/MessageStatusConsumer.onMessage(MessageStatusConsumer.java:90)[optimized]
        at weblogic/ejb/container/internal/MDListener.execute(MDListener.java:574)[inlined]
        at weblogic/ejb/container/internal/MDListener.run(MDListener.java:893)[optimized]
        at weblogic/work/ExecuteRequestAdapter.execute(ExecuteRequestAdapter.java:21)[optimized]
        at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)[optimized]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '1' for queue: 'default'" id=15 idx=0x44 tid=10840 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: java/util/LinkedList@0xa8cbed08[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
        at java/lang/Object.wait(Object.java:485)[inlined]
        at weblogic/ejb/container/internal/MDListener.getListener(MDListener.java:240)[optimized]
        ^-- Lock released while waiting: java/util/LinkedList@0xa8cbed08[fat lock]
        at weblogic/ejb/container/internal/MDListener.transactionalOnMessage(MDListener.java:480)[inlined]
        at weblogic/ejb/container/internal/MDListener.onMessage(MDListener.java:379)[optimized]
        at weblogic/jms/client/JMSSession.onMessage(JMSSession.java:4659)[optimized]
        at weblogic/jms/client/JMSSession.execute(JMSSession.java:4345)[optimized]
        at weblogic/jms/client/JMSSession.executeMessage(JMSSession.java:3821)[optimized]
        at weblogic/jms/client/JMSSession.access$000(JMSSession.java:115)
        at weblogic/jms/client/JMSSession$UseForRunnable.run(JMSSession.java:5170)
        at weblogic/work/ExecuteRequestAdapter.execute(ExecuteRequestAdapter.java:21)[optimized]
        at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)[optimized]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '2' for queue: 'default'" id=16 idx=0x48 tid=10841 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4d5f0d0[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
        at java/lang/Object.wait(Object.java:485)[inlined]
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)[optimized]
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4d5f0d0[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '3' for queue: 'default'" id=17 idx=0x4c tid=10842 prio=5 alive, native_blocked, daemon
        at java/util/HashMap.get(HashMap.java:385)[optimized]
        at com/appl/vsms/pi/outbound/ControllerCache.getEntry(ControllerCache.java:48)[inlined]
        at com/appl/vsms/bean/pi/MessageStatusConsumer.onMessage(MessageStatusConsumer.java:90)[optimized]
        at weblogic/ejb/container/internal/MDListener.execute(MDListener.java:574)[inlined]
        at weblogic/ejb/container/internal/MDListener.run(MDListener.java:893)[optimized]
        at weblogic/work/ExecuteRequestAdapter.execute(ExecuteRequestAdapter.java:21)[optimized]
        at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)[optimized]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '4' for queue: 'default'" id=18 idx=0x50 tid=10843 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4d5fe50[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
        at java/lang/Object.wait(Object.java:485)[inlined]
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)[optimized]
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4d5fe50[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '5' for queue: 'default'" id=19 idx=0x54 tid=10844 prio=5 alive, native_blocked, daemon
        at java/util/HashMap.get(HashMap.java:385)[optimized]
        at com/appl/vsms/pi/outbound/ControllerCache.getEntry(ControllerCache.java:48)[inlined]
        at com/appl/vsms/bean/pi/MessageStatusConsumer.onMessage(MessageStatusConsumer.java:90)[optimized]
        at weblogic/ejb/container/internal/MDListener.execute(MDListener.java:574)[inlined]
        at weblogic/ejb/container/internal/MDListener.run(MDListener.java:893)[optimized]
        at weblogic/work/ExecuteRequestAdapter.execute(ExecuteRequestAdapter.java:21)[optimized]
        at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)[optimized]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '6' for queue: 'default'" id=20 idx=0x58 tid=10845 prio=5 alive, native_blocked, daemon
        at java/util/HashMap.get(HashMap.java:385)[optimized]
        at com/appl/vsms/pi/outbound/ControllerCache.getEntry(ControllerCache.java:48)[inlined]
        at com/appl/vsms/bean/pi/MessageStatusConsumer.onMessage(MessageStatusConsumer.java:90)[optimized]
        at weblogic/ejb/container/internal/MDListener.execute(MDListener.java:574)[inlined]
        at weblogic/ejb/container/internal/MDListener.run(MDListener.java:893)[optimized]
        at weblogic/work/ExecuteRequestAdapter.execute(ExecuteRequestAdapter.java:21)[optimized]
        at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)[optimized]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '7' for queue: 'default'" id=21 idx=0x5c tid=10846 prio=5 alive, native_blocked, daemon
        at java/util/HashMap.get(HashMap.java:385)[optimized]
        at com/appl/vsms/pi/outbound/ControllerCache.getEntry(ControllerCache.java:48)[inlined]
        at com/appl/vsms/bean/pi/MessageStatusConsumer.onMessage(MessageStatusConsumer.java:90)[optimized]
        at weblogic/ejb/container/internal/MDListener.execute(MDListener.java:574)[inlined]
        at weblogic/ejb/container/internal/MDListener.run(MDListener.java:893)[optimized]
        at weblogic/work/ExecuteRequestAdapter.execute(ExecuteRequestAdapter.java:21)[optimized]
        at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)[optimized]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '8' for queue: 'default'" id=22 idx=0x60 tid=10847 prio=5 alive, native_blocked, daemon
        at java/util/HashMap.get(HashMap.java:385)[optimized]
        at com/appl/vsms/pi/outbound/ControllerCache.getEntry(ControllerCache.java:48)[inlined]
        at com/appl/vsms/bean/pi/MessageStatusConsumer.onMessage(MessageStatusConsumer.java:90)[optimized]
        at weblogic/ejb/container/internal/MDListener.execute(MDListener.java:574)[inlined]
        at weblogic/ejb/container/internal/MDListener.run(MDListener.java:893)[optimized]
        at weblogic/work/ExecuteRequestAdapter.execute(ExecuteRequestAdapter.java:21)[optimized]
        at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)[optimized]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '9' for queue: 'default'" id=23 idx=0x64 tid=10848 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4d62010[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
        at java/lang/Object.wait(Object.java:485)[inlined]
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)[optimized]
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4d62010[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '10' for queue: 'default'" id=24 idx=0x68 tid=10849 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4d626d0[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
        at java/lang/Object.wait(Object.java:485)[inlined]
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)[optimized]
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4d626d0[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '11' for queue: 'default'" id=25 idx=0x6c tid=10850 prio=5 alive, native_blocked, daemon
        at java/util/HashMap.get(HashMap.java:385)[optimized]
        at com/appl/vsms/pi/outbound/ControllerCache.getEntry(ControllerCache.java:48)[inlined]
        at com/appl/vsms/bean/pi/MessageStatusConsumer.onMessage(MessageStatusConsumer.java:90)[optimized]
        at weblogic/ejb/container/internal/MDListener.execute(MDListener.java:574)[inlined]
        at weblogic/ejb/container/internal/MDListener.run(MDListener.java:893)[optimized]
        at weblogic/work/ExecuteRequestAdapter.execute(ExecuteRequestAdapter.java:21)[optimized]
        at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)[optimized]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '12' for queue: 'default'" id=26 idx=0x70 tid=10851 prio=5 alive, native_blocked, daemon
        at java/util/HashMap.get(HashMap.java:385)[optimized]
        at com/appl/vsms/pi/outbound/ControllerCache.getEntry(ControllerCache.java:48)[inlined]
        at com/appl/vsms/bean/pi/MessageStatusConsumer.onMessage(MessageStatusConsumer.java:90)[optimized]
        at weblogic/ejb/container/internal/MDListener.execute(MDListener.java:574)[inlined]
        at weblogic/ejb/container/internal/MDListener.run(MDListener.java:893)[optimized]
        at weblogic/work/ExecuteRequestAdapter.execute(ExecuteRequestAdapter.java:21)[optimized]
        at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)[optimized]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '13' for queue: 'default'" id=27 idx=0x74 tid=10852 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4d648e8[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
        at java/lang/Object.wait(Object.java:485)[inlined]
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)[optimized]
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4d648e8[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '14' for queue: 'default'" id=28 idx=0x78 tid=10853 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4d65568[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
        at java/lang/Object.wait(Object.java:485)[inlined]
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)[optimized]
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4d65568[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '0' for queue: 'weblogic.kernel.Non-Blocking'" id=29 idx=0x7c tid=10854 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4d65cc0[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
        at java/lang/Object.wait(Object.java:485)[inlined]
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)[optimized]
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4d65cc0[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '1' for queue: 'weblogic.kernel.Non-Blocking'" id=30 idx=0x80 tid=10855 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4d663c0[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
        at java/lang/Object.wait(Object.java:485)[inlined]
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)[optimized]
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4d663c0[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '2' for queue: 'weblogic.kernel.Non-Blocking'" id=31 idx=0x84 tid=10856 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4d66ac0[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
        at java/lang/Object.wait(Object.java:485)[inlined]
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)[optimized]
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4d66ac0[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '0' for queue: 'weblogic.kernel.System'" id=32 idx=0x88 tid=10857 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4d671c0[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
        at java/lang/Object.wait(Object.java:485)[inlined]
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)[optimized]
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4d671c0[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '1' for queue: 'weblogic.kernel.System'" id=33 idx=0x8c tid=10858 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4d67f48[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
        at java/lang/Object.wait(Object.java:485)[inlined]
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)[optimized]
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4d67f48[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '2' for queue: 'weblogic.kernel.System'" id=34 idx=0x90 tid=10859 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4d68cc0[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
        at java/lang/Object.wait(Object.java:485)[inlined]
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)[optimized]
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4d68cc0[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '3' for queue: 'weblogic.kernel.System'" id=35 idx=0x94 tid=10860 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4d69a38[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
        at java/lang/Object.wait(Object.java:485)[inlined]
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)[optimized]
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4d69a38[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '4' for queue: 'weblogic.kernel.System'" id=36 idx=0x98 tid=10861 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4d977b8[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
        at java/lang/Object.wait(Object.java:485)[inlined]
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)[optimized]
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4d977b8[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '0' for queue: 'weblogic.Rejector'" id=37 idx=0x9c tid=10862 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4d99570[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at java/lang/Object.wait(J)V(Native Method)
        at java/lang/Object.wait(Object.java:485)
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4d99570[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '1' for queue: 'weblogic.Rejector'" id=38 idx=0xa0 tid=10863 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4cf2e38[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at java/lang/Object.wait(J)V(Native Method)
        at java/lang/Object.wait(Object.java:485)
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4cf2e38[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "weblogic.time.TimeEventGenerator" id=39 idx=0xa4 tid=10864 prio=9 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/time/common/internal/TimeTable@0xa4dbed18[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at java/lang/Object.wait(J)V(Native Method)
        at weblogic/time/common/internal/TimeTable.snooze(TimeTable.java:286)
        ^-- Lock released while waiting: weblogic/time/common/internal/TimeTable@0xa4dbed18[fat lock]
        at weblogic/time/common/internal/TimeEventGenerator.run(TimeEventGenerator.java:117)
        at java/lang/Thread.run(Thread.java:662)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "JMAPI event thread" id=40 idx=0xa8 tid=10865 prio=5 alive, in native, daemon
    "weblogic.timers.TimerThread" id=41 idx=0xac tid=10866 prio=9 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/timers/internal/TimerThread@0xa4dc3f70[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at java/lang/Object.wait(J)V(Native Method)
        at weblogic/timers/internal/TimerThread$Thread.run(TimerThread.java:262)
        ^-- Lock released while waiting: weblogic/timers/internal/TimerThread@0xa4dc3f70[fat lock]
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '0' for queue: 'weblogic.admin.RMI'" id=42 idx=0xb0 tid=10867 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4cf3528[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at java/lang/Object.wait(J)V(Native Method)
        at java/lang/Object.wait(Object.java:485)
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4cf3528[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '1' for queue: 'weblogic.admin.RMI'" id=43 idx=0xb4 tid=10868 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4cf35f0[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at java/lang/Object.wait(J)V(Native Method)
        at java/lang/Object.wait(Object.java:485)
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4cf35f0[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '2' for queue: 'weblogic.admin.RMI'" id=44 idx=0xb8 tid=10869 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4cf36b8[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
        at java/lang/Object.wait(Object.java:485)[inlined]
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)[optimized]
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4cf36b8[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '3' for queue: 'weblogic.admin.RMI'" id=45 idx=0xbc tid=10870 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4cf3780[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
        at java/lang/Object.wait(Object.java:485)[inlined]
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)[optimized]
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4cf3780[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '4' for queue: 'weblogic.admin.RMI'" id=46 idx=0xc0 tid=10871 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4cf3848[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
        at java/lang/Object.wait(Object.java:485)[inlined]
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)[optimized]
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4cf3848[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '0' for queue: 'ImageWorkManager'" id=47 idx=0xc4 tid=10896 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4cf57e0[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at java/lang/Object.wait(J)V(Native Method)
        at java/lang/Object.wait(Object.java:485)
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4cf57e0[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '1' for queue: 'ImageWorkManager'" id=48 idx=0xc8 tid=10899 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4cf58a8[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at java/lang/Object.wait(J)V(Native Method)
        at java/lang/Object.wait(Object.java:485)
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4cf58a8[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "JFR request timer" id=49 idx=0xcc tid=10900 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: java/util/TaskQueue@0xa4dc69d8[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at java/lang/Object.wait(J)V(Native Method)
        at java/lang/Object.wait(Object.java:485)
        at java/util/TimerThread.mainLoop(Timer.java:483)
        ^-- Lock released while waiting: java/util/TaskQueue@0xa4dc69d8[fat lock]
        at java/util/TimerThread.run(Timer.java:462)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '0' for queue: 'weblogic.logging.DomainLogBroadcasterClient'" id=53 idx=0xd0 tid=10902 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4cf6550[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at java/lang/Object.wait(J)V(Native Method)
        at java/lang/Object.wait(Object.java:485)
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4cf6550[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '0' for queue: 'weblogic.logging.LogBroadcaster'" id=54 idx=0xd4 tid=10903 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4cf6d50[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at java/lang/Object.wait(J)V(Native Method)
        at java/lang/Object.wait(Object.java:485)
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4cf6d50[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" id=55 idx=0xd8 tid=10904 prio=5 alive, blocked, native_blocked, daemon
        -- Blocked trying to get lock: java/lang/String@0xa4cf78f8[fat lock]
        at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)
        at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1411)[optimized]
        at jrockit/vm/Locks.lockFat(Locks.java:1512)[optimized]
        at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1054)[optimized]
        at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1005)[optimized]
        at jrockit/vm/Locks.monitorEnter(Locks.java:2179)[optimized]
        at weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:153)
        at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)
        at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)
        at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" id=56 idx=0xdc tid=10905 prio=5 alive, blocked, native_blocked, daemon
        -- Blocked trying to get lock: java/lang/String@0xa4cf78f8[fat lock]
        at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)
        at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1411)[optimized]
        at jrockit/vm/Locks.lockFat(Locks.java:1512)[optimized]
        at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1054)[optimized]
        at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1005)[optimized]
        at jrockit/vm/Locks.monitorEnter(Locks.java:2179)[optimized]
        at weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:153)
        at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)
        at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)
        at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" id=57 idx=0xe0 tid=10906 prio=5 alive, blocked, native_blocked, daemon
        -- Blocked trying to get lock: java/lang/String@0xa4cf78f8[fat lock]
        at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)
        at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1411)[inlined]
        at jrockit/vm/Locks.fatLockReacquire(Locks.java:1649)[optimized]
        at jrockit/vm/Locks.lockFat(Locks.java:1517)[optimized]
        at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1054)[optimized]
        at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1005)[optimized]
        at jrockit/vm/Locks.monitorEnter(Locks.java:2179)[optimized]
        at weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:153)
        at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)
        at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)
        at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '3' for queue: 'weblogic.socket.Muxer'" id=58 idx=0xe4 tid=10907 prio=5 alive, in native, daemon
        at jrockit/ext/epoll/EPoll.epollWait0(ILjava/nio/ByteBuffer;II)I(Native Method)
        at jrockit/ext/epoll/EPoll.epollWait(EPoll.java:123)
        at weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:156)
        ^-- Holding lock: java/lang/String@0xa4cf78f8[fat lock]
        at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)
        at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)
        at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "VDE Transaction Processor Thread" id=61 idx=0xe8 tid=10909 prio=2 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: com/octetstring/vde/backend/standard/TransactionProcessor@0xa46c3738[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at java/lang/Object.wait(J)V(Native Method)
        at java/lang/Object.wait(Object.java:485)
        at com/octetstring/vde/backend/standard/TransactionProcessor.waitChange(TransactionProcessor.java:367)
        ^-- Lock released while waiting: com/octetstring/vde/backend/standard/TransactionProcessor@0xa46c3738[fat lock]
        at com/octetstring/vde/backend/standard/TransactionProcessor.run(TransactionProcessor.java:212)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "DoSManager" id=64 idx=0xf0 tid=10911 prio=6 alive, sleeping, native_waiting, daemon
        at java/lang/Thread.sleep(J)V(Native Method)
        at com/octetstring/vde/DoSManager.run(DoSManager.java:433)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '0' for queue: 'UserLockout'" id=65 idx=0xf4 tid=10920 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4cf93c8[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at java/lang/Object.wait(J)V(Native Method)
        at java/lang/Object.wait(Object.java:485)
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4cf93c8[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '1' for queue: 'UserLockout'" id=66 idx=0xf8 tid=10921 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4cf9490[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at java/lang/Object.wait(J)V(Native Method)
        at java/lang/Object.wait(Object.java:485)
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4cf9490[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '2' for queue: 'UserLockout'" id=67 idx=0xfc tid=10922 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4cf9558[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at java/lang/Object.wait(J)V(Native Method)
        at java/lang/Object.wait(Object.java:485)
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4cf9558[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '3' for queue: 'UserLockout'" id=68 idx=0x100 tid=10923 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4cf9620[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at java/lang/Object.wait(J)V(Native Method)
        at java/lang/Object.wait(Object.java:485)
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4cf9620[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '4' for queue: 'UserLockout'" id=69 idx=0x104 tid=10924 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4cf96e8[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at java/lang/Object.wait(J)V(Native Method)
        at java/lang/Object.wait(Object.java:485)
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4cf96e8[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '0' for queue: 'weblogic.unicast.DispatchWorkManager'" id=70 idx=0x108 tid=10958 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4cfbbb0[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at java/lang/Object.wait(J)V(Native Method)
        at java/lang/Object.wait(Object.java:485)
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4cfbbb0[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '1' for queue: 'weblogic.unicast.DispatchWorkManager'" id=71 idx=0x10c tid=10959 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4cfbc78[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
        at java/lang/Object.wait(Object.java:485)[inlined]
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)[optimized]
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4cfbc78[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '2' for queue: 'weblogic.unicast.DispatchWorkManager'" id=72 idx=0x110 tid=10960 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4cfbd40[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
        at java/lang/Object.wait(Object.java:485)[inlined]
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)[optimized]
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4cfbd40[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '3' for queue: 'weblogic.unicast.DispatchWorkManager'" id=73 idx=0x114 tid=10961 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4cfbe08[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
        at java/lang/Object.wait(Object.java:485)[inlined]
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)[optimized]
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4cfbe08[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '4' for queue: 'weblogic.unicast.DispatchWorkManager'" id=74 idx=0x118 tid=10962 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4cfbed0[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
        at java/lang/Object.wait(Object.java:485)[inlined]
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)[optimized]
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4cfbed0[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '0' for queue: 'weblogic.unicast.ForwardingWorkManager'" id=75 idx=0x11c tid=10963 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4c66f40[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at java/lang/Object.wait(J)V(Native Method)
        at java/lang/Object.wait(Object.java:485)
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4c66f40[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '1' for queue: 'weblogic.unicast.ForwardingWorkManager'" id=76 idx=0x120 tid=10964 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4c67008[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at java/lang/Object.wait(J)V(Native Method)
        at java/lang/Object.wait(Object.java:485)
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4c67008[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '2' for queue: 'weblogic.unicast.ForwardingWorkManager'" id=77 idx=0x124 tid=10965 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4c670d0[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at java/lang/Object.wait(J)V(Native Method)
        at java/lang/Object.wait(Object.java:485)
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4c670d0[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '3' for queue: 'weblogic.unicast.ForwardingWorkManager'" id=78 idx=0x128 tid=10966 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4c67198[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at java/lang/Object.wait(J)V(Native Method)
        at java/lang/Object.wait(Object.java:485)
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4c67198[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '4' for queue: 'weblogic.unicast.ForwardingWorkManager'" id=79 idx=0x12c tid=10967 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4c67260[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at java/lang/Object.wait(J)V(Native Method)
        at java/lang/Object.wait(Object.java:485)
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4c67260[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '0' for queue: 'ClusterMessaging'" id=81 idx=0x134 tid=10969 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4c692e0[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
        at java/lang/Object.wait(Object.java:485)[inlined]
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)[optimized]
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4c692e0[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '0' for queue: 'wl_oldBootStrap'" id=82 idx=0x138 tid=10972 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4c69a90[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at java/lang/Object.wait(J)V(Native Method)
        at java/lang/Object.wait(Object.java:485)
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4c69a90[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '1' for queue: 'wl_oldBootStrap'" id=83 idx=0x13c tid=10973 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4c69b58[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at java/lang/Object.wait(J)V(Native Method)
        at java/lang/Object.wait(Object.java:485)
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4c69b58[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '2' for queue: 'wl_oldBootStrap'" id=84 idx=0x140 tid=10974 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4c69c20[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at java/lang/Object.wait(J)V(Native Method)
        at java/lang/Object.wait(Object.java:485)
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4c69c20[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '3' for queue: 'wl_oldBootStrap'" id=85 idx=0x144 tid=10975 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4c69ce8[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at java/lang/Object.wait(J)V(Native Method)
        at java/lang/Object.wait(Object.java:485)
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4c69ce8[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '4' for queue: 'wl_oldBootStrap'" id=86 idx=0x148 tid=10980 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4c69db0[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at java/lang/Object.wait(J)V(Native Method)
        at java/lang/Object.wait(Object.java:485)
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4c69db0[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '5' for queue: 'wl_oldBootStrap'" id=87 idx=0x14c tid=10984 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4c69e78[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at java/lang/Object.wait(J)V(Native Method)
        at java/lang/Object.wait(Object.java:485)
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4c69e78[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '6' for queue: 'wl_oldBootStrap'" id=88 idx=0x150 tid=10987 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4c69f40[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at java/lang/Object.wait(J)V(Native Method)
        at java/lang/Object.wait(Object.java:485)
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4c69f40[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '7' for queue: 'wl_oldBootStrap'" id=89 idx=0x154 tid=10988 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4c6a008[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
        at java/lang/Object.wait(Object.java:485)[inlined]
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)[optimized]
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4c6a008[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "Thread-12" id=90 idx=0x158 tid=10989 prio=5 alive, parked, native_blocked
        -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0xa4c66b48
        at jrockit/vm/Locks.park0(J)V(Native Method)
        at jrockit/vm/Locks.park(Locks.java:2230)
        at sun/misc/Unsafe.park(ZJ)V(Native Method)
        at java/util/concurrent/locks/LockSupport.park(LockSupport.java:156)
        at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
        at weblogic/utils/concurrent/JDK15ConcurrentBlockingQueue.take(JDK15ConcurrentBlockingQueue.java:89)
        at weblogic/store/internal/PersistentStoreImpl.getOutstandingWork(PersistentStoreImpl.java:672)
        at weblogic/store/internal/PersistentStoreImpl.run(PersistentStoreImpl.java:721)
        at java/lang/Thread.run(Thread.java:662)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '0' for queue: 'JTACoordinatorWM'" id=91 idx=0x15c tid=10990 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4c6d128[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
        at java/lang/Object.wait(Object.java:485)[inlined]
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)[optimized]
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4c6d128[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '1' for queue: 'JTACoordinatorWM'" id=92 idx=0x160 tid=10991 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4c6d1f0[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
        at java/lang/Object.wait(Object.java:485)[inlined]
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)[optimized]
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4c6d1f0[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '2' for queue: 'JTACoordinatorWM'" id=93 idx=0x164 tid=10992 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4c6d2b8[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
        at java/lang/Object.wait(Object.java:485)[inlined]
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)[optimized]
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4c6d2b8[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '0' for queue: 'JmsDispatcher'" id=94 idx=0x168 tid=10995 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4c6e688[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at java/lang/Object.wait(J)V(Native Method)
        at java/lang/Object.wait(Object.java:485)
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4c6e688[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '1' for queue: 'JmsDispatcher'" id=95 idx=0x16c tid=10996 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4c6e750[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at java/lang/Object.wait(J)V(Native Method)
        at java/lang/Object.wait(Object.java:485)
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4c6e750[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '2' for queue: 'JmsDispatcher'" id=96 idx=0x170 tid=10997 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4c6e818[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at java/lang/Object.wait(J)V(Native Method)
        at java/lang/Object.wait(Object.java:485)
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4c6e818[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '3' for queue: 'JmsDispatcher'" id=97 idx=0x174 tid=10998 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4c6e8e0[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
        at java/lang/Object.wait(J)V(Native Method)
        at java/lang/Object.wait(Object.java:485)
        at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
        ^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4c6e8e0[fat lock]
        at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
        at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
        -- end of trace
    "ExecuteThread: '4' for queue: 'JmsDispatcher'" id=98 idx=0x178 tid=10999 prio=5 alive, waiting, native_blocked, daemon
        -- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4c6e9a8[fat lock]
        at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
       

    Hi,
    Based on the below doc , you can narrow down the actual Thread that is causing this high CPU.
    Unexpected High CPU Usage with WebLogic Server (WLS) Support Pattern (Doc ID 779349.1)
    Thanks,
    Sharmela

Maybe you are looking for