Oracle occupy 100% CPU
Hi,
While running program from JDBC, Oracle process occupy 100% CPU. We have oracle 11gR2.we have 14GB RAM.Memory target is set to 4GB.can you please guide me how to troubleshoot.
user12116651 wrote:
While running program from JDBC, Oracle process occupy 100% CPU. We have oracle 11gR2.we have 14GB RAM.Memory target is set to 4GB.can you please guide me how to troubleshoot.Identify the physical o/s process consuming CPU via a command like top.
If this is an Oracle process, identify the session and process in the Oracle instance using the v$process and v$session views - the former contains the o/s process id and the latter is joined to get the Oracle session id and basic session details.
Next use the v$sess* views to analyse the wait states, events and statistics of that session. You can also determine the current SQL cursor being executed and use DBMS_XPLAN.display_cursor() to get its execution plan.
Details on the virtual performance v$ views are in the Oracle® Database Reference guide.
Similar Messages
-
Hi,
I have installed Oracle 10g R2 on 64bit linux. It was working fine, until last week. I got alert that oracle uses 100% CPU.
Some lines from alert:
4894 /opt/oracle/product/10.2.0/db/perl/bin/perl /opt/oracle/product/10.2.0/db/sysman/admin/scripts/db/dbresp.pl 56.3 12978 7720 22860 oracle
9244 /opt/oracle/product/10.2.0/db/perl/bin/perl /opt/oracle/product/10.2.0/db/sysman/admin/scripts/db/dbresp.pl 50.4 10357 7720 22860 oracle
11798 /opt/oracle/product/10.2.0/db/perl/bin/perl /opt/oracle/product/10.2.0/db/sysman/admin/scripts/db/dbresp.pl 41.1 4429 7720 22860 oracle
11740 /opt/oracle/product/10.2.0/db/perl/bin/perl /opt/oracle/product/10.2.0/db/sysman/admin/scripts/db/dbresp.pl 28.3 45350 7724 22864 oracle
1516 /opt/oracle/product/10.2.0/db/perl/bin/perl /opt/oracle/product/10.2.0/db/sysman/admin/scripts/db/dbresp.pl 24.2 29968 7728 22868 oracle
17006 /opt/oracle/product/10.2.0/db/perl/bin/perl /opt/oracle/product/10.2.0/db/sysman/admin/scripts/db/dbresp.pl 21.8 21913 7724 22864 oracle
19449 /opt/oracle/product/10.2.0/db/perl/bin/perl /opt/oracle/product/10.2.0/db/sysman/admin/scripts/db/dbresp.pl 21.5 20865 7720 22860 oracle
24877 /opt/oracle/product/10.2.0/db/perl/bin/perl /opt/oracle/product/10.2.0/db/sysman/admin/scripts/db/dbresp.pl 21.0 18708 7728 22868 oracle
26566 /opt/oracle/product/10.2.0/db/perl/bin/perl /opt/oracle/product/10.2.0/db/sysman/admin/scripts/db/dbresp.pl 18.4 5727 7724 22864 oracle
1788 /opt/oracle/product/10.2.0/db/perl/bin/perl /opt/oracle/product/10.2.0/db/sysman/admin/scripts/db/dbresp.pl 17.1 3319 7724 22864 oracle
When I log in to EM, I can see section
Host CPU
It is 100%.
I have only instance named "parama". and it uses 1% CPU ant the picture shows that 99% CPU is used By Other.
What is Other? Why does it use that much CPU?
I had another oracle instance few months ago, but I is not used now (it is not even started).
How do I restore normal system performance?
Can I kill these processes?Hello,
I have a similar problem. I see 2 perl processes related to Oracle that are taking up 99% of 2 cpus. These 2 perl processes have been running for days and keeping the load average high (above 2) on a mostly idle system.
The 2 perl processes are running:
/oracle/app/product/10.2.0/db_1/perl/bin/perl /oracle/app/product/10.2.0/db_1/sysman/admin/scripts/db/dbarchfull.pl
and
/oracle/app/product/10.2.0/db_1/perl/bin/perl /oracle/app/product/10.2.0/db_1/sysman/admin/scripts/db/dbresp.pl
Has anyone seen this problem? Can I kill the perl processes?
--Dan -
"du process" pops up and occupy 100% CPU
I didn't solve the issue. I've got a macbook pro 2010, partitioned with bootcamp and with os 10.6.8. "Du process" appears and disappears, and when pops up occupy 100% CPU. This process isn't present in my other macbooks with the same os. Is it possibile and useful (to get 100% cpu availability) to cancel it? How?
Thank you in advance for your help!Safari third party plug-ins and extensions can increase CPU usage.
From the Safari menu bar click Safari > Preferences then select the Extensions tab. Turn that OFF, quit and relaunch Safari to test.
If that helped, turn one extension on then quit and relaunch Safari to test until you find the incompatible extension then click uninstall.
If it's not an extensions issue, try troubleshooting third party plug-ins.
Back to Safari > Preferences. This time select the Security tab. Deselect: Allow plug-ins. Quit and relaunch Safari to test.
If that made a difference, instructions for troubleshooting plugins here. -
Oracle taking 100% CPU usage
Hi,
I have a situation. I am trying to execute a big query by involving 3 tables (joins). The query is taking about 20 minutes time to execute. While the query is executing the CPU usage for ORACLE is 100%. Please put somelight on this as of to why this is happening? Do I need to tweak my query a bit more. This is happening only recently as there are more number of records. ORACLE is installed on a Unix machine
Thanks in advance.The First Rule Of Performance Troubleshooting: IDENTIFY THE PERFORMANCE PROBLEM.
All you have done thus far is identifying a symptom. High CPU utilisation. This in itself is totally meaningless - nothing can be inferred from this symptom without additional troubleshooting.
This means determining WHAT is happening when that query/process runs. What is it doing? (look at wait states and event state summaries of the process)
WHY is it doing that? Look at the execution plan.
With performance tuning, one needs to deal with facts. So you need to pop the hood and get your hands dirty so see what is actually happening under the hood. -
hi i have a big problem whit the performance of my laptop. I bought a extra 4gb ram for the second slot that not exist; and still continuos getting my cpu usage at 100% whit 2 programs running like office and google crhome.
i have this specf of my latop:
- procesor: amd c-60 apu whit radeon hd graphics 1,00 ghz
- ram: 4gz
system type: 64-bit OS
winows 7 ultimate
please tell me if i have to install another OS, made changes in my configuration or just buy another laptop.
tksRundll32.exe Using High CPU on Windows 8.1
I use a Lenovo T440s laptop for my service calls and when I need to do work remotely. My laptop runs Windows 8.1 Professional. I love this laptop and it has proved to be tough as well as extremely fast. I noticed a couple weeks ago that my battery was going dead after about an hour of usage. I also noticed that my cooling fan in the laptop was running louder than normal. I decided to open Task Manger on my laptop to see what the issue was. I soon noticed that a process called rundll32.exe was using around 25 percent of my CPU. I killed the process and everything returned to normal.
A few days later, my battery was going dead very rapidly. My cooling fan was on almost constantly and was blowing warm air. I opened Task Manger and sure enough, rundll32.exe was using 25% of my CPU again. I killed the process and everything was fine. It wasn’t long and my laptop was doing the exact same thing as I described before. I decided it was time to get to the bottom of this issue. Here is how I troubleshot what program was using the rundll32.exe and in the process was killing my battery.
Step One: I downloaded Process Explorer. This is an excellent tool. It is similar to Task Manager but goes into way more depth on the processes and dlls that are running. I extracted the download and then ran Process Explorer as an administrator.
Step Two: One of the first things I noticed was rundll32.exe was using 23 percent of my CPU again.
I right-clicked on the rundll32.exe process and selected properties. I clicked on the Image tab and at once saw the culprit. A utility to monitor and gauge my battery usage, called PwrMgrBkGndMonitor, was draining my battery instead of saving it.
Step Three: After a little digging around in my installed programs I located a program called Lenovo Settings Dependency Package. PwrMgrBkGndMonitor was installed in the Lenovo Settings Dependency Package.
I didn’t use this program and decided I didn’t need it. I uninstalled the program and restarted my laptop. I haven’t had a problem with the rundll32.exe using up my CPU since I uninstalled Lenovo Settings Dependency Package.
If you are experiencing the same symptoms has stated above and you are running a Lenovo laptop with Windows 8.1, this may be your issue. However, before you uninstall the program I would recommend using Process Explorer. Process Explorer will help you determine the exact program that is giving you issues before blindly uninstalling and killing services. I hope my troubleshooting steps can help you find program is making rundll32.exe use high CPU usage. Let me know in the comments if this helped you!
Source: techspeeder.com -
Oracle.exe consuming 100% CPU on windows and database hang
Hi all,
every time my oracle database is hanging when the application run, the problem is the oracle.exe consum 100% CPU but not memory and the server hang and the dabase is going to inaccessible, we need to restart oracle instance service or server to bring the databas eback to normal but it's not permanent because the problem occurs once the application turn on.
Checking the log file i found the below error every time:
My database version is 9.2.0.7.0
OS: Windows 2003 Server Standard Edition Service Pack 2
RAM: 3,5Gb
CPU: Inte Xeon 3.20 GHz
ORA-00600: internal error code, arguments: [kghuclientasp_03], [0xBFEADCE0], [0], [0], [0], [], [], []
ORA-29913: error in executing ODCIEXTTABLEFETCH callout
ORA-29400: data cartridge error
KUP-04050: error while attempting to allocate 163500 bytes of memory
ORA-06512: at "SYS.ORACLE_LOADER", line 14
ORA-06512: at line 1
Fri Mar 05 05:35:15 2010
Errors in file e:\oracle\admin\optprod\udump\optprod_ora_5876.trc:
ORA-00603: ORACLE server session terminated by fatal error
ORA-04030: out of process memory when trying to allocate 8389132 bytes (pga heap,redo read buffer)
ORA-04030: out of process memory when trying to allocate 8389132 bytes (pga heap,redo read buffer)
ORA-04030: out of process memory when trying to allocate 8180 bytes (callheap,kcbtmal allocation)
Thank you
Lucienot.Is this a new application on this database?
Has it run well in the past?
I have had this happen before on a 32bit Windows server. Our problem was a poorly written procedure that kept pegging the cpu to 100%. You should be able to figure out what SQL is being used that is causing this problem, it will be the Top Working SQL most likely.
I also had this problem on a Logical Standby server which was trying to apply SQL to the SYS.AUD$ table. As soon as SQL Apply was started, the CPU went to 100%. Once I truncated that table, the cpu usage went back to normal. Not sure what you are using to monitor your database but if you can, try to find out what SQL is running when your CPU goes to 100%. -
Oracle.exe(oralce 10.1) on Window 2003 is using 100% CPU
Oracle.exe(oralce 10.1) on Window 2003 is using 100% CPU.
while using importing. CPU IS PENTIUM 4 AND RAM IS 2G
any tuning ideas ?WEll I just did google for you:
Take a look at:
http://www.dba-village.com/village/dvp_forum.OpenThread?ThreadIdA=15940&DestinationA=RSS
Hope this Helps
Regards -
Oracle on NT(taking 100% CPU)
I have my Oracle running on NT.It is a Dell machine .The database isna test database.When the instance starts up it is taking 100% CPU.No activity possible on it.All querries hanging.Any suggestions ?
Please look at patch name EM_NT_1224539.
null -
InterMedia search within report components causing Oracle to run at 100%cpu
Hi,
Running Win NT SP5
Oracle 8.1.7.1
IAS 1.0.2.2
Portal 3.0.9.8.1
I have a table with an intermedia index.
My report query has :
AND contains(i.description,:b_description) > 0
This appears to be the reason why my application components sometimes run othertime doesnt. It also seems to have caused my CPU to go to 100% .. such that Oracle process taking 100% CPU.
Take out this line.. and everything works fine!
Anyone else experienced this, any fix available ?
matt.Hi,
Running Win NT SP5
Oracle 8.1.7.1
IAS 1.0.2.2
Portal 3.0.9.8.1
I have a table with an intermedia index.
My report query has :
AND contains(i.description,:b_description) > 0
This appears to be the reason why my application components sometimes run othertime doesnt. It also seems to have caused my CPU to go to 100% .. such that Oracle process taking 100% CPU.
Take out this line.. and everything works fine!
Anyone else experienced this, any fix available ?
matt. -
Some Oracle processes are consuming 100% CPU
Hi,
Some of my oracle processes are consuming most amount of cpu continuously.What are the general steps to sort out this problem.I think the query is a tuned one.Hi,
What is your veriosn of Oracle? what is the OS? which oracle process is consuming 100% CPU? how did u find out it is conuming?
and you have mentiond query is tuned one... it does your query is taking time to execute?
there is no clarity in your question...
Regards,
Vijayaraghavan K -
Oracle.exe is consuming 100% cpu o windows
Hi,
We have 6 oracle 10g database on windows box and from task manager we are able to see oracle.exe is consuming 100% cpu
Need to know this oracle.exe belongs to which database and can i get the sql for this process
ThanksOn my Vista PC, I find that the easiest way to identify which instance an oracle.exe is, is the task manager. On the Processes tab, right-click the oracle.exe and then click Go To Service(s) and that takes me to the OracleServiceORCL or whichever insance it is.
I'm sure there is a better way that a Windows person could tell us about. Then having identified the instance, it is just normal tuning from there. -
100% CPU Usage Overhead running EM DBConsole 11g on OEL-5.2
After upgrading to OEL-5.2 and relinking all Oracle binaries, my old Oracle 11g installation, installed several months before on OEL-5.1, has been working well, including Enterprise Manager Database Console working nicely as always with respectful performance. Unfortunatelly, it lasted just several days.
Yesterday I decided to uninstall the 11g completely and perform new clean installation (software and database) with the same configuration options and settings as before, including EM dbconsole, all configured using dbca. After completing the installation (EM was started automatically by dbca), oracle continued to suck 80-85% CPU time. In further few minutes CPU utilization raised up to 99% due to only one (always the same PID) client process - "oracleorcl (LOCAL=NO)". For first ten minutes I didn't care too much since I always enable Automatic Management in dbca. But after two hours, I started to worry. The process was still running, consuming sustained 99% of CPU power. No other system activity, no database activity, no disks activity at all!
I was really puzzled since I installed and reinstalled the 11g at least 20 times on OEL-5.0 and 5.1, experimenting with ASM, raw devices, loopback devices and various combinations of installation options, but never experienced such a behaviour. It took me 3 minutes to log in to EM dbconsole as it was almost unusable performing too slow. After three hours CPU temperature was nearly 60 degrees celsius. I decided to shutdown EM and after that everything became quiet. Oracle was running normally. Started EM again, the problem was back again. Tracing enabled, it filled a 350 MB trace file in just 20 minutes. Reinstalling the software and database once again didn't help. Whenever EM is up, the CPU usage overhead of 99% persists.
Here is a cca 23 minutes session summary report taken from EM dbconsole's Performance page. The trace file is too big to list it here, but it shows the same.
Host CPU: 100%
Active Sessions: 100%The details for the Selected 5 Minute Interval (the last 5 min interval) are shown as follow:
TOP SESSIONS: SYSMAN, Program: OMS
Activity: 100%
TOP MODULES: OEM.CacheModeWaitPool, Service: orcl
Activity: 100%
TOP CLIENT: Unnamed
Activity: 99.1%
TOP ACTIONS: Unnamed (OEM.CacheModeWaitPool) (orcl)
Activity: 100%
TOP OBJECTS: SYSMAN.MGMT_JOB_EXEC_SUMMARY (Table)
Activity: 100%
TOP PL/SQL: SYSMAN.MGMT_JOB_ENGINE.INSERT_EXECUTION
PL/SQL Source: SYSMAN.MGMT_JOB_ENGINE
Line Number: 7135
Activity: 100%
TOP SQL: SELECT EXECUTION_ID, STATUS, STATUS_DETAIL FROM MGMT_JOB_EXEC_SUMMARY
WHERE JOB_ID = :B3 AND TARGET_LIST_INDEX = :B2 AND EXPECTED_START_TIME = :B1;
Activity: 100%
STATISTICS SUMMARY
cca 23 minutes session
with no other system activity
Per
Total Execution Per Row
Executions 105,103 1 10,510.30
Elapsed Time (sec) 1,358.95 0.01 135.90
CPU Time (sec) 1,070.42 0.01 107.04
Buffer Gets 85,585,518 814.30 8,558,551.80
Disk Reads 2 <0.01 0.20
Direct Writes 0 0.00 0.00
Rows 10 <0.01 1
Fetches 105,103 1.00 10,510.30
----------------------------------------Wow!!! Note: no disk, no database activity !
Has anyone experienced this or similar behaviour after clean 11g installation on OEL-5.2? If not, anyone has a clue what the hell is going on?
Thanks in advance.Hi Tommy,
I didn't want to experiment further with already working OEL-5.2, oracle and dbconsole on this machine, specially not after googling the problem and finding out that I am not alone in this world. There are another two threads on OTN forums (Database General) showing the same problem even on 2GB machines:
DBConsole easting a CPU
11g stuck. 50-100% CPU after fresh install
So, I took another, a smaller free machine I've got at home (1GB RAM, 2.2MHz Pentium4, three 80GB disks), on which I used to experiment with new releases of software (this is the machine on which I installed 11g for the first time when it was released on OEL-5.0, and I can recall that everything was OK with EM). This is what I did:
1. I installed OEL-5.0 on the machine, adjusted linux and kernel parameters, and performed full 11g installation. Database and EM dbconsole worked nice with acceptable performance. Without activity in the database, %CPU = zero !!! The whole system was perfectly quiet.
2. Since everything was OK, I shutdown EM and oracle, and performed the full upgrade to OEL-5.2. When the upgrade finished, restarted the system, relinked all oracle binaries, and started oracle and EM dbconsole. Both worked perfectly again, just as before the upgrade. I repeated restarting the database and dbconsole several times, always with the same result - it really rocks. Without database activity, %CPU = zero%.
3. Using dbca, I dropped the database and created the new one with the same configuration options. Wow! I'm again in trouble. A half an hour after the creation of the database, %CPU raised up to 99%. That's it.
The crucial question here is: what is that in OEL-5.2, not existing in the 5.0, that causes dbca/em scripts to be embarrassed at the time of EM agent configuration?
Here are the outputs you required picked 30 minutes after starting the database and EM dbconsole (sustained 99% CPU utilization). Note that this is just a 1GB machine.
Kernel command line: ro root=LABEL=/ elevator=deadline rhgb quiet
[root@localhost ~]# cat /proc/meminfo
MemTotal: 1034576 kB
MemFree: 27356 kB
Buffers: 8388 kB
Cached: 609660 kB
SwapCached: 18628 kB
Active: 675376 kB
Inactive: 287072 kB
HighTotal: 130304 kB
HighFree: 260 kB
LowTotal: 904272 kB
LowFree: 27096 kB
SwapTotal: 3148700 kB
SwapFree: 2940636 kB
Dirty: 72 kB
Writeback: 0 kB
AnonPages: 328700 kB
Mapped: 271316 kB
Slab: 21136 kB
PageTables: 14196 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 3665988 kB
Committed_AS: 1187464 kB
VmallocTotal: 114680 kB
VmallocUsed: 5860 kB
VmallocChunk: 108476 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
Hugepagesize: 4096 kB
[root@localhost ~]# cat /proc/slabinfo
slabinfo - version: 2.1
# name <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
rpc_buffers 8 8 2048 2 1 : tunables 24 12 8 : slabdata 4 4 0
rpc_tasks 8 15 256 15 1 : tunables 120 60 8 : slabdata 1 1 0
rpc_inode_cache 6 7 512 7 1 : tunables 54 27 8 : slabdata 1 1 0
ip_conntrack_expect 0 0 96 40 1 : tunables 120 60 8 : slabdata 0 0 0
ip_conntrack 68 68 228 17 1 : tunables 120 60 8 : slabdata 4 4 0
ip_fib_alias 7 113 32 113 1 : tunables 120 60 8 : slabdata 1 1 0
ip_fib_hash 7 113 32 113 1 : tunables 120 60 8 : slabdata 1 1 0
fib6_nodes 22 113 32 113 1 : tunables 120 60 8 : slabdata 1 1 0
ip6_dst_cache 13 15 256 15 1 : tunables 120 60 8 : slabdata 1 1 0
ndisc_cache 1 15 256 15 1 : tunables 120 60 8 : slabdata 1 1 0
RAWv6 4 5 768 5 1 : tunables 54 27 8 : slabdata 1 1 0
UDPv6 9 12 640 6 1 : tunables 54 27 8 : slabdata 2 2 0
tw_sock_TCPv6 0 0 128 30 1 : tunables 120 60 8 : slabdata 0 0 0
request_sock_TCPv6 0 0 128 30 1 : tunables 120 60 8 : slabdata 0 0 0
TCPv6 1 3 1280 3 1 : tunables 24 12 8 : slabdata 1 1 0
jbd_1k 0 0 1024 4 1 : tunables 54 27 8 : slabdata 0 0 0
dm_mpath 0 0 28 127 1 : tunables 120 60 8 : slabdata 0 0 0
dm_uevent 0 0 2460 3 2 : tunables 24 12 8 : slabdata 0 0 0
dm_tio 0 0 16 203 1 : tunables 120 60 8 : slabdata 0 0 0
dm_io 0 0 20 169 1 : tunables 120 60 8 : slabdata 0 0 0
jbd_4k 1 1 4096 1 1 : tunables 24 12 8 : slabdata 1 1 0
scsi_cmd_cache 10 10 384 10 1 : tunables 54 27 8 : slabdata 1 1 0
sgpool-128 36 36 2048 2 1 : tunables 24 12 8 : slabdata 18 18 0
sgpool-64 33 36 1024 4 1 : tunables 54 27 8 : slabdata 9 9 0
sgpool-32 34 40 512 8 1 : tunables 54 27 8 : slabdata 5 5 0
sgpool-16 35 45 256 15 1 : tunables 120 60 8 : slabdata 3 3 0
sgpool-8 60 60 128 30 1 : tunables 120 60 8 : slabdata 2 2 0
scsi_io_context 0 0 104 37 1 : tunables 120 60 8 : slabdata 0 0 0
ext3_inode_cache 4376 8216 492 8 1 : tunables 54 27 8 : slabdata 1027 1027 0
ext3_xattr 165 234 48 78 1 : tunables 120 60 8 : slabdata 3 3 0
journal_handle 8 169 20 169 1 : tunables 120 60 8 : slabdata 1 1 0
journal_head 684 1008 52 72 1 : tunables 120 60 8 : slabdata 14 14 0
revoke_table 18 254 12 254 1 : tunables 120 60 8 : slabdata 1 1 0
revoke_record 0 0 16 203 1 : tunables 120 60 8 : slabdata 0 0 0
uhci_urb_priv 0 0 28 127 1 : tunables 120 60 8 : slabdata 0 0 0
UNIX 56 112 512 7 1 : tunables 54 27 8 : slabdata 16 16 0
flow_cache 0 0 128 30 1 : tunables 120 60 8 : slabdata 0 0 0
cfq_ioc_pool 0 0 92 42 1 : tunables 120 60 8 : slabdata 0 0 0
cfq_pool 0 0 96 40 1 : tunables 120 60 8 : slabdata 0 0 0
crq_pool 0 0 44 84 1 : tunables 120 60 8 : slabdata 0 0 0
deadline_drq 140 252 44 84 1 : tunables 120 60 8 : slabdata 3 3 0
as_arq 0 0 56 67 1 : tunables 120 60 8 : slabdata 0 0 0
mqueue_inode_cache 1 6 640 6 1 : tunables 54 27 8 : slabdata 1 1 0
isofs_inode_cache 0 0 368 10 1 : tunables 54 27 8 : slabdata 0 0 0
hugetlbfs_inode_cache 1 11 340 11 1 : tunables 54 27 8 : slabdata 1 1 0
ext2_inode_cache 0 0 476 8 1 : tunables 54 27 8 : slabdata 0 0 0
ext2_xattr 0 0 48 78 1 : tunables 120 60 8 : slabdata 0 0 0
dnotify_cache 2 169 20 169 1 : tunables 120 60 8 : slabdata 1 1 0
dquot 0 0 128 30 1 : tunables 120 60 8 : slabdata 0 0 0
eventpoll_pwq 1 101 36 101 1 : tunables 120 60 8 : slabdata 1 1 0
eventpoll_epi 1 30 128 30 1 : tunables 120 60 8 : slabdata 1 1 0
inotify_event_cache 1 127 28 127 1 : tunables 120 60 8 : slabdata 1 1 0
inotify_watch_cache 23 92 40 92 1 : tunables 120 60 8 : slabdata 1 1 0
kioctx 135 135 256 15 1 : tunables 120 60 8 : slabdata 9 9 0
kiocb 0 0 128 30 1 : tunables 120 60 8 : slabdata 0 0 0
fasync_cache 0 0 16 203 1 : tunables 120 60 8 : slabdata 0 0 0
shmem_inode_cache 553 585 436 9 1 : tunables 54 27 8 : slabdata 65 65 0
posix_timers_cache 0 0 88 44 1 : tunables 120 60 8 : slabdata 0 0 0
uid_cache 5 59 64 59 1 : tunables 120 60 8 : slabdata 1 1 0
ip_mrt_cache 0 0 128 30 1 : tunables 120 60 8 : slabdata 0 0 0
tcp_bind_bucket 32 203 16 203 1 : tunables 120 60 8 : slabdata 1 1 0
inet_peer_cache 1 59 64 59 1 : tunables 120 60 8 : slabdata 1 1 0
secpath_cache 0 0 32 113 1 : tunables 120 60 8 : slabdata 0 0 0
xfrm_dst_cache 0 0 384 10 1 : tunables 54 27 8 : slabdata 0 0 0
ip_dst_cache 6 15 256 15 1 : tunables 120 60 8 : slabdata 1 1 0
arp_cache 2 15 256 15 1 : tunables 120 60 8 : slabdata 1 1 0
RAW 2 7 512 7 1 : tunables 54 27 8 : slabdata 1 1 0
UDP 3 7 512 7 1 : tunables 54 27 8 : slabdata 1 1 0
tw_sock_TCP 3 30 128 30 1 : tunables 120 60 8 : slabdata 1 1 0
request_sock_TCP 4 30 128 30 1 : tunables 120 60 8 : slabdata 1 1 0
TCP 43 49 1152 7 2 : tunables 24 12 8 : slabdata 7 7 0
blkdev_ioc 3 127 28 127 1 : tunables 120 60 8 : slabdata 1 1 0
blkdev_queue 23 24 956 4 1 : tunables 54 27 8 : slabdata 6 6 0
blkdev_requests 137 161 172 23 1 : tunables 120 60 8 : slabdata 7 7 0
biovec-256 7 8 3072 2 2 : tunables 24 12 8 : slabdata 4 4 0
biovec-128 7 10 1536 5 2 : tunables 24 12 8 : slabdata 2 2 0
biovec-64 7 10 768 5 1 : tunables 54 27 8 : slabdata 2 2 0
biovec-16 7 15 256 15 1 : tunables 120 60 8 : slabdata 1 1 0
biovec-4 8 59 64 59 1 : tunables 120 60 8 : slabdata 1 1 0
biovec-1 406 406 16 203 1 : tunables 120 60 8 : slabdata 2 2 300
bio 564 660 128 30 1 : tunables 120 60 8 : slabdata 21 22 204
utrace_engine_cache 0 0 32 113 1 : tunables 120 60 8 : slabdata 0 0 0
utrace_cache 0 0 32 113 1 : tunables 120 60 8 : slabdata 0 0 0
sock_inode_cache 149 230 384 10 1 : tunables 54 27 8 : slabdata 23 23 0
skbuff_fclone_cache 20 20 384 10 1 : tunables 54 27 8 : slabdata 2 2 0
skbuff_head_cache 86 210 256 15 1 : tunables 120 60 8 : slabdata 14 14 0
file_lock_cache 22 40 96 40 1 : tunables 120 60 8 : slabdata 1 1 0
Acpi-Operand 1147 1196 40 92 1 : tunables 120 60 8 : slabdata 13 13 0
Acpi-ParseExt 0 0 44 84 1 : tunables 120 60 8 : slabdata 0 0 0
Acpi-Parse 0 0 28 127 1 : tunables 120 60 8 : slabdata 0 0 0
Acpi-State 0 0 44 84 1 : tunables 120 60 8 : slabdata 0 0 0
Acpi-Namespace 615 676 20 169 1 : tunables 120 60 8 : slabdata 4 4 0
delayacct_cache 233 312 48 78 1 : tunables 120 60 8 : slabdata 4 4 0
taskstats_cache 12 53 72 53 1 : tunables 120 60 8 : slabdata 1 1 0
proc_inode_cache 622 693 356 11 1 : tunables 54 27 8 : slabdata 63 63 0
sigqueue 8 27 144 27 1 : tunables 120 60 8 : slabdata 1 1 0
radix_tree_node 6220 8134 276 14 1 : tunables 54 27 8 : slabdata 581 581 0
bdev_cache 37 42 512 7 1 : tunables 54 27 8 : slabdata 6 6 0
sysfs_dir_cache 4980 4992 48 78 1 : tunables 120 60 8 : slabdata 64 64 0
mnt_cache 36 60 128 30 1 : tunables 120 60 8 : slabdata 2 2 0
inode_cache 1113 1254 340 11 1 : tunables 54 27 8 : slabdata 114 114 81
dentry_cache 11442 18560 136 29 1 : tunables 120 60 8 : slabdata 640 640 180
filp 7607 10000 192 20 1 : tunables 120 60 8 : slabdata 500 500 120
names_cache 19 19 4096 1 1 : tunables 24 12 8 : slabdata 19 19 0
avc_node 14 72 52 72 1 : tunables 120 60 8 : slabdata 1 1 0
selinux_inode_security 814 1170 48 78 1 : tunables 120 60 8 : slabdata 15 15 0
key_jar 14 30 128 30 1 : tunables 120 60 8 : slabdata 1 1 0
idr_layer_cache 170 203 136 29 1 : tunables 120 60 8 : slabdata 7 7 0
buffer_head 38892 39024 52 72 1 : tunables 120 60 8 : slabdata 542 542 0
mm_struct 108 135 448 9 1 : tunables 54 27 8 : slabdata 15 15 0
vm_area_struct 11169 14904 84 46 1 : tunables 120 60 8 : slabdata 324 324 144
fs_cache 82 177 64 59 1 : tunables 120 60 8 : slabdata 3 3 0
files_cache 108 140 384 10 1 : tunables 54 27 8 : slabdata 14 14 0
signal_cache 142 171 448 9 1 : tunables 54 27 8 : slabdata 19 19 0
sighand_cache 127 135 1344 3 1 : tunables 24 12 8 : slabdata 45 45 0
task_struct 184 246 1360 3 1 : tunables 24 12 8 : slabdata 82 82 0
anon_vma 3313 5842 12 254 1 : tunables 120 60 8 : slabdata 23 23 0
pgd 84 84 4096 1 1 : tunables 24 12 8 : slabdata 84 84 0
pid 237 303 36 101 1 : tunables 120 60 8 : slabdata 3 3 0
size-131072(DMA) 0 0 131072 1 32 : tunables 8 4 0 : slabdata 0 0 0
size-131072 0 0 131072 1 32 : tunables 8 4 0 : slabdata 0 0 0
size-65536(DMA) 0 0 65536 1 16 : tunables 8 4 0 : slabdata 0 0 0
size-65536 2 2 65536 1 16 : tunables 8 4 0 : slabdata 2 2 0
size-32768(DMA) 0 0 32768 1 8 : tunables 8 4 0 : slabdata 0 0 0
size-32768 9 9 32768 1 8 : tunables 8 4 0 : slabdata 9 9 0
size-16384(DMA) 0 0 16384 1 4 : tunables 8 4 0 : slabdata 0 0 0
size-16384 6 6 16384 1 4 : tunables 8 4 0 : slabdata 6 6 0
size-8192(DMA) 0 0 8192 1 2 : tunables 8 4 0 : slabdata 0 0 0
size-8192 5 5 8192 1 2 : tunables 8 4 0 : slabdata 5 5 0
size-4096(DMA) 0 0 4096 1 1 : tunables 24 12 8 : slabdata 0 0 0
size-4096 205 205 4096 1 1 : tunables 24 12 8 : slabdata 205 205 0
size-2048(DMA) 0 0 2048 2 1 : tunables 24 12 8 : slabdata 0 0 0
size-2048 260 270 2048 2 1 : tunables 24 12 8 : slabdata 135 135 0
size-1024(DMA) 0 0 1024 4 1 : tunables 54 27 8 : slabdata 0 0 0
size-1024 204 204 1024 4 1 : tunables 54 27 8 : slabdata 51 51 0
size-512(DMA) 0 0 512 8 1 : tunables 54 27 8 : slabdata 0 0 0
size-512 367 464 512 8 1 : tunables 54 27 8 : slabdata 58 58 0
size-256(DMA) 0 0 256 15 1 : tunables 120 60 8 : slabdata 0 0 0
size-256 487 495 256 15 1 : tunables 120 60 8 : slabdata 33 33 0
size-128(DMA) 0 0 128 30 1 : tunables 120 60 8 : slabdata 0 0 0
size-128 2242 2490 128 30 1 : tunables 120 60 8 : slabdata 83 83 0
size-64(DMA) 0 0 64 59 1 : tunables 120 60 8 : slabdata 0 0 0
size-32(DMA) 0 0 32 113 1 : tunables 120 60 8 : slabdata 0 0 0
size-64 1409 2950 64 59 1 : tunables 120 60 8 : slabdata 50 50 0
size-32 3596 3842 32 113 1 : tunables 120 60 8 : slabdata 34 34 0
kmem_cache 145 150 256 15 1 : tunables 120 60 8 : slabdata 10 10 0
[root@localhost ~]# slabtop -d 5
Active / Total Objects (% used) : 97257 / 113249 (85.9%)
Active / Total Slabs (% used) : 4488 / 4488 (100.0%)
Active / Total Caches (% used) : 101 / 146 (69.2%)
Active / Total Size (% used) : 15076.34K / 17587.55K (85.7%)
Minimum / Average / Maximum Object : 0.01K / 0.16K / 128.00K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
25776 25764 99% 0.05K 358 72 1432K buffer_head
16146 15351 95% 0.08K 351 46 1404K vm_area_struct
15138 7779 51% 0.13K 522 29 2088K dentry_cache
9720 9106 93% 0.19K 486 20 1944K filp
7714 7032 91% 0.27K 551 14 2204K radix_tree_node
5070 5018 98% 0.05K 65 78 260K sysfs_dir_cache
4826 4766 98% 0.01K 19 254 76K anon_vma
4824 3406 70% 0.48K 603 8 2412K ext3_inode_cache
3842 3691 96% 0.03K 34 113 136K size-32
2190 2174 99% 0.12K 73 30 292K size-128
1711 1364 79% 0.06K 29 59 116K size-64
1210 1053 87% 0.33K 110 11 440K inode_cache
1196 1147 95% 0.04K 13 92 52K Acpi-Operand
1170 814 69% 0.05K 15 78 60K selinux_inode_security
936 414 44% 0.05K 13 72 52K journal_head
747 738 98% 0.43K 83 9 332K shmem_inode_cache
693 617 89% 0.35K 63 11 252K proc_inode_cache
676 615 90% 0.02K 4 169 16K Acpi-Namespace
609 136 22% 0.02K 3 203 12K biovec-1
495 493 99% 0.25K 33 15 132K size-256
480 384 80% 0.12K 16 30 64K bio
440 399 90% 0.50K 55 8 220K size-512
312 206 66% 0.05K 4 78 16K delayacct_cache
303 209 68% 0.04K 3 101 12K pid
290 290 100% 0.38K 29 10 116K sock_inode_cache
[root@localhost ~]# cat /etc/sysctl.conf
# Kernel sysctl configuration file for Red Hat Linux
# Controls IP packet forwarding
net.ipv4.ip_forward=0
# Controls source route verification
net.ipv4.conf.default.rp_filter=1
# Do not accept source routing
net.ipv4.conf.default.accept_source_route=0
# Oracle
net.ipv4.ip_local_port_range=1024 65000
net.core.rmem_default=4194304
net.core.rmem_max=4194304
net.core.wmem_default=262144
net.core.wmem_max=262144
net.ipv4.tcp_rmem=4096 65536 4194304
net.ipv4.tcp_wmem=4096 65536 4194304
# Keepalive Oracle
net.ipv4.tcp_keepalive_time=3000
net.ipv4.tcp_keepalive_intvl=30
net.ipv4.tcp_keepalive_probes=15
net.ipv4.tcp_retries2=3
net.ipv4.tcp_syn_retries=2
net.ipv4.tcp_sack=0
net.ipv4.tcp_timestamps=0
net.ipv4.tcp_window_scaling=0
# Oracle
fs.file-max = 6553600
fs.aio-max-nr=3145728
kernel.shmmni=4096
kernel.sem=250 32000 100 142
kernel.shmmax=2147483648
kernel.shmall=3279547
kernel.msgmnb=65536
kernel.msgmni=2878
kernel.msgmax=8192
kernel.exec-shield=0
# Controls the System Request debugging functionality of the kernel
kernel.sysrq=1
kernel.panic=60
kernel.core_uses_pid=1
[root@localhost ~]# free | grep Swap
Swap: 3148700 319916 2828784
[root@localhost ~]# cat /etc/fstab | grep "/dev/shm"
tmpfs /dev/shm tmpfs size=1024M 0 0
[root@localhost ~]# df | grep "/dev/shm"
tmpfs 1048576 452128 596448 44% /dev/shm
NON-DEFAULT DB PARAMETERS:
db_block_size 8192
memory_target 633339904 /* automatic memory management */
open_cursors 300
processes 256
disk_async_io TRUE
filesystemio_options SETALL -
Sessions with Event "class slave wait" taking 100% CPU
Hi experts,
I have an issue in an 11.2.0.2 RAC database with 2 standby nodes that are driving me nuts.
All started when I've been told there are 2 Oracle processes consuming too much CPU in the environment.
I found 2 processes which takes 100% CPU each. These processes are Oracle processes into the DB and checking them out although they are registered in v$process and v$session, there is nothing related to them in v$bgprocess so I could not find out which oracle processes started these 2 OS processes.
The view v$session shows them ACTIVE in the event "class slave wait". There aren't too much information about this event, at least I failed to find...
Using dbms_monitor and dbms_system to create trace events did not create any tracefile. Only ORADEBUG was able to create events but with nearly no information inside the trace files.
Questions I need to have answered or at least hints that I can follow to find the answer:
1) what is causing the CPUs to be consumed at 100%? Which process?
2) Why does these processes are using that much of CPU?
3) What can be done to safely get over it?
Honestly I don't know where else to look at except trying to get some help. Could someone give me a hand please?
BR,
Lauro Ojeda
Edited by: LauroOjeda on 26/01/2013 06:44Hi Pal, thanks for your reply!
Here are the answers:
1. If it is RAC there is no such construct as a "standby node" and you say you have two of them. Please be specific ... is this RAC or Data Guard or a combination? Or do you have a three node cluster with all services pointing to only a single server wasting all of the resources of the other three?
A: This is a combination of them. We have 2 nodes in a RAC environment shipping logs to two physical standby databases in another site.
2. Two processes can not be at 100% of CPU any more than I can be 200% awake. Please show us how you arrived at this conclusion, on what hardware and operating system, and include a formatted (read the FAQ to learn how) extract showing what you are seeing.
A: top in Linux shows two oracle processes consuming 100% (or nearly of it) of 1 CPU each. This is a 16 cores server (each primary node) so 2 of these are working on full capacity to service the described processes.
3. What processes? Name them and again run a SQL statement and post the output so we can see what you are looking at.
A: Like I said before I failed to find which background process they belongs to as there are no indications of them in v$bgprocess and in v$session/v$process either. I can see them in v$process and v$session but again, there are no indications of for which bg process they belong to.
4. Is anything slow? Are there any problems with the system or are you only reacting to two numbers you think are too high?
A: No, system is not slow because of that but the client wants to investigate and have it sorted as it is not normal.
Additionally I found a bug in metalink which seems to be the culprit, but I'm unsure yet. This is Bug 12929268 : HIGH CPU ON ORA_O00N PROCESS
Is any other information that I may provide you?
Thanks for help!
Lauro Ojeda -
Oracle consumes 100% memory on Solaris 10
Hi,
Our database (Oracle 10g, R2) is running is running on Solaris, When I use the unix command prstat -a, it shows that, 100% Memomry utilized. Below are the details.
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
12934 oracle 2573M 2559M sleep 59 0 0:00:00 0.1% oracle/1
5914 sirsi 4912K 4664K sleep 59 0 0:00:29 0.0% prstat/1
12937 oracle 4896K 4592K cpu3 49 0 0:00:00 0.0% prstat/1
833 oracle 2572M 2558M sleep 59 0 0:01:05 0.0% oracle/1
114 root 7464K 6632K sleep 59 0 0:01:20 0.0% picld/12
829 oracle 2573M 2559M sleep 59 0 0:01:04 0.0% oracle/1
823 oracle 2574M 2560M sleep 59 0 0:00:46 0.0% oracle/11
811 oracle 2573M 2559M sleep 59 0 0:00:43 0.0% oracle/1
146 root 2288K 1312K sleep 59 0 0:00:22 0.0% in.mpathd/1
831 oracle 2576M 2562M sleep 59 0 0:00:24 0.0% oracle/1
639 root 3664K 2392K sleep 59 0 0:00:00 0.0% snmpXdmid/2
700 nobody 7520K 3752K sleep 59 0 0:00:00 0.0% httpd/1
701 nobody 7520K 3752K sleep 59 0 0:00:00 0.0% httpd/1
637 root 3080K 2048K sleep 59 0 0:00:00 0.0% dmispd/1
472 root 5232K 2320K sleep 59 0 0:00:00 0.0% dtlogin/1
720 root 2912K 2400K sleep 59 0 0:00:01 0.0% vold/5
629 root 2376K 1664K sleep 59 0 0:00:00 0.0% snmpdx/1
702 nobody 7520K 3736K sleep 59 0 0:00:00 0.0% httpd/1
378 root 3928K 1784K sleep 59 0 0:00:00 0.0% sshd/1
699 nobody 7520K 3704K sleep 59 0 0:00:00 0.0% httpd/1
697 root 9384K 6520K sleep 59 0 0:00:01 0.0% snmpd/1
695 root 7360K 5376K sleep 59 0 0:00:04 0.0% httpd/1
375 root 12M 8088K sleep 59 0 0:00:01 0.0% fmd/15
354 root 3728K 2040K sleep 59 0 0:00:00 0.0% syslogd/13
415 root 2016K 1440K sleep 59 0 0:00:00 0.0% smcboot/1
416 root 2008K 1016K sleep 59 0 0:00:00 0.0% smcboot/1
338 root 4736K 1296K sleep 59 0 0:00:00 0.0% automountd/2
340 root 5080K 2384K sleep 59 0 0:00:00 0.0% automountd/3
263 daemon 2384K 1760K sleep 60 -20 0:00:00 0.0% lockd/2
256 root 1280K 936K sleep 59 0 0:00:00 0.0% utmpd/1
395 root 7592K 2560K sleep 59 0 0:00:02 0.0% sendmail/1
273 root 2232K 1496K sleep 59 0 0:00:00 0.0% ttymon/1
254 root 2072K 1224K sleep 59 0 0:00:00 0.0% sf880drd/1
417 root 2008K 1016K sleep 59 0 0:00:00 0.0% smcboot/1
272 root 5152K 4016K sleep 59 0 0:00:02 0.0% inetd/4
206 root 1232K 536K sleep 59 0 0:00:00 0.0% efdaemon/1
394 smmsp 7568K 1904K sleep 59 0 0:00:00 0.0% sendmail/1
128 root 2904K 2056K sleep 59 0 0:00:00 0.0% devfsadm/6
241 daemon 2640K 1528K sleep 59 0 0:00:00 0.0% rpcbind/1
245 daemon 2672K 1992K sleep 59 0 0:00:00 0.0% statd/1
251 root 2000K 1248K sleep 59 0 0:00:00 0.0% sac/1
123 root 3992K 3008K sleep 59 0 0:00:07 0.0% nscd/26
NPROC USERNAME SIZE RSS MEMORY TIME CPU
24 oracle 48G 48G 100% 0:04:48 0.1%
10 sirsi 1101M 35M 0.1% 0:00:32 0.0%
37 root 148M 97M 0.2% 0:02:18 0.0%
10 nobody 73M 36M 0.1% 0:00:00 0.0%
1 smmsp 7568K 1904K 0.0% 0:00:00 0.0%
4 daemon 12M 7920K 0.0% 0:00:00 0.0%
Total: 86 processes, 260 lwps, load averages: 0.02, 0.02, 0.02
Can anyone suggest why Oracle consumes 100% Memory? and how do we resolve this?.
Regards,
Sabdar Syed.Many Unix tools add to each dedicated server process memory the SGA size because under Unix each dedicated server is attaching the SGA shared memory segment to its process address space: so these Unix tools are not so reliable for Oracle.
To check the Oracle memory usage, it is generally more recommendded to use the V$ views such as V$SGASTAT and V$PGASTAT. -
Weblogic reaches 100% CPU utilisation
I am using WebLogic Server Version 10.3.1.0. I got an application deployed in this server, which is developed with java1.6, ejb, hibernate. I start the server with 8GB of RAM assigned to it and after a day or two the weblogic server causes the use of the CPU to reach to 100% and as a result the server sort of doesn't response anymore. I will be creating 30-40 thread per managed server and one server has two managed server. The volume will be around 30k per day. I dont know which is causing the 100% cpu utilisation and here below are the thread dumps which have taken at the time of high cpu in a server. Please let me know where is the exact issue and where is the stuck thread (if any)? In which part the issue is occuring?
10774:
===== FULL THREAD DUMP ===============
Thu Jun 19 16:57:39 2014
Oracle JRockit(R) R28.2.8-10-156881-1.6.0_51-20130611-1146-linux-x86_64
"Main Thread" id=1 idx=0x4 tid=10776 prio=5 alive, waiting, native_blocked
-- Waiting for notification on: weblogic/t3/srvr/T3Srvr@0xa4db4c58[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at weblogic/t3/srvr/T3Srvr.waitForDeath(T3Srvr.java:981)
^-- Lock released while waiting: weblogic/t3/srvr/T3Srvr@0xa4db4c58[fat lock]
at weblogic/t3/srvr/T3Srvr.run(T3Srvr.java:490)
at weblogic/Server.main(Server.java:71)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"(Signal Handler)" id=2 idx=0x8 tid=10777 prio=5 alive, native_blocked, daemon
"(OC Main Thread)" id=3 idx=0xc tid=10778 prio=5 alive, native_waiting, daemon
"(GC Worker Thread 1)" id=? idx=0x10 tid=10779 prio=5 alive, daemon
"(GC Worker Thread 2)" id=? idx=0x14 tid=10780 prio=5 alive, daemon
"(GC Worker Thread 3)" id=? idx=0x18 tid=10781 prio=5 alive, daemon
"(GC Worker Thread 4)" id=? idx=0x1c tid=10782 prio=5 alive, daemon
"(Code Generation Thread 1)" id=4 idx=0x20 tid=10783 prio=5 alive, native_waiting, daemon
"(Code Optimization Thread 1)" id=5 idx=0x24 tid=10784 prio=5 alive, native_waiting, daemon
"(VM Periodic Task)" id=6 idx=0x28 tid=10785 prio=10 alive, native_waiting, daemon
"Finalizer" id=7 idx=0x2c tid=10786 prio=8 alive, native_waiting, daemon
at jrockit/memory/Finalizer.waitForFinalizees(J[Ljava/lang/Object;)I(Native Method)
at jrockit/memory/Finalizer.access$700(Finalizer.java:12)
at jrockit/memory/Finalizer$4.run(Finalizer.java:201)
at java/lang/Thread.run(Thread.java:662)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"Reference Handler" id=8 idx=0x30 tid=10787 prio=10 alive, native_waiting, daemon
at java/lang/ref/Reference.waitForActivatedQueue(J)Ljava/lang/ref/Reference;(Native Method)
at java/lang/ref/Reference.access$100(Reference.java:11)
at java/lang/ref/Reference$ReferenceHandler.run(Reference.java:82)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"(Sensor Event Thread)" id=9 idx=0x34 tid=10788 prio=5 alive, native_blocked, daemon
"VM JFR Buffer Thread" id=10 idx=0x38 tid=10789 prio=5 alive, in native, daemon
"Timer-0" id=13 idx=0x3c tid=10793 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: java/util/TaskQueue@0xa4d5d260[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
at java/lang/Object.wait(Object.java:485)[optimized]
at java/util/TimerThread.mainLoop(Timer.java:483)
^-- Lock released while waiting: java/util/TaskQueue@0xa4d5d260[fat lock]
at java/util/TimerThread.run(Timer.java:462)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '0' for queue: 'default'" id=14 idx=0x40 tid=10839 prio=5 alive, native_blocked, daemon
at java/util/HashMap.get(HashMap.java:385)[optimized]
at com/appl/vsms/pi/outbound/ControllerCache.getEntry(ControllerCache.java:48)[inlined]
at com/appl/vsms/bean/pi/MessageStatusConsumer.onMessage(MessageStatusConsumer.java:90)[optimized]
at weblogic/ejb/container/internal/MDListener.execute(MDListener.java:574)[inlined]
at weblogic/ejb/container/internal/MDListener.run(MDListener.java:893)[optimized]
at weblogic/work/ExecuteRequestAdapter.execute(ExecuteRequestAdapter.java:21)[optimized]
at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)[optimized]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '1' for queue: 'default'" id=15 idx=0x44 tid=10840 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: java/util/LinkedList@0xa8cbed08[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
at java/lang/Object.wait(Object.java:485)[inlined]
at weblogic/ejb/container/internal/MDListener.getListener(MDListener.java:240)[optimized]
^-- Lock released while waiting: java/util/LinkedList@0xa8cbed08[fat lock]
at weblogic/ejb/container/internal/MDListener.transactionalOnMessage(MDListener.java:480)[inlined]
at weblogic/ejb/container/internal/MDListener.onMessage(MDListener.java:379)[optimized]
at weblogic/jms/client/JMSSession.onMessage(JMSSession.java:4659)[optimized]
at weblogic/jms/client/JMSSession.execute(JMSSession.java:4345)[optimized]
at weblogic/jms/client/JMSSession.executeMessage(JMSSession.java:3821)[optimized]
at weblogic/jms/client/JMSSession.access$000(JMSSession.java:115)
at weblogic/jms/client/JMSSession$UseForRunnable.run(JMSSession.java:5170)
at weblogic/work/ExecuteRequestAdapter.execute(ExecuteRequestAdapter.java:21)[optimized]
at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)[optimized]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '2' for queue: 'default'" id=16 idx=0x48 tid=10841 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4d5f0d0[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
at java/lang/Object.wait(Object.java:485)[inlined]
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)[optimized]
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4d5f0d0[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '3' for queue: 'default'" id=17 idx=0x4c tid=10842 prio=5 alive, native_blocked, daemon
at java/util/HashMap.get(HashMap.java:385)[optimized]
at com/appl/vsms/pi/outbound/ControllerCache.getEntry(ControllerCache.java:48)[inlined]
at com/appl/vsms/bean/pi/MessageStatusConsumer.onMessage(MessageStatusConsumer.java:90)[optimized]
at weblogic/ejb/container/internal/MDListener.execute(MDListener.java:574)[inlined]
at weblogic/ejb/container/internal/MDListener.run(MDListener.java:893)[optimized]
at weblogic/work/ExecuteRequestAdapter.execute(ExecuteRequestAdapter.java:21)[optimized]
at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)[optimized]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '4' for queue: 'default'" id=18 idx=0x50 tid=10843 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4d5fe50[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
at java/lang/Object.wait(Object.java:485)[inlined]
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)[optimized]
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4d5fe50[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '5' for queue: 'default'" id=19 idx=0x54 tid=10844 prio=5 alive, native_blocked, daemon
at java/util/HashMap.get(HashMap.java:385)[optimized]
at com/appl/vsms/pi/outbound/ControllerCache.getEntry(ControllerCache.java:48)[inlined]
at com/appl/vsms/bean/pi/MessageStatusConsumer.onMessage(MessageStatusConsumer.java:90)[optimized]
at weblogic/ejb/container/internal/MDListener.execute(MDListener.java:574)[inlined]
at weblogic/ejb/container/internal/MDListener.run(MDListener.java:893)[optimized]
at weblogic/work/ExecuteRequestAdapter.execute(ExecuteRequestAdapter.java:21)[optimized]
at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)[optimized]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '6' for queue: 'default'" id=20 idx=0x58 tid=10845 prio=5 alive, native_blocked, daemon
at java/util/HashMap.get(HashMap.java:385)[optimized]
at com/appl/vsms/pi/outbound/ControllerCache.getEntry(ControllerCache.java:48)[inlined]
at com/appl/vsms/bean/pi/MessageStatusConsumer.onMessage(MessageStatusConsumer.java:90)[optimized]
at weblogic/ejb/container/internal/MDListener.execute(MDListener.java:574)[inlined]
at weblogic/ejb/container/internal/MDListener.run(MDListener.java:893)[optimized]
at weblogic/work/ExecuteRequestAdapter.execute(ExecuteRequestAdapter.java:21)[optimized]
at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)[optimized]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '7' for queue: 'default'" id=21 idx=0x5c tid=10846 prio=5 alive, native_blocked, daemon
at java/util/HashMap.get(HashMap.java:385)[optimized]
at com/appl/vsms/pi/outbound/ControllerCache.getEntry(ControllerCache.java:48)[inlined]
at com/appl/vsms/bean/pi/MessageStatusConsumer.onMessage(MessageStatusConsumer.java:90)[optimized]
at weblogic/ejb/container/internal/MDListener.execute(MDListener.java:574)[inlined]
at weblogic/ejb/container/internal/MDListener.run(MDListener.java:893)[optimized]
at weblogic/work/ExecuteRequestAdapter.execute(ExecuteRequestAdapter.java:21)[optimized]
at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)[optimized]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '8' for queue: 'default'" id=22 idx=0x60 tid=10847 prio=5 alive, native_blocked, daemon
at java/util/HashMap.get(HashMap.java:385)[optimized]
at com/appl/vsms/pi/outbound/ControllerCache.getEntry(ControllerCache.java:48)[inlined]
at com/appl/vsms/bean/pi/MessageStatusConsumer.onMessage(MessageStatusConsumer.java:90)[optimized]
at weblogic/ejb/container/internal/MDListener.execute(MDListener.java:574)[inlined]
at weblogic/ejb/container/internal/MDListener.run(MDListener.java:893)[optimized]
at weblogic/work/ExecuteRequestAdapter.execute(ExecuteRequestAdapter.java:21)[optimized]
at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)[optimized]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '9' for queue: 'default'" id=23 idx=0x64 tid=10848 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4d62010[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
at java/lang/Object.wait(Object.java:485)[inlined]
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)[optimized]
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4d62010[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '10' for queue: 'default'" id=24 idx=0x68 tid=10849 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4d626d0[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
at java/lang/Object.wait(Object.java:485)[inlined]
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)[optimized]
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4d626d0[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '11' for queue: 'default'" id=25 idx=0x6c tid=10850 prio=5 alive, native_blocked, daemon
at java/util/HashMap.get(HashMap.java:385)[optimized]
at com/appl/vsms/pi/outbound/ControllerCache.getEntry(ControllerCache.java:48)[inlined]
at com/appl/vsms/bean/pi/MessageStatusConsumer.onMessage(MessageStatusConsumer.java:90)[optimized]
at weblogic/ejb/container/internal/MDListener.execute(MDListener.java:574)[inlined]
at weblogic/ejb/container/internal/MDListener.run(MDListener.java:893)[optimized]
at weblogic/work/ExecuteRequestAdapter.execute(ExecuteRequestAdapter.java:21)[optimized]
at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)[optimized]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '12' for queue: 'default'" id=26 idx=0x70 tid=10851 prio=5 alive, native_blocked, daemon
at java/util/HashMap.get(HashMap.java:385)[optimized]
at com/appl/vsms/pi/outbound/ControllerCache.getEntry(ControllerCache.java:48)[inlined]
at com/appl/vsms/bean/pi/MessageStatusConsumer.onMessage(MessageStatusConsumer.java:90)[optimized]
at weblogic/ejb/container/internal/MDListener.execute(MDListener.java:574)[inlined]
at weblogic/ejb/container/internal/MDListener.run(MDListener.java:893)[optimized]
at weblogic/work/ExecuteRequestAdapter.execute(ExecuteRequestAdapter.java:21)[optimized]
at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)[optimized]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '13' for queue: 'default'" id=27 idx=0x74 tid=10852 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4d648e8[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
at java/lang/Object.wait(Object.java:485)[inlined]
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)[optimized]
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4d648e8[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '14' for queue: 'default'" id=28 idx=0x78 tid=10853 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4d65568[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
at java/lang/Object.wait(Object.java:485)[inlined]
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)[optimized]
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4d65568[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '0' for queue: 'weblogic.kernel.Non-Blocking'" id=29 idx=0x7c tid=10854 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4d65cc0[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
at java/lang/Object.wait(Object.java:485)[inlined]
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)[optimized]
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4d65cc0[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '1' for queue: 'weblogic.kernel.Non-Blocking'" id=30 idx=0x80 tid=10855 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4d663c0[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
at java/lang/Object.wait(Object.java:485)[inlined]
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)[optimized]
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4d663c0[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '2' for queue: 'weblogic.kernel.Non-Blocking'" id=31 idx=0x84 tid=10856 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4d66ac0[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
at java/lang/Object.wait(Object.java:485)[inlined]
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)[optimized]
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4d66ac0[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '0' for queue: 'weblogic.kernel.System'" id=32 idx=0x88 tid=10857 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4d671c0[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
at java/lang/Object.wait(Object.java:485)[inlined]
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)[optimized]
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4d671c0[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '1' for queue: 'weblogic.kernel.System'" id=33 idx=0x8c tid=10858 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4d67f48[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
at java/lang/Object.wait(Object.java:485)[inlined]
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)[optimized]
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4d67f48[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '2' for queue: 'weblogic.kernel.System'" id=34 idx=0x90 tid=10859 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4d68cc0[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
at java/lang/Object.wait(Object.java:485)[inlined]
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)[optimized]
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4d68cc0[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '3' for queue: 'weblogic.kernel.System'" id=35 idx=0x94 tid=10860 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4d69a38[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
at java/lang/Object.wait(Object.java:485)[inlined]
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)[optimized]
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4d69a38[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '4' for queue: 'weblogic.kernel.System'" id=36 idx=0x98 tid=10861 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4d977b8[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
at java/lang/Object.wait(Object.java:485)[inlined]
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)[optimized]
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4d977b8[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '0' for queue: 'weblogic.Rejector'" id=37 idx=0x9c tid=10862 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4d99570[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4d99570[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '1' for queue: 'weblogic.Rejector'" id=38 idx=0xa0 tid=10863 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4cf2e38[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4cf2e38[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"weblogic.time.TimeEventGenerator" id=39 idx=0xa4 tid=10864 prio=9 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/time/common/internal/TimeTable@0xa4dbed18[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at weblogic/time/common/internal/TimeTable.snooze(TimeTable.java:286)
^-- Lock released while waiting: weblogic/time/common/internal/TimeTable@0xa4dbed18[fat lock]
at weblogic/time/common/internal/TimeEventGenerator.run(TimeEventGenerator.java:117)
at java/lang/Thread.run(Thread.java:662)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"JMAPI event thread" id=40 idx=0xa8 tid=10865 prio=5 alive, in native, daemon
"weblogic.timers.TimerThread" id=41 idx=0xac tid=10866 prio=9 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/timers/internal/TimerThread@0xa4dc3f70[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at weblogic/timers/internal/TimerThread$Thread.run(TimerThread.java:262)
^-- Lock released while waiting: weblogic/timers/internal/TimerThread@0xa4dc3f70[fat lock]
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '0' for queue: 'weblogic.admin.RMI'" id=42 idx=0xb0 tid=10867 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4cf3528[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4cf3528[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '1' for queue: 'weblogic.admin.RMI'" id=43 idx=0xb4 tid=10868 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4cf35f0[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4cf35f0[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '2' for queue: 'weblogic.admin.RMI'" id=44 idx=0xb8 tid=10869 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4cf36b8[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
at java/lang/Object.wait(Object.java:485)[inlined]
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)[optimized]
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4cf36b8[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '3' for queue: 'weblogic.admin.RMI'" id=45 idx=0xbc tid=10870 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4cf3780[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
at java/lang/Object.wait(Object.java:485)[inlined]
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)[optimized]
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4cf3780[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '4' for queue: 'weblogic.admin.RMI'" id=46 idx=0xc0 tid=10871 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4cf3848[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
at java/lang/Object.wait(Object.java:485)[inlined]
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)[optimized]
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4cf3848[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '0' for queue: 'ImageWorkManager'" id=47 idx=0xc4 tid=10896 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4cf57e0[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4cf57e0[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '1' for queue: 'ImageWorkManager'" id=48 idx=0xc8 tid=10899 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4cf58a8[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4cf58a8[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"JFR request timer" id=49 idx=0xcc tid=10900 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: java/util/TaskQueue@0xa4dc69d8[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at java/util/TimerThread.mainLoop(Timer.java:483)
^-- Lock released while waiting: java/util/TaskQueue@0xa4dc69d8[fat lock]
at java/util/TimerThread.run(Timer.java:462)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '0' for queue: 'weblogic.logging.DomainLogBroadcasterClient'" id=53 idx=0xd0 tid=10902 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4cf6550[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4cf6550[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '0' for queue: 'weblogic.logging.LogBroadcaster'" id=54 idx=0xd4 tid=10903 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4cf6d50[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4cf6d50[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" id=55 idx=0xd8 tid=10904 prio=5 alive, blocked, native_blocked, daemon
-- Blocked trying to get lock: java/lang/String@0xa4cf78f8[fat lock]
at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)
at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1411)[optimized]
at jrockit/vm/Locks.lockFat(Locks.java:1512)[optimized]
at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1054)[optimized]
at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1005)[optimized]
at jrockit/vm/Locks.monitorEnter(Locks.java:2179)[optimized]
at weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:153)
at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)
at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)
at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" id=56 idx=0xdc tid=10905 prio=5 alive, blocked, native_blocked, daemon
-- Blocked trying to get lock: java/lang/String@0xa4cf78f8[fat lock]
at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)
at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1411)[optimized]
at jrockit/vm/Locks.lockFat(Locks.java:1512)[optimized]
at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1054)[optimized]
at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1005)[optimized]
at jrockit/vm/Locks.monitorEnter(Locks.java:2179)[optimized]
at weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:153)
at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)
at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)
at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" id=57 idx=0xe0 tid=10906 prio=5 alive, blocked, native_blocked, daemon
-- Blocked trying to get lock: java/lang/String@0xa4cf78f8[fat lock]
at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)
at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1411)[inlined]
at jrockit/vm/Locks.fatLockReacquire(Locks.java:1649)[optimized]
at jrockit/vm/Locks.lockFat(Locks.java:1517)[optimized]
at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1054)[optimized]
at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1005)[optimized]
at jrockit/vm/Locks.monitorEnter(Locks.java:2179)[optimized]
at weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:153)
at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)
at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)
at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '3' for queue: 'weblogic.socket.Muxer'" id=58 idx=0xe4 tid=10907 prio=5 alive, in native, daemon
at jrockit/ext/epoll/EPoll.epollWait0(ILjava/nio/ByteBuffer;II)I(Native Method)
at jrockit/ext/epoll/EPoll.epollWait(EPoll.java:123)
at weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:156)
^-- Holding lock: java/lang/String@0xa4cf78f8[fat lock]
at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)
at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)
at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"VDE Transaction Processor Thread" id=61 idx=0xe8 tid=10909 prio=2 alive, waiting, native_blocked, daemon
-- Waiting for notification on: com/octetstring/vde/backend/standard/TransactionProcessor@0xa46c3738[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at com/octetstring/vde/backend/standard/TransactionProcessor.waitChange(TransactionProcessor.java:367)
^-- Lock released while waiting: com/octetstring/vde/backend/standard/TransactionProcessor@0xa46c3738[fat lock]
at com/octetstring/vde/backend/standard/TransactionProcessor.run(TransactionProcessor.java:212)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"DoSManager" id=64 idx=0xf0 tid=10911 prio=6 alive, sleeping, native_waiting, daemon
at java/lang/Thread.sleep(J)V(Native Method)
at com/octetstring/vde/DoSManager.run(DoSManager.java:433)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '0' for queue: 'UserLockout'" id=65 idx=0xf4 tid=10920 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4cf93c8[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4cf93c8[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '1' for queue: 'UserLockout'" id=66 idx=0xf8 tid=10921 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4cf9490[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4cf9490[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '2' for queue: 'UserLockout'" id=67 idx=0xfc tid=10922 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4cf9558[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4cf9558[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '3' for queue: 'UserLockout'" id=68 idx=0x100 tid=10923 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4cf9620[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4cf9620[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '4' for queue: 'UserLockout'" id=69 idx=0x104 tid=10924 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4cf96e8[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4cf96e8[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '0' for queue: 'weblogic.unicast.DispatchWorkManager'" id=70 idx=0x108 tid=10958 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4cfbbb0[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4cfbbb0[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '1' for queue: 'weblogic.unicast.DispatchWorkManager'" id=71 idx=0x10c tid=10959 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4cfbc78[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
at java/lang/Object.wait(Object.java:485)[inlined]
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)[optimized]
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4cfbc78[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '2' for queue: 'weblogic.unicast.DispatchWorkManager'" id=72 idx=0x110 tid=10960 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4cfbd40[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
at java/lang/Object.wait(Object.java:485)[inlined]
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)[optimized]
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4cfbd40[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '3' for queue: 'weblogic.unicast.DispatchWorkManager'" id=73 idx=0x114 tid=10961 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4cfbe08[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
at java/lang/Object.wait(Object.java:485)[inlined]
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)[optimized]
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4cfbe08[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '4' for queue: 'weblogic.unicast.DispatchWorkManager'" id=74 idx=0x118 tid=10962 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4cfbed0[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
at java/lang/Object.wait(Object.java:485)[inlined]
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)[optimized]
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4cfbed0[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '0' for queue: 'weblogic.unicast.ForwardingWorkManager'" id=75 idx=0x11c tid=10963 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4c66f40[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4c66f40[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '1' for queue: 'weblogic.unicast.ForwardingWorkManager'" id=76 idx=0x120 tid=10964 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4c67008[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4c67008[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '2' for queue: 'weblogic.unicast.ForwardingWorkManager'" id=77 idx=0x124 tid=10965 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4c670d0[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4c670d0[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '3' for queue: 'weblogic.unicast.ForwardingWorkManager'" id=78 idx=0x128 tid=10966 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4c67198[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4c67198[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '4' for queue: 'weblogic.unicast.ForwardingWorkManager'" id=79 idx=0x12c tid=10967 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4c67260[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4c67260[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '0' for queue: 'ClusterMessaging'" id=81 idx=0x134 tid=10969 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4c692e0[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
at java/lang/Object.wait(Object.java:485)[inlined]
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)[optimized]
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4c692e0[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '0' for queue: 'wl_oldBootStrap'" id=82 idx=0x138 tid=10972 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4c69a90[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4c69a90[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '1' for queue: 'wl_oldBootStrap'" id=83 idx=0x13c tid=10973 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4c69b58[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4c69b58[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '2' for queue: 'wl_oldBootStrap'" id=84 idx=0x140 tid=10974 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4c69c20[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4c69c20[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '3' for queue: 'wl_oldBootStrap'" id=85 idx=0x144 tid=10975 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4c69ce8[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4c69ce8[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '4' for queue: 'wl_oldBootStrap'" id=86 idx=0x148 tid=10980 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4c69db0[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4c69db0[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '5' for queue: 'wl_oldBootStrap'" id=87 idx=0x14c tid=10984 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4c69e78[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4c69e78[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '6' for queue: 'wl_oldBootStrap'" id=88 idx=0x150 tid=10987 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4c69f40[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4c69f40[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '7' for queue: 'wl_oldBootStrap'" id=89 idx=0x154 tid=10988 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4c6a008[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
at java/lang/Object.wait(Object.java:485)[inlined]
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)[optimized]
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4c6a008[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"Thread-12" id=90 idx=0x158 tid=10989 prio=5 alive, parked, native_blocked
-- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0xa4c66b48
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2230)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at java/util/concurrent/locks/LockSupport.park(LockSupport.java:156)
at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at weblogic/utils/concurrent/JDK15ConcurrentBlockingQueue.take(JDK15ConcurrentBlockingQueue.java:89)
at weblogic/store/internal/PersistentStoreImpl.getOutstandingWork(PersistentStoreImpl.java:672)
at weblogic/store/internal/PersistentStoreImpl.run(PersistentStoreImpl.java:721)
at java/lang/Thread.run(Thread.java:662)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '0' for queue: 'JTACoordinatorWM'" id=91 idx=0x15c tid=10990 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4c6d128[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
at java/lang/Object.wait(Object.java:485)[inlined]
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)[optimized]
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4c6d128[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '1' for queue: 'JTACoordinatorWM'" id=92 idx=0x160 tid=10991 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4c6d1f0[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
at java/lang/Object.wait(Object.java:485)[inlined]
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)[optimized]
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4c6d1f0[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '2' for queue: 'JTACoordinatorWM'" id=93 idx=0x164 tid=10992 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4c6d2b8[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
at java/lang/Object.wait(Object.java:485)[inlined]
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)[optimized]
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4c6d2b8[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '0' for queue: 'JmsDispatcher'" id=94 idx=0x168 tid=10995 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4c6e688[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4c6e688[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '1' for queue: 'JmsDispatcher'" id=95 idx=0x16c tid=10996 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4c6e750[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4c6e750[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '2' for queue: 'JmsDispatcher'" id=96 idx=0x170 tid=10997 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4c6e818[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4c6e818[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '3' for queue: 'JmsDispatcher'" id=97 idx=0x174 tid=10998 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4c6e8e0[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at weblogic/kernel/ExecuteThread.waitForRequest(ExecuteThread.java:91)
^-- Lock released while waiting: weblogic/kernel/ServerExecuteThread@0xa4c6e8e0[fat lock]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:115)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"ExecuteThread: '4' for queue: 'JmsDispatcher'" id=98 idx=0x178 tid=10999 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/kernel/ServerExecuteThread@0xa4c6e9a8[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
Hi,
Based on the below doc , you can narrow down the actual Thread that is causing this high CPU.
Unexpected High CPU Usage with WebLogic Server (WLS) Support Pattern (Doc ID 779349.1)
Thanks,
Sharmela
Maybe you are looking for
-
Insert new page at run time to pull in images
Hello , I'm using Adobe LC Designer 8.2, and wondering if there is a way to insert a new page - not in design view but at run time since I have a dynamic plot as image (jpg) to bring into a form (1 image per page). How would l load the image in & cr
-
Workflow scenarios for FS-CD module
Hi All I am working for FS-CD module for workflow and I am working on FPY1 transaction code for triggering some workitems based on the clients requirements. Could anyone provide me some workflow scenarios for the above said transaction and also for t
-
How to make servlets reloadable in iplanet web server in solaris??
hi all, does anyone know how to make servlets reloadable in iplanet ??? Thanks for help
-
Collaboration Room Template problem with Roominformation iView
Hello all, I'm trying to make a custom collab room template, but I'm having troubles with the Roominformation iView. It always gives me an error <i>Portal Runtime Error An exception occurred while processing your request Exception id: 11:28_30/10/07
-
You will not accept my new credit card number. On...
How can I obtain more credit as you will not accept my new credit card. The number has not changed, only the security number has changed. Solved! Go to Solution.