ROW CACHE ENQUEUE LOCK/ibrary cache load lock leads to database hung
(lowercase, curly brackets, no spaces)
We faced database hung on 3 node 11i erp 9i rac database.
We saw the library cache load lock timed out events reported in alert log.
Then few ora-600 and later ROW CACHE ENQUEUE LOCK timed out event. Eventually database was hung and we had to bounce the services .
we created support sr 7845542.992 for RCA.
The support says to increase shared pool size to avoid shared pool fragmentation and avoid reload ,additionaly to upgrade to 10g database.
I am not covinced adding additional pool size would solve this or upgrade to 10 .furthermore even 10g has such issues reported.
I saw couple of bugs mentioned such issue can happen due deadlock of session holding latches .
kindly let me know your view on issue
If required i can attach statspack for more information. (lowercase, curly brackets, no spaces)
Many Thanks, i was keen to have your update .
There are 8 cpus on each node . Reloads very high during time period ,but normally there are not high reloads.
Statspack details for 3 nodes
STATSPACK report for
DB Name DB Id Instance Inst Num Release Cluster Host
PROD 21184234 PROD1 1 9.2.0.8.0 YES npi-or-db-p-
11.npi.corp
Snap Id Snap Time Sessions Curs/Sess Comment
Begin Snap: 149817 30-Oct-09 13:00:09 574 #########
End Snap: 149837 30-Oct-09 14:00:17 602 #########
Elapsed: 60.13 (mins)
Cache Sizes (end)
~~~~~~~~~~~~~~~~~
Buffer Cache: 8,192M Std Block Size: 8K
Shared Pool Size: 1,024M Log Buffer: 10,240K
Load Profile
~~~~~~~~~~~~ Per Second Per Transaction
Redo size: 122,414.93 11,449.13
Logical reads: 69,550.76 6,504.89
Block changes: 928.41 86.83
Physical reads: 196.24 18.35
Physical writes: 28.65 2.68
User calls: 343.97 32.17
Parses: 558.61 52.25
Hard parses: 43.48 4.07
Sorts: 467.24 43.70
Logons: 0.63 0.06
Executes: 2,046.99 191.45
Transactions: 10.69
% Blocks changed per Read: 1.33 Recursive Call %: 97.59
Rollback per transaction %: 5.07 Rows per Sort: 15.85
Instance Efficiency Percentages (Target 100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer Nowait %: 100.00 Redo NoWait %: 100.00
Buffer Hit %: 99.72 In-memory Sort %: 100.00
Library Hit %: 96.79 Soft Parse %: 92.22
Execute to Parse %: 72.71 Latch Hit %: 99.77
Parse CPU to Parse Elapsd %: 60.10 % Non-Parse CPU: 78.07
-> s - second
-> cs - centisecond - 100th of a second
-> ms - millisecond - 1000th of a second
-> us - microsecond - 1000000th of a second
-> ordered by wait time desc, waits desc (idle events last)
Avg
Total Wait wait Waits
Event Waits Timeouts Time (s) (ms) /txn
db file sequential read 249,234 0 1,537 6 6.5
db file scattered read 61,776 0 769 12 1.6
row cache lock 780,098 10 566 1 20.2
library cache lock 697,849 157 432 1 18.1
latch free 127,926 4,715 387 3 3.3
global cache cr request 370,770 3,091 309 1 9.6
PL/SQL lock timer 59 58 112 1903 0.0
wait for scn from all nodes 303,572 18 103 0 7.9
library cache pin 26,231 2 100 4 0.7
global cache null to x 17,717 716 92 5 0.5
buffer busy waits 5,388 18 74 14 0.1
db file parallel read 5,245 0 69 13 0.1
log file sync 20,407 29 66 3 0.5
enqueue 52,200 70 60 1 1.4
buffer busy global CR 4,845 33 55 11 0.1
CGS wait for IPC msg 412,512 407,106 50 0 10.7
ksxr poll remote instances 1,279,565 483,046 48 0 33.2
log file parallel write 160,040 0 42 0 4.1
library cache load lock 1,491 2 29 20 0.0
global cache open x 19,507 344 28 1 0.5
buffer busy global cache 957 0 22 23 0.0
global cache s to x 16,516 180 20 1 0.4
db file parallel write 11,120 0 12 1 0.3
log file sequential read 618 0 11 18 0.0
DFS lock handle 23,768 0 10 0 0.6
control file sequential read 8,563 0 4 0 0.2
KJC: Wait for msg sends to c 1,549 57 4 3 0.0
lock escalate retry 76 76 4 52 0.0
SQL*Net break/reset to clien 12,546 0 3 0 0.3
SQL*Net more data to client 85,773 0 3 0 2.2
control file parallel write 1,265 0 2 1 0.0
global cache null to s 648 23 1 2 0.0
global cache busy 200 0 1 5 0.0
global cache open s 1,493 28 1 1 0.0
log file switch completion 12 0 1 61 0.0
PX Deq Credit: send blkd 161 70 1 4 0.0
kksfbc child completion 119 118 1 5 0.0
PX Deq: reap credit 5,948 5,456 0 0 0.2
PX Deq: Execute Reply 83 29 0 3 0.0
process startup 8 0 0 25 0.0
LGWR wait for redo copy 992 12 0 0 0.0
IPC send completion sync 450 450 0 0 0.0
PX Deq: Parse Reply 100 28 0 1 0.0
undo segment extension 10,380 10,372 0 0 0.3
PX Deq: Join ACK 146 65 0 1 0.0
buffer deadlock 222 221 0 0 0.0
async disk IO 1,179 0 0 0 0.0
wait list latch free 2 0 0 16 0.0
PX Deq: Msg Fragment 112 28 0 0 0.0
Library Cache Activity for DB: PROD Instance: PROD1 Snaps: 149817 -149837
->"Pct Misses" should be very low
Get Pct Pin Pct Invali-
Namespace Requests Miss Requests Miss Reloads dations
BODY 116,007 1.1 133,347 19.9 24,338 0
CLUSTER 4,224 0.6 5,131 1.0 0 0
INDEX 15,048 24.1 13,798 26.4 2 0
JAVA DATA 82 0.0 692 39.6 136 0
JAVA RESOURCE 66 39.4 206 25.2 12 0
PIPE 1,140 0.5 1,160 0.5 0 0
SQL AREA 1,197,908 12.6 13,517,660 1.5 111,833 73
TABLE/PROCEDURE 3,847,439 0.8 4,230,265 7.9 142,200 0
TRIGGER 8,444 2.4 8,657 18.5 1,274 0
GES Lock GES Pin GES Pin GES Inval GES Invali-
Namespace Requests Requests Releases Requests dations
BODY 1 1,234 1,258 985 0
CLUSTER 3,222 25 25 25 0
INDEX 13,792 3,641 3,631 3,629 0
JAVA DATA 0 0 0 0 0
JAVA RESOURCE 0 26 25 0 0
PIPE 0 0 0 0 0
SQL AREA 0 0 0 0 0
TABLE/PROCEDURE 857,137 13,130 13,264 10,762 0
TRIGGER 0 200 202 200 0
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
STATSPACK report for
DB Name DB Id Instance Inst Num Release Cluster Host
PROD 21184234 PROD2 2 9.2.0.8.0 YES npi-or-db-p-
12.npi.corp
Snap Id Snap Time Sessions Curs/Sess Comment
Begin Snap: 149847 30-Oct-09 14:00:05 493 #########
End Snap: 149857 30-Oct-09 15:00:02 432 #########
Elapsed: 59.95 (mins)
Cache Sizes (end)
~~~~~~~~~~~~~~~~~
Buffer Cache: 8,192M Std Block Size: 8K
Shared Pool Size: 1,024M Log Buffer: 10,240K
Load Profile
~~~~~~~~~~~~ Per Second Per Transaction
Redo size: 71,853.44 32,058.65
Logical reads: 273,904.84 122,207.36
Block changes: 889.13 396.70
Physical reads: 40.40 18.03
Physical writes: 20.97 9.35
User calls: 153.74 68.60
Parses: 66.19 29.53
Hard parses: 2.66 1.19
Sorts: 25.70 11.47
Logons: 0.16 0.07
Executes: 726.41 324.10
Transactions: 2.24
% Blocks changed per Read: 0.32 Recursive Call %: 92.41
Rollback per transaction %: 4.84 Rows per Sort: 193.55
Instance Efficiency Percentages (Target 100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer Nowait %: 100.00 Redo NoWait %: 99.99
Buffer Hit %: 99.99 In-memory Sort %: 100.00
Library Hit %: 99.35 Soft Parse %: 95.97
Execute to Parse %: 90.89 Latch Hit %: 99.99
Parse CPU to Parse Elapsd %: 36.55 % Non-Parse CPU: 98.28
Wait Events for DB: PROD Instance: PROD2 Snaps: 149847 -149857
-> s - second
-> cs - centisecond - 100th of a second
-> ms - millisecond - 1000th of a second
-> us - microsecond - 1000000th of a second
-> ordered by wait time desc, waits desc (idle events last)
Avg
Total Wait wait Waits
Event Waits Timeouts Time (s) (ms) /txn
enqueue 65,823 33,667 90,459 1374 8.2
row cache lock 38,996 560 1,795 46 4.8
PX Deq Credit: send blkd 522 499 1,223 2344 0.1
PX Deq: Parse Reply 466 416 987 2117 0.1
db file sequential read 50,130 0 421 8 6.2
library cache lock 78,842 172 210 3 9.8
db file scattered read 6,904 0 152 22 0.9
global cache cr request 84,801 575 113 1 10.5
latch free 8,096 736 65 8 1.0
log file sync 5,676 27 41 7 0.7
wait for scn from all nodes 18,891 10 24 1 2.3
CGS wait for IPC msg 394,678 392,142 21 0 49.0
library cache pin 1,339 0 17 13 0.2
global cache null to x 2,145 48 16 8 0.3
global cache s to x 3,242 32 16 5 0.4
buffer busy waits 366 10 15 40 0.0
ksxr poll remote instances 70,990 31,295 14 0 8.8
db file parallel read 359 0 11 31 0.0
global cache open x 2,708 55 10 4 0.3
async disk IO 3,474 0 8 2 0.4
global cache open s 3,470 10 6 2 0.4
log file parallel write 13,076 0 5 0 1.6
global cache busy 58 40 5 90 0.0
PL/SQL lock timer 1 1 5 4877 0.0
DFS lock handle 3,362 0 5 1 0.4
log file sequential read 412 0 4 10 0.1
db file parallel write 2,774 0 3 1 0.3
library cache load lock 59 0 3 58 0.0
buffer busy global CR 722 0 3 4 0.1
control file sequential read 6,398 0 3 0 0.8
SQL*Net break/reset to clien 16,078 0 2 0 2.0
name-service call wait 26 0 2 67 0.0
control file parallel write 1,248 0 2 1 0.2
process startup 24 0 1 49 0.0
KJC: Wait for msg sends to c 3,491 4 1 0 0.4
SQL*Net more data to client 23,724 0 1 0 2.9
buffer busy global cache 23 0 0 19 0.0
global cache null to s 114 0 0 4 0.0
PX Deq: reap credit 5,646 5,509 0 0 0.7
log file switch completion 4 0 0 58 0.0
lock escalate retry 54 54 0 1 0.0
IPC send completion sync 119 118 0 0 0.0
direct path read 2,820 0 0 0 0.3
direct path read (lob) 3,632 0 0 0 0.5
PX Deq: Join ACK 88 37 0 0 0.0
direct path write 2,470 0 0 0 0.3
kksfbc child completion 6 6 0 6 0.0
buffer deadlock 3 3 0 11 0.0
global cache quiesce wait 4 4 0 8 0.0
Library Cache Activity for DB: PROD Instance: PROD2 Snaps: 149847 -149857
->"Pct Misses" should be very low
Get Pct Pin Pct Invali-
Namespace Requests Miss Requests Miss Reloads dations
BODY 27,353 0.5 28,091 6.5 1,643 0
CLUSTER 203 1.0 269 1.5 0 0
INDEX 526 9.9 271 19.9 0 0
JAVA DATA 18 0.0 120 6.7 4 0
JAVA RESOURCE 20 45.0 56 26.8 3 0
JAVA SOURCE 1 100.0 1 100.0 0 0
PIPE 999 0.4 1,043 0.4 0 0
SQL AREA 131,793 7.6 3,406,577 0.4 7,012 0
TABLE/PROCEDURE 926,987 0.2 1,907,993 1.0 8,845 0
TRIGGER 1,519 0.1 1,532 4.9 69 0
GES Lock GES Pin GES Pin GES Inval GES Invali-
Namespace Requests Requests Releases Requests dations
BODY 1 129 277 117 0
CLUSTER 168 2 2 2 0
INDEX 271 52 56 52 0
JAVA DATA 0 0 0 0 0
JAVA RESOURCE 0 9 6 0 0
JAVA SOURCE 0 1 1 1 0
PIPE 0 0 0 0 0
SQL AREA 0 0 0 0 0
TABLE/PROCEDURE 89,523 764 868 460 0
TRIGGER 0 2 14 2 0
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DB Name DB Id Instance Inst Num Release Cluster Host
PROD 21184234 PROD3 3 9.2.0.8.0 YES npi-or-db-p-
13.npi.corp
Snap Id Snap Time Sessions Curs/Sess Comment
Begin Snap: 149808 30-Oct-09 14:00:00 31 #########
End Snap: 149809 30-Oct-09 15:00:02 34 11,831.4
Elapsed: 60.03 (mins)
Cache Sizes (end)
~~~~~~~~~~~~~~~~~
Buffer Cache: 8,192M Std Block Size: 8K
Shared Pool Size: 1,024M Log Buffer: 10,240K
Load Profile
~~~~~~~~~~~~ Per Second Per Transaction
Redo size: 1,518.14 36,700.35
Logical reads: 1,333.43 32,235.02
Block changes: 5.09 123.01
Physical reads: 54.31 1,312.88
Physical writes: 3.91 94.44
User calls: 1.46 35.40
Parses: 2.24 54.21
Hard parses: 0.04 0.93
Sorts: 0.84 20.28
Logons: 0.06 1.45
Executes: 3.11 75.23
Transactions: 0.04
% Blocks changed per Read: 0.38 Recursive Call %: 94.31
Rollback per transaction %: 45.64 Rows per Sort: 215.97
Instance Efficiency Percentages (Target 100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer Nowait %: 99.99 Redo NoWait %: 100.00
Buffer Hit %: 96.21 In-memory Sort %: 100.00
Library Hit %: 99.07 Soft Parse %: 98.29
Execute to Parse %: 27.94 Latch Hit %: 99.98
Parse CPU to Parse Elapsd %: 69.88 % Non-Parse CPU: 97.92
Wait Events for DB: PROD Instance: PROD3 Snaps: 149808 -149809
-> s - second
-> cs - centisecond - 100th of a second
-> ms - millisecond - 1000th of a second
-> us - microsecond - 1000000th of a second
-> ordered by wait time desc, waits desc (idle events last)
Avg
Total Wait wait Waits
Event Waits Timeouts Time (s) (ms) /txn
enqueue 19,510 7,472 15,509 795 130.9
PX Deq: Parse Reply 1,152 1,071 2,577 2237 7.7
row cache lock 2,202 518 1,579 717 14.8
db file scattered read 31,556 0 354 11 211.8
db file sequential read 17,272 0 67 4 115.9
db file parallel read 1,722 0 34 20 11.6
global cache cr request 53,754 91 32 1 360.8
wait for scn from all nodes 1,897 13 10 5 12.7
CGS wait for IPC msg 403,358 401,478 10 0 2,707.1
DFS lock handle 4,753 0 8 2 31.9
direct path read 1,248 0 6 5 8.4
PX Deq: Execute Reply 110 38 6 51 0.7
global cache open s 160 10 5 31 1.1
control file sequential read 6,442 0 3 0 43.2
name-service call wait 26 0 2 78 0.2
latch free 129 109 2 13 0.9
KJC: Wait for msg sends to c 153 24 1 9 1.0
control file parallel write 1,245 0 1 1 8.4
buffer busy waits 199 0 1 6 1.3
process startup 20 0 1 44 0.1
global cache null to x 74 2 1 9 0.5
global cache null to s 19 0 1 29 0.1
global cache open x 268 1 1 2 1.8
library cache lock 1,150 0 0 0 7.7
PX Deq: Join ACK 129 48 0 3 0.9
log file parallel write 1,157 0 0 0 7.8
async disk IO 219 0 0 1 1.5
direct path write 1,024 0 0 0 6.9
ksxr poll remote instances 6,740 4,595 0 0 45.2
PX Deq: reap credit 6,580 6,511 0 0 44.2
buffer busy global CR 73 0 0 2 0.5
log file sequential read 11 0 0 10 0.1
log file sync 100 0 0 1 0.7
global cache s to x 282 2 0 0 1.9
db file parallel write 95 0 0 1 0.6
library cache pin 142 0 0 0 1.0
SQL*Net break/reset to clien 28 0 0 1 0.2
IPC send completion sync 81 81 0 0 0.5
PX Deq: Signal ACK 32 14 0 1 0.2
PX Deq Credit: send blkd 3 1 0 7 0.0
SQL*Net more data to client 841 0 0 0 5.6
PX Deq: Msg Fragment 37 17 0 0 0.2
log file single write 4 0 0 1 0.0
db file single write 1 0 0 1 0.0
SQL*Net message from client 4,213 0 13,673 3246 28.3
gcs remote message 214,784 75,745 7,016 33 1,441.5
wakeup time manager 233 233 6,812 29237 1.6
PX Idle Wait 2,338 2,294 5,686 2432 15.7
PX Deq: Execution Msg 2,151 1,979 4,796 2229 14.4
Library Cache Activity for DB: PROD Instance: PROD3 Snaps: 149808 -149809
->"Pct Misses" should be very low
Get Pct Pin Pct Invali-
Namespace Requests Miss Requests Miss Reloads dations
BODY 1,290 0.0 1,290 0.0 0 0
CLUSTER 18 0.0 8 0.0 0 0
SQL AREA 4,893 2.0 36,371 0.5 2 0
TABLE/PROCEDURE 1,555 3.9 3,834 4.9 71 0
TRIGGER 286 0.0 286 0.0 0 0
GES Lock GES Pin GES Pin GES Inval GES Invali-
Namespace Requests Requests Releases Requests dations
BODY 1 0 0 0 0
CLUSTER 4 0 0 0 0
SQL AREA 0 0 0 0 0
TABLE/PROCEDURE 863 224 42 42 0
TRIGGER 0 0 0 0 0
-------------------------------------------------------------
Similar Messages
-
Performance issues; waited too long for a row cache enqueue lock!
hi Experts,
OS: Oracle Solaris on SPARC (64-bit)
DB version:
SQL> select * from V$VERSION;
BANNER
Oracle Database 11g Release 11.2.0.1.0 - 64bit Production
PL/SQL Release 11.2.0.1.0 - Production
CORE 11.2.0.1.0 Production
TNS for Solaris: Version 11.2.0.1.0 - Production
NLSRTL Version 11.2.0.1.0 - Production
SQL>We have seen 100% CPU usage and high database load, so I checked the instance and have seen there were many blocking sessions and more than 71 sessions running the same select ;
elect tablespace_name as tbsname from (select tablespace_name,sum(bytes)/1024/1024 free_mb,0 total_mb,0 max_mb from dba_free_space group by tablespace_name union select tablespace_name, 0 current_mb,sum(bytes)/1024/1024 total_mb, sum(decode(maxbytes, 0, bytes, maxbytes))/1024/1024 max_mb from dba_data_files group by tablespace_name) group by tablespace_name having round((sum(total_mb)-sum(free_mb))/sum(max_mb)*100) > 95 Blocking sessions are running queries like this;
SELECT * from MYTABLE WHERE MYCOL=:1 FOR UPDATE;This select queries are coming from a cron job running every 10 minutes to check the tablespaces; so I first killed (kill -9 pid) those select statements so the load and CPU decreased to 13% of CPU usage. Blocking sessions still there and I didn't killed them waiting for app guys confirmation... after few hours and the CPU usage never went down the 13%; I have seen many errors;
WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK! pid=...System State dumped to trace file .....trcAfter that , we decided to restart the DB to release the locks!
I would like to understand why during loads we were no able to run those select statements, statspack schedule snapshot reports were not able to finish, also automatic
database statistics... why 5 for update statements locked the whole DB?user12035575 wrote:
SELECT FOR UPDATE will only lock the table row until the transaction is completed.
"WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK" happens when it needs to acquire a lock on data dictionary. Did you check the trace file associated with the statement?The trace file is too long, which information I need to focus more? -
Error: WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK! pid=26
Hi every one,
Today, i met a problem: Application cannot connect to database because database hang ( I also cannot connect to database with sqlplus) . Check alert log, only one error:
WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK! pid=26This error not only appear first time, but also happen every one month. I must reset server for database release all memory but I think It isn't a true solution!
Could you give a recommend for this.
Regards.The Row Cache is actually the Data Dictionary Cache. It is where definitions from the data dictionary (tablespaces, objects, users etc) are loaded into memory.
There would be an associated trace file written with the occurrence of this warning.
See Oracle Support Note 278316.1 for more information
Hemant K Chitale -
High library cache load lock waits in AWR
Hi All,
Today i faced a significant performance problem related to shared pool. I made some observations, thought it would be a nice idea to share them with Oracle experts. Please feel free to add your observations/recommendations and correct me where i am wrong.
Here are the excerpts from AWR report created for the problem timing. Database server is on 10.2.0.3 and running with 2*16 configuration. DB cache size is 4,000M and shared pool size is of 3008M.
Snap Id Snap Time Sessions Cursors/Session
Begin Snap: 9994 29-Jun-09 10:00:07 672 66.3
End Snap: 10001 29-Jun-09 17:00:49 651 64.4
Elapsed: 420.70 (mins)
DB Time: 4,045.34 (mins) -- Very poor response time visible from difference between DB time and elapsed time.
Load Profile
Per Second Per Transaction
Redo size: 248,954.70 23,511.82
Logical reads: 116,107.04 10,965.40
Block changes: 1,357.13 128.17
Physical reads: 125.49 11.85
Physical writes: 51.49 4.86
User calls: 224.69 21.22
Parses: 235.22 22.21
Hard parses: 4.83 0.46
Sorts: 102.94 9.72
Logons: 1.12 0.11
Executes: 821.11 77.55
Transactions: 10.59 -- User calls and Parse count are almost same, means most of the calls are for parse. Most of the parses are soft. Per transaction 22 parses are very high figure.
-- Not much disk I/O activity. Most of the reads are being satisfy from memory.
Instance Efficiency
Buffer Nowait %: 100.00 Redo NoWait %: 100.00
Buffer Hit %: 99.92 In-memory Sort %: 100.00
Library Hit %: 98.92 Soft Parse %: 97.95
Execute to Parse %: 71.35 Latch Hit %: 99.98
Parse CPU to Parse Elapsd %: 16.82 % Non-Parse CPU: 91.41 -- Low execute to parse ratio denotes CPU is significantly busy in parsing. Soft Parse% showing, most of the parse are soft parses. It means we should concentrate on soft parsing activity.
-- Parse CPU to Parse Elapsed % is quite low, means some bottleneck is there related to parsing. It could be a side-effect of huge parsing pressure. Like CPU cycles are not available.
Shared Pool Statistics
Begin End
Memory Usage %: 81.01 81.92
% SQL with executions>1: 88.51 86.93
% Memory for SQL w/exec>1: 86.16 86.76 -- Shared Pool memory seems ok (in 80% range)
-- 88% of the SQLs are repeating ones. It's a good sign.
Top 5 Timed Events
Event Waits Time(s) Avg Wait(ms) % Total Call Time Wait Class
library cache load lock 24,243 64,286 2,652 26.5 Concurrency
db file sequential read 1,580,769 42,267 27 17.4 User I/O
CPU time 33,039 13.6
latch: library cache 53,013 29,194 551 12.0 Concurrency
db file scattered read 151,669 13,550 89 5.6 User I/O Problem-1: Contention on Library cache: May be due to under-sized shared pool, incorrect parameters, poor application design, But since we already observed that most of the parses are soft parses and shared pool usgae in 80%, seems problem related to holding cursors. open_cursors/session_cached_cursors are red flags.
Problem-2: User I/O, may be due to poor SQLs, I/O sub-system, or poor physical design (wrong indexes are being used as DB file seq reads)
Wait Class
Wait Class Waits %Time -outs Total Wait Time (s) Avg wait (ms) Waits /txn
Concurrency 170,577 44.58 109,020 639 0.64
User I/O 2,001,978 0.00 59,662 30 7.49
System I/O 564,771 0.00 8,069 14 2.11
Application 145,106 1.25 6,352 44 0.54
Commit 176,671 0.37 4,528 26 0.66
Other 27,557 6.31 2,532 92 0.10
Network 6,862,704 0.00 696 0 25.68
Configuration 3,858 3.71 141 37 0.01
Wait Events
Event Waits %Time -outs Total Wait Time (s) Avg wait (ms) Waits /txn
library cache load lock 24,243 83.95 64,286 2652 0.09
db file sequential read 1,580,769 0.00 42,267 27 5.91
latch: library cache 53,013 0.00 29,194 551 0.20
db file scattered read 151,669 0.00 13,550 89 0.57
latch: shared pool 25,403 0.00 12,969 511 0.10
log file sync 176,671 0.37 4,528 26 0.66
enq: TM - contention 1,455 90.93 3,975 2732 0.01 Instance Activity Stats
opened cursors cumulative 5,290,760 209.60 19.80
parse count (failures) 6,181 0.24 0.02
parse count (hard) 121,841 4.83 0.46
parse count (total) 5,937,336 235.22 22.21
parse time cpu 283,787 11.24 1.06
parse time elapsed 1,687,096 66.84 6.31 Latch Activity
library cache 85,042,375 0.15 0.43 29194 304,831 7.16
library cache load lock 257,089 0.00 1.20 0 69,065 0.00
library cache lock 41,467,300 0.02 0.07 6 2,714 0.07
library cache lock allocation 730,422 0.00 0.44 0 0
library cache pin 28,453,986 0.01 0.16 8 167 0.00
library cache pin allocation 509,000 0.00 0.38 0 0 Init.ora parameters
cursor_sharing= EXACT
open_cursors= 3000
session_cached_cursors= 0
-- open_cursors value is too high. I have checked that maximum usage by a single session is 12%.
-- session_cached_cursors are 0 causing soft parsing. 500/600 is good number to start with.
cursor_sharing exact may cause hard parses. But here, hard parsing is comparatively small, we can ignore this.
From v$librarycache
NAMESPACE GETS GETHITS GETHITRATIO PINS PINHITRATIO RELOADS INVALIDATIONS
SQL AREA 162827 25127 .154317159 748901435 .999153087 107941 81886-- high invalidation count due to DDL like activities.
-- high reloads due to small library cache.
-- hit ratio too small.
-- Need to pin frequently executed objects into library cache.
P.S. Same question asked on Oracle_L, but due to formatting reasons, pasing duplicate contents here.
Regards,
Neeraj Bhatia
Edited by: Neeraj.Bhatia2 on Jul 13, 2009 6:51 AMThanks Charles. I really appreciate your efforts to diagnose the issue.
I agree with you performance issue is caused by soft parsing, which can be solved by holding cursors (session_cached_cursors). It may be due to oversized shared pool, which is causing delay in searching child cursors.
My second thought is, there is large number of reloads, which can be due to under-sized shared pool, if invalidation activities are not going (CBO statistics collection, DDL etc), cursors are being flushed frequently.
CPU utilization is continuously high (above 90%). Pasting additional information from same AWR report.
Namespace Get Requests Pct Miss Pin Requests Pct Miss Reloads Invalidations
BODY 225,345 0.76 4,965,541 0.15 5,533 0
CLUSTER 1,278 1.41 2,542 1.73 26 0
INDEX 5,982 9.31 13,922 7.35 258 0
SQL AREA 141,465 54.10 27,831,235 1.21 69,863 19,085 Latch Miss Sources
Latch Name Where NoWait Misses Sleeps Waiter Sleeps
library cache lock kgllkdl: child: no lock handle 0 8,250 5,792 Time Model Statistics
Statistic Name Time (s) % of DB Time
sql execute elapsed time 206,979.31 85.27
PL/SQL execution elapsed time 94,651.78 39.00
DB CPU 33,039.29 13.61
parse time elapsed 22,635.47 9.33
inbound PL/SQL rpc elapsed time 14,763.48 6.08
hard parse elapsed time 14,136.77 5.82
connection management call elapsed time 1,625.07 0.67
PL/SQL compilation elapsed time 760.76 0.31
repeated bind elapsed time 664.81 0.27
hard parse (sharing criteria) elapsed time 500.11 0.21
Java execution elapsed time 252.95 0.10
failed parse elapsed time 167.23 0.07
hard parse (bind mismatch) elapsed time 124.11 0.05
sequence load elapsed time 23.34 0.01
DB time 242,720.12
background elapsed time 11,645.52
background cpu time 247.25 According to this DB CPU is 65% utilization (DB CPU + Background CPU / Total Available CPU seconds). While at the same time DB host was 95% utilized (confirmed from DBA_HIST_SYSMETRIC_SUMMARY).
Operating System Statistics
Statistic Total
BUSY_TIME 3,586,030
IDLE_TIME 1,545,064
IOWAIT_TIME 22,237
NICE_TIME 0
SYS_TIME 197,661
USER_TIME 3,319,452
LOAD 11
RSRC_MGR_CPU_WAIT_TIME 0
PHYSICAL_MEMORY_BYTES 867,180
NUM_CPUS 2 -
Enqueue: Elem.lock 0 contains unpermitted lock mode . lock object EZBEV_BO
Hello Guru's,
we have a unusual problem in the syslog of our ERP System.
Enqueue: Elem.lock 0 contains unpermitted lock mode . lock object EZBEV_BO
We can't found any note or forum entry to solve this.
We checked SAP Notes: 1324099, 891372, 606425, 447242, 1260601 and searched in the forum, but nothing helpful found.
System Data:
ERP 6.0 / NW7.01 with EHP4
SAP Kernel : 701_REL
created on : Jan 18 2010 21:47:07
Kernel Patch number : 77
Source ID : 0.077
DBSL Patch No. : 075
Database
ORACLE 10.2.0.2
Operating System
AIX 5.3
Can someone help?
regards
Chrisdeveloper error
-
Enqueue: Elem.lock 0 contains unpermitted lock mode /. lock object
Hello All,
SCM 5.0 system which is upgraded from 4.1:
System log entries contains:
<b>Enqueue: Elem.lock 0 contains unpermitted lock mode /. lock object /sapapo/E</b>
Work process log file contains :
LOG GE8===>0 /sapapo/E_MATLOC
I have gone through few notes like 606425, 447242.( lower version)
From which I came to the conclusion that it is program error.
Also group GE and No:8 What does it mean?
Please let me know your thoughts.Thanks.
Regards,
Tushar
Message was edited by:
Tushar Chavan
Message was edited by:
Tushar ChavanHi,
I found one more note :
Note 891372 - Lock entries for TemSe obj remain after completion of proc
The issue looks pretty similar. I think the notes you found are better suited for sap apo.
Log a call with SAP and check what they have to say.
Regards,
Siddhesh -
The specified cache entry could not be loaded during initialization
The OBIEE server is failing every few minutes of working.
I saw in the NQServer.log:
The specified cache entry could not be loaded during initialization.
Cache entry, '/home/oracleadmin/OracleBIData/cache/NQS_rprt_734086_39693_00000001.TBL', is invlaid due to either non-existing file name, size does not match, exceeding maximum space limit nor exceeding maximum entry limit, and it's been removed.
I checked the directory and there is no space limitation.
Can someone please clarify?
The server/Config/NQSConfig.INI is as follows:
[ CACHE ]
ENABLE = YES;
DATA_STORAGE_PATHS = "/home/oraceladmin/OracleBIData/cache" 2000 MB;
MAX_ROWS_PER_CACHE_ENTRY = 100000; // 0 is unlimited size
MAX_CACHE_ENTRY_SIZE = 1 MB;
MAX_CACHE_ENTRIES = 1000;
POPULATE_AGGREGATE_ROLLUP_HITS = NO;
USE_ADVANCED_HIT_DETECTION = NO;
MAX_SUBEXPR_SEARCH_DEPTH = 7;It's more than enough if you are using it for .tmp and .tbl files storage. Have you checked work_directory_path in NQSConfig.ini file .Is it on diffrent mount point.Usually we wil create a seperate mount point and use that as storage location.
b. Instead of using SAPurgeAllCache() , can I add shell script which delete the files fromt the cache directory and the execute run-sa.sh?You need to understand diffrence between OBIEE rpd cache and .tbl and .tmp file. SAPurgeAllCache() will purge cache from RPD.It wont purge .tmp or .tbl files.These files will be stored in some physical location.When a report start executing .tmp file will start evoluting and after execution of report it will be purged by OBIEE server automatically.
see this blog for some insight into this topic
Re: TMP Files uneder /OracleBIData/tmp
Regards,
Sandeep -
Routing and caching layer vs. F5 load balancer or both.
We have 4 WFE servers farm, For load balancing, I can see following option.
Purchase the F5 load balancer, don't use the RM.
Have a one more server that will do Routing using RM and caching. Don't use the
F5 load balancer.
Have both F5 load balancer and RM
I am wondering what approach we should take. I have read a lot of articles in MSDN, all suggests that we should have a routing & caching layer at front of WFE server. Considering we have just 4 WFE Servers do we need to have a separate routing and cache
layer along with F5 load balancer. I am really confused here.
Regards Restless SpiritUsing Request Management isn't a replacement for using a hardware load balancer. It will be used internally within SharePoint to route things appropriately. It depends on the number of users your farm will support, but you may or may not need
a dedicated distributed cache / request management layer. Refer to this
diagram to see where your farm fits in.
Corey Roth - SharePoint Server MVP blog:
www.dotnetmafia.com twitter: @coreyroth |
SP2 Apps -
Hi All
I have iMovie 11 and everytime I load it, it locks up whilst loading movies from iphoto. I have to force quite and when I reload again it asks about excluding an avi file in iphoto library. I exclude but then the same happens again.
Is there anyway of stopping it doing this. I have loads of movies from an old camera whcih probably got imported and would take an age to find them all
Thanks
GlennHi again Glenn,
A quick way to find all your avi clips in iPhoto is to click on Events in the iPhoto Library, then in the bottom left corner use Search. In the Search box, type .avi or .AVI - this should bring up all the avi files in your iPhoto Library, showing a thumbnail for each video.
You can also create a Smart Album in iPhoto that will bring together all your movies into one album in the Library. From iPhoto's menu select File > New Smart Album, then adjust the settings to show as below:
By a process of elimination you may be able to track down any problem files in this Smart Album. Of course, the Search field I mentioned above will allow you to isolate just the avi files. Note that whenever you add a new video clip to iPhoto it will automatically appear in the Smart Album (a brilliant feature in iPhoto).
Anyway, first things first - let us know what the Format and FPS is for one of the avi clips (as I asked in my first post here).
John -
Enqueue: Transfer error while reading lock entries.
Hi
We have an error in our SM21 log in ERP Prod system.
we have enque process on our CI server.
any body could provide the solution?
"Enqueue: Transfer error while reading lock entries
> Destination: prodecccs_ERP_00
> ENQUE_READ exception code: 8
> Name or password is incorrect (repeat logon)"
Thanks,
Ravi.Thanks Ravi,
that's a big help. unfortunately for us, we're already on a release that includes that note (SAPKB70209). So I'll get a message opened to see if they can update that note so we can apply the fix. at least I know what component to open it under now.
I owe you one man.
--NICK -
My iPhone locked up after loading iOS 7.0.2
My iPhone locked up after loading iOS 7.0.2 and it will not reset using the home key / on-off button reset. The screen returns to and iTunes icon and a 'cord' end that looks like the end of the power cord.
Force the phone into recovery mode, as described here, & restore it:
http://support.apple.com/kb/HT1808 -
Itunes Locks seconds after loading
Recently my itunes has started to lock just after loading, it opens then begins to check the podcasts then locks, i can do nothing after that point apart from close it from the task manager. i have repaired, uninstalled and reinstalled it but still nothing, it does it regardless of having the ipod connected.
seems to be the podcast issue, and seems to be working after i deleted some partly downloaded tmps
-
Loader locks images it has loaded
Hi
I am trying to create a simple flash that loads an image from my HardDrive. The following is my simple program:
package {
import flash.display.Sprite;
import flash.events.*;
import flash.display.Loader;
import flash.net.URLRequest;
import flash.display.Bitmap;
public class Test extends Sprite {
private var imgLoader:Loader;
public function Test() {
imgLoader= new Loader();
imgLoader.contentLoaderInfo.addEventListener(Event.COMPLETE, onLoaderReady, false, 0, true);
imgLoader.load(new URLRequest("test_image.png"));
//imgLoader.close();
private function onLoaderReady(e:Event) {
//imgLoader.close();
var b = Bitmap(e.target.content);
b.x = 100;
b.y = 100;
addChild(b);
however, I have noted that while this flash is opened, even though the loader has finished loading the image and the image is currently being displayed, I can not edit the image with any other program. (It will complain that the image is currently being used by another program.)
I have found that if i use the close() function for the Loader, it will release the lock. however, if i try to close the loader in my event complete function (because i want to close the loader only after when the image has finished loading), it will complain with the following error:
Error: Error #2029: This URLStream object does not have a stream opened.
Does anyone knows how to release the lock the Loader has acquired after when the image has finished loading?
thankyou very much in advance.i'm thinking of writing a small program so that the client (swf) will load an image. While the client is being opened, the server might want to modify the image. after the image has been modified, it can notify the client that it has been modified and the client will refresh the image and get the latest image.
Date: Thu, 12 Jan 2012 22:58:51 -0700
From: [email protected]
To: [email protected]
Subject: Loader locks images it has loaded
Re: Loader locks images it has loaded
created by kglad in Action Script 3 - View the full discussion
i don't think there's a work-around and i can't see any purpose. why do you want to do that?
Replies to this message go to everyone subscribed to this thread, not directly to the person who posted the message. To post a reply, either reply to this email or visit the message page: http://forums.adobe.com/message/4137609#4137609
To unsubscribe from this thread, please visit the message page at http://forums.adobe.com/message/4137609#4137609. In the Actions box on the right, click the Stop Email Notifications link.
Start a new discussion in Action Script 3 by email or at Adobe Forums
For more information about maintaining your forum email notifications please go to http://forums.adobe.com/message/2936746#2936746. -
Integration Builder - runtime data cache - Error when creating cache list
All,
When I go into the Integration Builder --> Administration --> Runtime tab --> Data Cache
I get a message saying "Error when creating cache list".
All cache areas look ok and I do not see any other error.
Also, RFC INTEGRATION_DIRECTORY_HMI looks good.
Please advise.
Thanks a lot.Hi,
Use transaction SXI_CACHE to update the Integration Directory cache. Alternatively, you can use the following URLs to update the CPA cache. Use XIDIRUSER to refresh the cache.
For complete cache refresh u2013 http://<hostname>:<port>/CPACache/refresh?mode=full
For delta cache refresh u2013 http://<hostname>:<port>/CPACache/refresh?mode=delta
If this does not solve the issue, check transaction SLDCHECK to ensure that connection to SLD is available. If the connection fails, check the configuration in the transaction SLDAPICUST. Make sure that the password maintained is correct and the maintained service user is not locked.
Now in the Integration Repository go to Environment u2192 Clear SLD Data Cache. Also go to Integration Directoy and clear the cache using menu Environment u2192 Clear SLD Data Cache.
Open the XI Start Page and click on Administration. On the Repository tab, choose Cache Overview. Refresh the cache using the buttons/icons on the right. Use XIDIRUSER to refresh the cache. Carry out cache refresh in the same way on the Directory and Runtime tabs.
regards,
ganesh. -
Query performance problem - events 2505-read cache and 2510-write cache
Hi,
I am experiencing severe performance problems with a query, specifically with events 2505 (Read Cache) and 2510 (Write Cache) which went up to 11000 seconds on some executions. Data Manager (400 s), OLAP data selection (90 s) and OLAP user exit (250 s) are other the other event with noticeable times. All other events are very quick.
The query settings (RSRT) are
persistent cache across each app server -> cluster table,
update cache in delta process is checked ->group on infoprovider type
use cache despite virtual characteristics/key figs checked (one info-cube has1 virtual key figure which should have a static result for a day)
=>Do you know how I can get more details than what's in 0TCT_C02 to break down the read and write cache events time or do you have any recommandation?
I have checked and no dataloads were in progres on the info-providers and no master data loads (change run). Overall system performance was acceptable for other queries.
ThanksHi,
Looks like you're using BDB, not BDB JE, and this is the BDB JE forum. Could you please repost here?:
Berkeley DB
Thanks,
mark
Maybe you are looking for
-
Open SD orders and partner type?
Hi, I need to extract 'open sd orders' and 'partner type' i.e all partner types and not only just SP or SH The transaction VA05 does no allow all partner functions - and I need to sort by all types is there another way? If I want to use SQVI .
-
Oracle 9 and XA transaction recovery
Just thought I would pass on a bit of advice if you are attempting to enable XA transaction recovery from either Application Server 7.0 or 8.1 with Oracle version 9.2.0.6 and possibly above. In 9.2.0.6 Oracle changed the way they responded to XA reco
-
Why does a change in NLS_LANG affect formatting output? User has a simple script: select spriden_id, spriden_last_name, spriden_first_name from spriden where spriden_id = '900012540' and spriden_change_ind is null select spriden_id, rpad(spriden_last
-
Web applet loadlibrary problem
Hello, According to http://java.sun.com/sfaq/example/loadLibrary.html A web applet should be able to load a library from a method in the classpath. However, when I run my web applet, I get the error: java.security.AccessControlException: access denie
-
I've been plugging in my Samsung Galaxy S2 phone into my thinkpad to charge however the USB charging on the laptop itself is incredibly weak. I left the phone off after being drained and plugged it into my Think pad overnight. When I woke up, the bat