Lot of enqueue_locks

Question

DB-11.1.0.7
OS- RHEL 5.4
How to get rid of such large enqueue_locks ?
SELECT *  FROM v$sysstat WHERE class=4;
SELECT chr(bitand(p1,-16777216)/16777215)||chr(bitand(p1, 16711680)/65535) "Lock",
to_char( bitand(p1, 65535) )    "Mode"
FROM v$session_wait
WHERE event = 'enqueue';
STATISTIC# NAME                           CLASS      VALUE    STAT_ID
        27 enqueue timeouts                   4     825598 2425496215
        28 enqueue waits                      4    1430814 2307006529
        29 enqueue deadlocks                  4          0  472183780
        30 enqueue requests                   4  217866983 2440542518
        31 enqueue conversions                4   56459244   57745673
        32 enqueue releases                   4  217028625 2553146675
6 rows selected.
select * from v$enqueue_stat where cum_wait_time>0
order by cum_wait_time desc;
   INST_ID EQ TOTAL_REQ# TOTAL_WAIT#  SUCC_REQ# FAILED_REQ# CUM_WAIT_TIME
         1 CF    5667288       41379    5655029       12252      11953280
         1 TM   11263749          57   10986876      276839      10057870
         1 TX   30026842       92249   30027442          79       1392060
         1 PS  101041109     1229093  100537143      495123       1018750
         1 KO     140319       14888     140319           0        939470
         1 ST       2216          43       2216           0        520480
         1 HW    1292297       28164    1290811        1486        320530
         1 SQ      58810        1236      58810           0        234370
         1 TC       8440        1528       8440           0        234060
         1 FB    4353897       11118    4353898           0        127860
         1 RO      60453        5914      60453           0         99650
         1 CI     254390          38     254390           0         53420
         1 JD    1213641           6    1213641           0         52840
         1 BF      87739        3524      60598       27140         31180
         1 RF   32872044        1212   32872044           0         11430
         1 PR     139956         208     139956           0          9800
         1 US      29876          74      29875           0          6690
         1 JS   17529859          76   17529859           0          1840
         1 WF      33387           4      32993         394          1140
         1 PV      13121           3      13121           0            80
         1 TH     162560           1     162559           1            30Hidden parameter '_enqueue_locks' is set to 10466
Can i ask for giving direction to fine tune this enqueue issue ?

Answer 1

OraDBA02 wrote:
Load Profile
     Per Second     Per Transaction     Per Exec     Per Call
Redo size:     10,301,526.5     7,315,836.4           
Logical reads:     161,642.8     114,793.9           
Block changes:     32,621.4     23,166.7           
Physical reads:     95,054.7     67,505.0           
Physical writes:     4,437.8     3,151.6           
User calls:     30.6     21.8           
Executes:     12,801.8     9,091.4           
Top 5 Timed Foreground Events
Event     Waits     Time(s)     Avg wait (ms)     % DB time     Wait Class
db file sequential read     1,096,941     9,564     9     34.20     User I/O
direct path read     652,742     8,386     13     29.99     User I/O
DB CPU           5,711           20.42     
direct path read temp     223,382     433     2     1.55     User I/O
log file switch completion     2,209     275     125     0.98     Configuration
That's a fairly dramatic change from the last output - a huge volume of direct path reads - 652,742 reads could mean as much as 652GB - which is reflected in the 95,000 blocks per second. Your hardware can deliver a lot of reads when it tries ... but your job is to find out why. It's possible that 11.1 has simply changed some execution paths on you - or it might have decided to do some serial direct read table scans.
You said increasing logfile size to 2 GB. That would be a drastic increase ! May i ask you how did u derive that 2 GB ? Log switches are pretty high for about 2 hrs only when LOAD is executed, After that its pretty stable. We have FSFO configuration on my physical standby as well. Would there be any effect on by log-shipping activities on standby having 2 GB log file size ?
Your redo rate suggested you were switching every 15 seconds. Log file switches lead to large numbers of CF enqueue requests; I just suggested a size that would lead to switches every 4 minutes. I don't know all the subtle details of fast start failover - the log files and standby log files have to be matched, of course; but I don't think the absolute size of the log files is a consideration since you'll be pumping the redo to the remote site in real time, not at a log file switch. How frequently do the log files switch when you are not loading ? If you increase the log file size you can always set the archive_lag_target parameter to that many seconds.
All 'db file sequential reads' are on PK (all Index Unique Scans on Local PK) as all merges are based on my pk. Presumably the merges update the tables, and so there are random accesses to tables and to other indexes, though.
High #of executions are also due to concurrent merges and could of merges (small volume) are having Nested-loop in PK.Are you using the merge command with a high volume on each call, or have you done something like creating a loop that does a uses the merge with one row of data at a time, or does "merge" mean something other than the merge command ? The merge command exists as an efficient mechanism for merging a large volume of data into a VERY large volume of data with a single SQL operation.
I have 16 cores and 64 GB RAM.
May i ask you how Oracle derives '_enqueue_locks' parameter value ? I have transactions=150,sessions=500,processes=450 and dml_locks=60. Can you advice me whether i should increase this hidden parameter or what ? If yes, would there by any sub sequences ?
I don't know how enqueuelocks is derived. It used to be something like: 10 * sessions + 2 * processes + db_files + a couple of other things
Why do you have transactions set lower than processes, and how come your dml_locks parameter is so low. Transactions defaults to something like sessions + 10%, and dml_locks defaults to something like 5 * processes -- those are both ballpark figures, I can't remember exactly what the rules are.
I showed you in a previous post how to check if anything needed changing. Query v$resource_limit and check the max_allocation with the limit. It's not often that anyone uses anything like the limit set for enqueue_locks, but possibly you mess
Your comment on commit rate is also interesting (user commits/sec is just 1.41) . Yes, our merges are row-by-row processing and i am sure job is putting COMMIT once entire merge is executed. Is that an optimal load design ?I think this answer the question further up the page - is your "merge" a real merge command or a "update, if no rows updated then insert pl/sql".
Since 'log file sync' is not a problematic area, i think LGWR is neither starving for CPU nor for Memory. I am sure my IO is also good.I wasn't worried about your commit rate - I was curious that you did a small number of commits with a very large number of executions. There are no indications in stats you've shown me to suggest that the log writer has, or is, a problem. However you are doing too much I/O and need to reduce it if you want to make your processes go faster.
Regards
Jonathan Lewis