Tuning row lock contention wait events

Question

Hello everyone,
Working on 10g/windows
Top 5 events
EVENT TOTAL_WAITS TIME_WAITED AVG_MS PERCENT
CPU 9462339 48
enq: TX - row lock contention 12531 3660728 2921.34 18
control file parallel write 1300731 3088079 23.74 16
log file parallel write 1510503 1264080 8.37 6
log file sync 1072553 968007 9.03 5
Distribution of row lock wait during the last 4 days in the database server
END_INTERVAL_TIME TOTAL_WAITS TIME_WAITED_MICRO AVG_WAIT_MS
2008-04-01 16:00:58 909 2721008230 2993.41
2008-04-01 15:00:27 50 149941140 2998.82
2008-03-31 12:00:42 193 575595397 2982.36
2008-03-29 23:00:13 172 513058700 2982.9
2008-03-29 22:00:37 164 483940046 2950.85
2008-03-27 22:00:35 565 1667120838 2950.66
2008-03-26 18:00:59 348 1042918982 2996.89
My analysis:
It's obvious that the row lock contention wait time is huge, and this direct me to find out SQL stmt, causing this.
all the SQL statement was SELECT ....... FOR UPDATE stmt.
I was also able to find out locked tables.
My tuning idea:
1. I'm thinking to reorganize hot tables as well as their indexes, but by instinct it seems to not give so much value to avoid the huge row lock wait time.
2. I'm also seeing if I can reduce the number of rows per block, by increasing PCTFREE and diminishing PCTUSED, so the contention will spread over many blocks instead of one heavy block.
Question
As SQL stmt related to those locked tables are select ... for update, how could I tune this kind of stmt?
Does someone have other idea to come up with this row lock contention?
Tanks for your effort and help

Answer 1

Taking another look at your suggested function based index, it depends on the data type of the DEV.POS_FOLIO_ID.POS_FOLIO_ID column. If the column is defined as a number, and it is a primary key, there will already be a usable index on that column.
Yesterday, I wrote this: "Once I understood why or how the sessions were trying to insert duplicate primary key values, I would try to determine why the average number of seconds for the wait event is almost 3 seconds (maybe a timeout)."
After fixing the formatting of the top 5 wait events (total duration unknown):
EVENT                        TOTAL_WAITS  TIME_WAITED   AVG_MS PERCENT
CPU                                         94,623.39             48
enq: TX - row lock contention     12,531    36,607.28  2921.34    18
control file parallel write    1,300,731    30,880.79    23.74    16
log file parallel write        1,510,503    12,640.80     8.37     6
log file sync                  1,072,553     9,680.07     9.03     512,531 * 3 second time out = 37,593 seconds = 10.44 hours.
What if the reason for the 3 second average wait time is due to a timeout. I performed a little experiment... I changed a row in a test table and then made a pot of coffee.
In session 1:
CREATE TABLE T1 (
  C1 NUMBER(10),
  C2 NUMBER(10),
  PRIMARY KEY (C1));
INSERT INTO T1
SELECT
  ROWNUM,
  ROWNUM*10
FROM
  DUAL
CONNECT BY
  LEVEL<=1000000;
COMMIT;I now have a test table with 1,000,000 rows. I start monitoring the changes in the wait events roughly every 60 seconds, and V$SESSION_WAIT and V$LOCK roughly 4 times per second.
Back in session 1:
UPDATE
  T1
SET
  C1=-C1
WHERE
  C1<=100;I have now modified the first 100 rows that were inserted into the table, time to make the pot of coffee.
In session 2, I try to insert a row with a primary key value of -10:
INSERT INTO T1 VALUES (
  -10,
  10);Session 2 hangs.
If I take the third 60 second snap of the system wide wait events as the zero point, and the 11th snap as the end point. There were 149 waits on ENQ: TX - ROW LOCK CONTENTION, 148 time outs, 446.62 seconds of total time in the wait event, with an average wait time of 2.997450 seconds.
Rolling down to the session level wait events, SID 208 (my session 2) had 149 waits on ENQ: TX - ROW LOCK CONTENTION, for a total time of 446.61 seconds with an average wait time of 2.997383 seconds. All of the 149 waits and the wait time was in this one session that was locked up for the full duration of this time period because session 1 was making a pot of coffee.
Rolling down to V$SESSION_WAIT (sampled roughly 4 times per second): At the start of the third time interval, SID 208 has been in the ENQ: TX - ROW LOCK CONTENTION wait event for 39 seconds and is actively waiting trying to execute SQL with a hash value of 1001532423, the wait object is -1, wait file is 0, wait block is 0, wait row is 0, P1 is 1415053316, P2 is 196646, P3 is 4754.
At the end of the 11th time interval: , SID 208 has been in the ENQ: TX - ROW LOCK CONTENTION wait event for 483 seconds and is actively waiting trying to execute SQL with a hash value of 1001532423, the wait object is -1, wait file is 0, wait block is 0, wait row is 0, P1 is 1415053316, P2 is 196646, P3 is 4754.
Rolling down to V$LOCK (sampled roughly 4 times per second): I see that SID 214 (session 1) is blocking SID 208 (session 2). SID 214 has a TX lock in mode 6 with ID1 of 196646 and ID2 of 4754. SID 208 is requesting a TX lock in mode 4 with ID1 of 196646 and ID2 of 4754.
So, it seems that I need a faster coffee pot rather than an additional index on my table. It could be that the above process would have found that the application associated with SID 214 was abandoned or crashed and for some reason the lock was not released for a long period of time, a little less than 10.44 hours in your case.
Charles Hooper
IT Manager/Oracle DBA
K&M Machine-Fabricating, Inc.