Get blocker from the (self) deadlock trace file

Question

Hi,
Recently I had an issue on a 10.2.0.4 single instance database where deadlocks were occurring. The following test case reproduces the problem (I create three parent tables, one child table with indexed foreign keys to all three parent tables and a procedure which performs an insert into the child table in an autonomous transaction):
create table parent_1(id number primary key);
create table parent_2(id number primary key);
create table parent_3(id number primary key);
create table child( id_c number primary key,
                   id_p1 number,
                   id_p2 number,
                   id_p3 number,
                   constraint fk_id_p1 foreign key (id_p1) references parent_1(id),
                   constraint fk_id_p2 foreign key (id_p2) references parent_2(id),
                   constraint fk_id_p3 foreign key (id_p3) references parent_3(id)
create index i_id_p1 on child(id_p1);
create index i_id_p2 on child(id_p2);
create index i_id_p3 on child(id_p3);
create or replace procedure insert_into_child as
pragma autonomous_transaction;
begin
  insert into child(id_c, id_p1, id_p2, id_p3) values(1,1,1,1);
  commit;
end;
insert into parent_1 values(1);
insert into parent_2 values(1);
commit;And now the action that causes the deadlock:
SQL> insert into parent_3 values(1);
1 row created.
SQL> exec insert_into_child;
BEGIN insert_into_child; END;
ERROR at line 1:
ORA-00060: deadlock detected while waiting for resource
ORA-06512: at "SCOTT.INSERT_INTO_CHILD", line 4
ORA-06512: at line 1My question is: how can I determine which table the insert into CHILD was waiting on? It could be waiting on PARENT_1, PARENT_2, PARENT_3, a combination of them or even on CHILD if I tried to insert a duplicate primary key in CHILD. Since we have the full testcase we know that it was waiting on PARENT_3 (or better said, it was waiting for the "parent" transaction to perform a commit/rollback), but is it possible to determine that solely from the deadlock trace file? I'm asking that because to pinpoint the problem I had to perform redo log mining, pl/sql tracing with DBMS_TRACE and manual debugging on a clone of the production database which was restored to a SCN just before the deadlock occurred. So, I had to do quite a lot of work to get to the blocker table and if this information is already in the deadlock trace file, it would have saved me a lot of time.
Below is the deadlock trace file. From the "DML LOCK" part I guess that the child table (tab=227042) holds a mode 3 lock (SX), all the other three parent tables have mode 2 locks (SS), but from this extract I can't see that parent_3 (tab=227040) is blocking the insert into child:
Deadlock graph:
                       ---------Blocker(s)--------  ---------Waiter(s)---------
Resource Name          process session holds waits  process session holds waits
TX-00070029-00749150        23     476     X             23     476           S
session 476: DID 0001-0017-00000003     session 476: DID 0001-0017-00000003
Rows waited on:
Session 476: obj - rowid = 000376E2 - AAA3biAAEAAA4BwAAA
  (dictionary objn - 227042, file - 4, block - 229488, slot - 0)
Information on the OTHER waiting sessions:
End of information on OTHER waiting sessions.
Current SQL statement for this session:
INSERT INTO CHILD(ID_C, ID_P1, ID_P2, ID_P3) VALUES(1,1,1,1)
----- PL/SQL Call Stack -----
  object      line  object
  handle    number  name
3989eef50         4  procedure SCOTT.INSERT_INTO_CHILD
391f3d870         1  anonymous block
        SO: 397691978, type: 36, owner: 39686af98, flag: INIT/-/-/0x00
        DML LOCK: tab=227042 flg=11 chi=0
                  his[0]: mod=3 spn=35288
        (enqueue) TM-000376E2-00000000  DID: 0001-0017-00000003
        lv: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  res_flag: 0x6
        res: 0x398341fe8, mode: SX, lock_flag: 0x0
        own: 0x3980df420, sess: 0x3980df420, proc: 0x39859c660, prv: 0x398341ff8
        SO: 397691878, type: 36, owner: 39686af98, flag: INIT/-/-/0x00
        DML LOCK: tab=227040 flg=11 chi=0
                  his[0]: mod=2 spn=35288
        (enqueue) TM-000376E0-00000000  DID: 0001-0017-00000003
        lv: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  res_flag: 0x6
        res: 0x3983386e8, mode: SS, lock_flag: 0x0
        own: 0x3980df420, sess: 0x3980df420, proc: 0x39859c660, prv: 0x3983386f8
        SO: 397691778, type: 36, owner: 39686af98, flag: INIT/-/-/0x00
        DML LOCK: tab=227038 flg=11 chi=0
                  his[0]: mod=2 spn=35288
        (enqueue) TM-000376DE-00000000  DID: 0001-0017-00000003
        lv: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  res_flag: 0x6
        res: 0x398340f58, mode: SS, lock_flag: 0x0
        own: 0x3980df420, sess: 0x3980df420, proc: 0x39859c660, prv: 0x398340f68
        SO: 397691678, type: 36, owner: 39686af98, flag: INIT/-/-/0x00
        DML LOCK: tab=227036 flg=11 chi=0
                  his[0]: mod=2 spn=35288
        (enqueue) TM-000376DC-00000000  DID: 0001-0017-00000003
        lv: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  res_flag: 0x6
        res: 0x39833f358, mode: SS, lock_flag: 0x0
        own: 0x3980df420, sess: 0x3980df420, proc: 0x39859c660, prv: 0x39833f368
      ----------------------------------------Thank you in advance for any comments,
Jure

Answer 1

Hi Jonathan,
thank you very much for your reply which more than answers my question. I think it actually clears a lot of doubts I had about TX locks, since your mentioning of "undo segment header transaction table" pointed me in the right direction for further research on this topic (honestly, I didn't know what's "behind" TX locks). So if I understood correctly, to determine which table is the blocker (in the testcase presented above), you have to have some kind of history of executed SQL statements (e.g. by mining redo logs)?
The statement you wrote:
At this point, and with your example, the waiting session is waiting on a TX (transaction) lock - this means it has not idea (and no interest) in the actual data involved, it is merely waiting for an undo segment header transaction table slot to clear. and the example with the savepoint you gave, made me think of some of the consequences of that behaviour. That is probably the reason why it is not possible to get the "blocker" table from v$lock (although sometimes it's possible to get it from v$session.row_wait_obj#) when a session tries to change a row another session holds in exclusive mode, e.g.:
create table t1 (id number);
insert into t1 values (1);
commit;
Session 126:
SID = 126> update t1 set id=2 where id=1;
1 row updated.
Session 146:
SID = 146> update t1 set id=2 where id=1;
{session hangs}
In a separate session:
SQL> SELECT   CASE
  2                  WHEN TYPE = 'TM'
  3                     THEN (SELECT object_name
  4                             FROM user_objects
  5                            WHERE object_id = l.id1)
  6               END object_name,
  7                  SID, TYPE, id1, id2, lmode, request, BLOCK
  8          FROM v$lock l
  9         WHERE SID IN (126, 146)
10     ORDER BY SID, TYPE, 1
11  /
OBJECT_NAME    SID TY        ID1        ID2      LMODE    REQUEST      BLOCK
T1             126 TM      68447          0          3          0          0
               126 TX     262153       4669          6          0          1
T1             146 TM      68447          0          3          0          0
               146 TX     262153       4669          0          6          0The only thing I can tell from this output is that session 146 is trying to get a TX lock in exclusive mode, and session 126 is blocking it, the reason of the blocking being unknown from this view alone.
Since I'd like to get a better understanding on the mechanics behind this (e.g. why the blocked session can't know the segment that is waiting for, since it has to go to the same segment's data block to find the address of the undo segment header transaction table slot? ; can we get the content/structure of the transaction table in the data block - probably by making a block dump?), do you have any source where a more in depth explanation what happens "behind the scenes" is available (perhaps in Oracle Core?)? Some time ago I found a link on your blog http://jonathanlewis.wordpress.com/2010/06/21/locks/ which points to Franck Pachot's article where he nicely explains the various locking modes: http://knol.google.com/k/oracle-table-lock-modes#. There I also found Kyle Hailey's presentation about locks http://www.perfvision.com/papers/09_enqueues.ppt where slide 23 nicely depicts what's going on when acquiring TX locks. Of course I'll try to search on my own, but any other source (especially from an authority like you) is more than welcome.
Thank you again and regards,
Jure