Deadlocks and very high wait times

Question

We are seeing a very high number of deadlocks in the system. The deadlocks trace all show a 'enq: TX - row lock contention' with wait times of around 2929700+ seconds ex:
last wait for 'enq: TX - row lock contention' blocking sess=0x70000006d85e1b8 seq=55793 wait_time=2929704 seconds since wait started=4
name|mode=54580006, usn<<16 | slot=1d0010, sequence=705f
Dumping Session Wait History
for 'enq: TX - row lock contention' count=1 wait_time=2929704
name|mode=54580006, usn<<16 | slot=1d0010, sequence=705f
for 'latch: enqueue hash chains' count=1 wait_time=1649
address=70000006dbb4a20, number=13, tries=0
for 'enq: TX - row lock contention' count=1 wait_time=2929708
name|mode=54580006, usn<<16 | slot=1d0010, sequence=705f
for 'SQL*Net message from client' count=1 wait_time=101740
driver id=54435000, #bytes=1, =0
for 'SQL*Net message to client' count=1 wait_time=1
driver id=54435000, #bytes=1, =0
for 'direct path write temp' count=1 wait_time=921
file number=fb, first dba=6521b, block cnt=2
for 'SQL*Net more data from client' count=1 wait_time=3
driver id=54435000, #bytes=10, =0
for 'SQL*Net more data from client' count=1 wait_time=5
driver id=54435000, #bytes=1e, =0
for 'SQL*Net more data from client' count=1 wait_time=10
driver id=54435000, #bytes=2c, =0
for 'SQL*Net more data from client' count=1 wait_time=5
driver id=54435000, #bytes=3a, =0
Any ideas on how to resolve this ?
Thanks
Surya

Answer 1

Sorry for the typo, that Ora-0060 error we are seeing. Here is the deadlock graph:
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, Oracle Label Security, OLAP, Data Mining Scoring Engine
and Real Application Testing options
ORACLE_HOME = /orasw/product/10.2.0.4.0
System name: AIX
Node name: spda5001
Release: 3
Version: 5
Machine: 00074D5AD400
Instance name: IAMS01P1
Redo thread mounted by this instance: 1
Oracle process number: 21
Unix process pid: 2306444, image: oracle@spda5001
*** 2011-12-24 05:05:39.885
*** SERVICE NAME:(IAMS01P) 2011-12-24 05:05:39.884
*** SESSION ID:(443.2130) 2011-12-24 05:05:39.884
DEADLOCK DETECTED ( ORA-00060 )
[Transaction Deadlock]
The following deadlock is not an ORACLE error. It is a
deadlock due to user error in the design of an application
or from issuing incorrect ad-hoc SQL. The following
information may aid in determining the deadlock:
Deadlock graph:
---------Blocker(s)-------- ---------Waiter(s)---------
Resource Name process session holds waits process session holds waits
TX-00080020-000c3957 21 443 X 58 391 X
TX-001d0010-0000705f 58 391 X 21 443 X
session 443: DID 0001-0015-0000002E session 391: DID 0001-003A-00000081
session 391: DID 0001-003A-00000081 session 443: DID 0001-0015-0000002E
Rows waited on:
Session 391: obj - rowid = 0001098B - AAATtpAAGAAADROAAD
(dictionary objn - 67979, file - 6, block - 13390, slot - 3)
Session 443: obj - rowid = 00010B25 - AAARRwAAGAAAAdgAAN
(dictionary objn - 68389, file - 6, block - 1888, slot - 13)
Information on the OTHER waiting sessions:
Session 391:
pid=58 serial=16572 audsid=52790041 user: 93/IAMS_USR
O/S info: user: , term: , ospid: 1234, machine: mac3023
program:
Current SQL Statement:
update spt_identity set created=:1, modified=:2, owner=:3, assigned_scope=:4, assigned_scope_path=:5, extended1=:6, extended2=:7, extended3=:8, extended4=:9, extended5=:10, extended6=:11, extended7=:12, extended8=:13, extended9=:14, extended10=:15, extended11=:16, extended12=:17, extended13=:18, extended14=:19, extended15=:20, extended16=:21, extended17=:22, extended18=:23, extended19=:24, extended20=:25, name=:26, description=:27, protected=:28, iiqlock=:29, attributes=:30, manager=:31, display_name=:32, firstname=:33, lastname=:34, email=:35, manager_status=:36, inactive=:37, last_login=:38, last_refresh=:39, password=:40, password_expiration=:41, password_history=:42, bundle_summary=:43, assigned_role_summary=:44, correlated=:45, auth_question_lock_start=:46, failed_auth_question_attempts=:47, controls_assigned_scope=:48, certifications=:49, activity_config=:50, preferences=:51, history=:52, scorecard=:53, uipreferences=:54, attribute_meta_data=:55, workgroup=:56 where id=:57
End of information on OTHER waiting sessions.
Current SQL statement for this session:
update spt_workflow_case set created=:1, modified=:2, owner=:3, assigned_scope=:4, assigned_scope_path=:5, stack=:6, attributes=:7, launcher=:8, host=:9, launched=:10, completed=:11, progress=:12, percent_complete=:13, type=:14, messages=:15, name=:16, description=:17, complete=:18, target_class=:19, target_id=:20, target_name=:21, workflow=:22 where id=:23