Ref) heavy lock contention during object serialization
During load testing of our application we faced heavy contention in ReferenceQueue.poll. Using Java 1.5.0_12. Following is the stack trace
"pool-1-thread-23" prio=10 tid=0x00282d18 nid=0x41 waiting for monitor
entry [0xb07fe000..0xb07ffaf0]
at java.lang.ref.ReferenceQueue.poll(ReferenceQueue.java:82)
- waiting to lock <0xe7569aa0> (a
java.lang.ref.ReferenceQueue$Lock)
at
java.io.ObjectStreamClass.processQueue(ObjectStreamClass.java:2206)
at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:253)
at
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1035)
at
java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:302)
"pool-1-thread-29" prio=10 tid=0x002895a0 nid=0x47 waiting for monitor
entry [0xb01fe000..0xb01ffbf0]
at java.lang.ref.ReferenceQueue.poll(ReferenceQueue.java:82)
- waiting to lock <0xe7569aa0> (a
java.lang.ref.ReferenceQueue$Lock)
at
java.io.ObjectStreamClass.processQueue(ObjectStreamClass.java:2206)
at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:253)
at
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1035)
at
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:13
75)
at
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1347)
at
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1
290)
at
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
at
java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:302)
Now the same case was issued as a bug 6525425 and is fixed in u14. The problem being now that u14 is not available for download. So is there a workaround for this issue as it is affecting our load testing.
Hi Amit!
I saw you are using FOR UPDATE in your query.
If you want reduce row locks, then you not must use FOR UPDATE.
As you know, FOR UPDATE is locking rows.
Regards
Mahir M. Quluzade
Similar Messages
-
Guys,
I really appreciate your views on this.. Please can some one who have worked on RAC and have an understanding how RAC works, guide me.
We currently are running a loadtest on one of the new RAC system and we are seeing excessive row lock contention for one table. The table basically has very few rows, say about 6-8 and pretty much every user uses this table to lock rows before fetching some data from other tables. When have a heavy load, we see very high wait on this table and enq TX : Row lock contention.
What is the best way to avoid this ? Is there anyway, we can modify the design of the application and ensure locking.. The typical query that locks the rows in the table looks like this
SELECT WL0.CLUB_NAME, WL0.SCHEDULE_ID FROM VF_BINGO_NEXT_CLUB_DRAW WL0 WHERE ( WL0.CLUB_NAME = :1 ) FOR UPDATE As said, they table has only 6-8 rows, so query plan etc doesn't apply. Please can someone who have extensive application knowledge guide me thru ?
Many thanks in advance.
G> But is there a way to alliviate the concurrency issue, if there is a genuine
requirement to hold lock on a table. What other options do we have other
than locking ?
The purpose of a lock is to ensure data consistency - only 1 process can change that row. So what I find puzzling is why so many app sessions want to change that single row. What data does that row hold that requires continual change? Just what is the purpose of this data if it is consistently and continually changing? What business requirement does it attempt to solve?
To be honest, this sounds like a major design problem to me.
Like the surrogate key generator approach I mentioned. You create a PK_SEQUENCE table with columns (tablename, pk_value). Any insert against a table requires a lock on PK_SEQUENCE for the table being inserted into - a read of the PK_VALUE, incrementing it by one and then updating that row with the latest sequence. This way each INSERT gets a "nice sequential number" to use as surrogate key for new rows.
Even when this is done as an autonomous transaction, it introduces a very expensive resource - why? Because only one session at a time can be serviced by that resource.
If there are a lot of INSERTs into a table, this approach will quickly become a severe bottleneck as every single insert requires a new surrogate key value and a lock on that PK_SEQUENCE table to obtain that value.
This problem is solved by allowing/enabling such a resource to serve multiple sessions concurrently. Which is for example what Oracle Sequence objects do within this example I've described.
The bottom line is that the design you describe introduced a bottleneck by create an expensive and serialised resource that can only serve a single session at a time.
You need to re-look at the business requirement - and find another way to solve it than to introduce this type of serialised resource and contention. -
Object serialization failure during OutOfMemory error (EOFException)
Hello all,
We are seeing a very strange situation when writing a serializable object to a flat file, specifically during an OutOfMemory condition. Our application saves the state of an object if an error occurs, and retries at a later point by reviving the object from its serialized form. We recently encountered a series of EOFExcetions when trying to reload the serialized object. Looking at the serialized data, we see that the file is indeed incomplete, and that it appears to be the serialized representation of the data contained within the object that is missing (the structure of the object appears to be stored in the serialized file, but not the runtime data).
The code that produces and consumes these serialized objects is used in a variety of locations, and even in the case where these EOF errors occurred usually works as expected. The difference appears to be that the error condition which lead to the object serialization was in-fact an OutOfMemory error. That is, we had an OOM error elsewhere in our application, which caused our objects to be serialized to disk. During this serialization process we end up with corrupt (incomplete) serialization data.
So.. the question is: Is it possible that the OutOfMemory situation causes the JVM to incorrectly serialize an object (without an error), and specifically to omit runtime data (variables) from the serialized form of the object?
Thanks/jasonpolites wrote:
So.. the question is: Is it possible that the OutOfMemory situation causes the JVM to incorrectly serialize an object (without an error), and specifically to omit runtime data (variables) from the serialized form of the object?you're sure that the serialization process completed without an error? how do you know the the OOME did not cause the serialization to fail (because the serialization process itself will require more memory, hence if it is happening while the system is near the memory peak, it is likely to fail as well)? generally, when a jvm starts throwing OOME's all bets are off. failures in random places can easily cause the whole internal state of a system to become invalid. -
BDB read performance problem: lock contention between GC and VM threads
Problem: BDB read performance is really bad when the size of the BDB crosses 20GB. Once the database crosses 20GB or near there, it takes more than one hour to read/delete/add 200K keys.
After a point, of these 200K keys there are about 15-30K keys that are new and this number eventually should come down and there should not be any new keys after a point.
Application:
Transactional Data Store application. Single threaded process, that's trying to read one key's data, delete the data and add new data. The keys are really small (20 bytes) and the data is large (grows from 1KB to 100KB)
On on machine, I have a total of 3 processes running with each process accessing its own BDB on a separate RAID1+0 drive. So, according to me there should really be no disk i/o wait that's slowing down the reads.
After a point (past 20GB), There are about 4-5 million keys in my BDB and the data associated with each key could be anywhere between 1KB to 100KB. Eventually every key will have 100KB data associated with it.
Hardware:
16 core Intel Xeon, 96GB of RAM, 8 drive, running 2.6.18-194.26.1.0.1.el5 #1 SMP x86_64 x86_64 x86_64 GNU/Linux
BDB config: BTREE
bdb version: 4.8.30
bdb cache size: 4GB
bdb page size: experimented with 8KB, 64KB.
3 processes, each process accesses its own BDB on a separate RAIDed(1+0) drive.
envConfig.setAllowCreate(true);
envConfig.setTxnNoSync(ourConfig.asynchronous);
envConfig.setThreaded(true);
envConfig.setInitializeLocking(true);
envConfig.setLockDetectMode(LockDetectMode.DEFAULT);
When writing to BDB: (Asynchrounous transactions)
TransactionConfig tc = new TransactionConfig();
tc.setNoSync(true);
When reading from BDB (Allow reading from Uncommitted pages):
CursorConfig cc = new CursorConfig();
cc.setReadUncommitted(true);
BDB stats: BDB size 49GB
$ db_stat -m
3GB 928MB Total cache size
1 Number of caches
1 Maximum number of caches
3GB 928MB Pool individual cache size
0 Maximum memory-mapped file size
0 Maximum open file descriptors
0 Maximum sequential buffer writes
0 Sleep after writing maximum sequential buffers
0 Requested pages mapped into the process' address space
2127M Requested pages found in the cache (97%)
57M Requested pages not found in the cache (57565917)
6371509 Pages created in the cache
57M Pages read into the cache (57565917)
75M Pages written from the cache to the backing file (75763673)
60M Clean pages forced from the cache (60775446)
2661382 Dirty pages forced from the cache
0 Dirty pages written by trickle-sync thread
500593 Current total page count
500593 Current clean page count
0 Current dirty page count
524287 Number of hash buckets used for page location
4096 Assumed page size used
2248M Total number of times hash chains searched for a page (2248788999)
9 The longest hash chain searched for a page
2669M Total number of hash chain entries checked for page (2669310818)
0 The number of hash bucket locks that required waiting (0%)
0 The maximum number of times any hash bucket lock was waited for (0%)
0 The number of region locks that required waiting (0%)
0 The number of buffers frozen
0 The number of buffers thawed
0 The number of frozen buffers freed
63M The number of page allocations (63937431)
181M The number of hash buckets examined during allocations (181211477)
16 The maximum number of hash buckets examined for an allocation
63M The number of pages examined during allocations (63436828)
1 The max number of pages examined for an allocation
0 Threads waited on page I/O
0 The number of times a sync is interrupted
Pool File: lastPoints
8192 Page size
0 Requested pages mapped into the process' address space
2127M Requested pages found in the cache (97%)
57M Requested pages not found in the cache (57565917)
6371509 Pages created in the cache
57M Pages read into the cache (57565917)
75M Pages written from the cache to the backing file (75763673)
$ db_stat -l
0x40988 Log magic number
16 Log version number
31KB 256B Log record cache size
0 Log file mode
10Mb Current log file size
856M Records entered into the log (856697337)
941GB 371MB 67KB 112B Log bytes written
2GB 262MB 998KB 478B Log bytes written since last checkpoint
31M Total log file I/O writes (31624157)
31M Total log file I/O writes due to overflow (31527047)
97136 Total log file flushes
686 Total log file I/O reads
96414 Current log file number
4482953 Current log file offset
96414 On-disk log file number
4482862 On-disk log file offset
1 Maximum commits in a log flush
1 Minimum commits in a log flush
160KB Log region size
195 The number of region locks that required waiting (0%)
$ db_stat -c
7 Last allocated locker ID
0x7fffffff Current maximum unused locker ID
9 Number of lock modes
2000 Maximum number of locks possible
2000 Maximum number of lockers possible
2000 Maximum number of lock objects possible
160 Number of lock object partitions
0 Number of current locks
1218 Maximum number of locks at any one time
5 Maximum number of locks in any one bucket
0 Maximum number of locks stolen by for an empty partition
0 Maximum number of locks stolen for any one partition
0 Number of current lockers
8 Maximum number of lockers at any one time
0 Number of current lock objects
1218 Maximum number of lock objects at any one time
5 Maximum number of lock objects in any one bucket
0 Maximum number of objects stolen by for an empty partition
0 Maximum number of objects stolen for any one partition
400M Total number of locks requested (400062331)
400M Total number of locks released (400062331)
0 Total number of locks upgraded
1 Total number of locks downgraded
0 Lock requests not available due to conflicts, for which we waited
0 Lock requests not available due to conflicts, for which we did not wait
0 Number of deadlocks
0 Lock timeout value
0 Number of locks that have timed out
0 Transaction timeout value
0 Number of transactions that have timed out
1MB 544KB The size of the lock region
0 The number of partition locks that required waiting (0%)
0 The maximum number of times any partition lock was waited for (0%)
0 The number of object queue operations that required waiting (0%)
0 The number of locker allocations that required waiting (0%)
0 The number of region locks that required waiting (0%)
5 Maximum hash bucket length
$ db_stat -CA
Default locking region information:
7 Last allocated locker ID
0x7fffffff Current maximum unused locker ID
9 Number of lock modes
2000 Maximum number of locks possible
2000 Maximum number of lockers possible
2000 Maximum number of lock objects possible
160 Number of lock object partitions
0 Number of current locks
1218 Maximum number of locks at any one time
5 Maximum number of locks in any one bucket
0 Maximum number of locks stolen by for an empty partition
0 Maximum number of locks stolen for any one partition
0 Number of current lockers
8 Maximum number of lockers at any one time
0 Number of current lock objects
1218 Maximum number of lock objects at any one time
5 Maximum number of lock objects in any one bucket
0 Maximum number of objects stolen by for an empty partition
0 Maximum number of objects stolen for any one partition
400M Total number of locks requested (400062331)
400M Total number of locks released (400062331)
0 Total number of locks upgraded
1 Total number of locks downgraded
0 Lock requests not available due to conflicts, for which we waited
0 Lock requests not available due to conflicts, for which we did not wait
0 Number of deadlocks
0 Lock timeout value
0 Number of locks that have timed out
0 Transaction timeout value
0 Number of transactions that have timed out
1MB 544KB The size of the lock region
0 The number of partition locks that required waiting (0%)
0 The maximum number of times any partition lock was waited for (0%)
0 The number of object queue operations that required waiting (0%)
0 The number of locker allocations that required waiting (0%)
0 The number of region locks that required waiting (0%)
5 Maximum hash bucket length
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Lock REGINFO information:
Lock Region type
5 Region ID
__db.005 Region name
0x2accda678000 Region address
0x2accda678138 Region primary address
0 Region maximum allocation
0 Region allocated
Region allocations: 6006 allocations, 0 failures, 0 frees, 1 longest
Allocations by power-of-two sizes:
1KB 6002
2KB 0
4KB 0
8KB 0
16KB 1
32KB 0
64KB 2
128KB 0
256KB 1
512KB 0
1024KB 0
REGION_JOIN_OK Region flags
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Lock region parameters:
524317 Lock region region mutex [0/9 0% 5091/47054587432128]
2053 locker table size
2053 object table size
944 obj_off
226120 locker_off
0 need_dd
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Lock conflict matrix:
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Locks grouped by lockers:
Locker Mode Count Status ----------------- Object ---------------
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Locks grouped by object:
Locker Mode Count Status ----------------- Object ---------------
Diagnosis:
I'm seeing way to much lock contention on the Java Garbage Collector threads and also the VM thread when I strace my java process and I don't understand the behavior.
We are spending more than 95% of the time trying to acquire locks and I don't know what these locks are. Any info here would help.
Earlier I thought the overflow pages were the problem as 100KB data size was exceeding all overflow page limits. So, I implemented duplicate keys concept by chunking of my data to fit to overflow page limits.
Now I don't see any overflow pages in my system but I still see bad bdb read performance.
$ strace -c -f -p 5642 --->(607 times the lock timed out, errors)
Process 5642 attached with 45 threads - interrupt to quit
% time seconds usecs/call calls errors syscall
98.19 7.670403 2257 3398 607 futex
0.84 0.065886 8 8423 pread
0.69 0.053980 4498 12 fdatasync
0.22 0.017094 5 3778 pwrite
0.05 0.004107 5 808 sched_yield
0.00 0.000120 10 12 read
0.00 0.000110 9 12 open
0.00 0.000089 7 12 close
0.00 0.000025 0 1431 clock_gettime
0.00 0.000000 0 46 write
0.00 0.000000 0 1 1 stat
0.00 0.000000 0 12 lseek
0.00 0.000000 0 26 mmap
0.00 0.000000 0 88 mprotect
0.00 0.000000 0 24 fcntl
100.00 7.811814 18083 608 total
The above stats show that there is too much time spent locking (futex calls) and I don't understand that because
the application is really single-threaded. I have turned on asynchronous transactions so the writes might be
flushed asynchronously in the background but spending that much time locking and timing out seems wrong.
So, there is possibly something I'm not setting or something weird with the way JVM is behaving on my box.
I grep-ed for futex calls in one of my strace log snippet and I see that there is a VM thread that grabbed the mutex
maximum number(223) of times and followed by Garbage Collector threads: the following is the lock counts and thread-pids
within the process:
These are the 10 GC threads (each thread has grabbed lock on an avg 85 times):
86 [8538]
85 [8539]
91 [8540]
91 [8541]
92 [8542]
87 [8543]
90 [8544]
96 [8545]
87 [8546]
97 [8547]
96 [8548]
91 [8549]
91 [8550]
80 [8552]
VM Periodic Task Thread" prio=10 tid=0x00002aaaf4065000 nid=0x2180 waiting on condition (Main problem??)
223 [8576] ==> grabbing a lock 223 times -- not sure why this is happening…
"pool-2-thread-1" prio=10 tid=0x00002aaaf44b7000 nid=0x21c8 runnable [0x0000000042aa8000] -- main worker thread
34 [8648] (main thread grabs futex only 34 times when compared to all the other threads)
The load average seems ok; though my system thinks it has very less memory left and that
I think is because its using up a lot of memory for the file system cache?
top - 23:52:00 up 6 days, 8:41, 1 user, load average: 3.28, 3.40, 3.44
Tasks: 229 total, 1 running, 228 sleeping, 0 stopped, 0 zombie
Cpu(s): 3.2%us, 0.9%sy, 0.0%ni, 87.5%id, 8.3%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 98999820k total, 98745988k used, 253832k free, 530372k buffers
Swap: 18481144k total, 1304k used, 18479840k free, 89854800k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
8424 rchitta 16 0 7053m 6.2g 4.4g S 18.3 6.5 401:01.88 java
8422 rchitta 15 0 7011m 6.1g 4.4g S 14.6 6.5 528:06.92 java
8423 rchitta 15 0 6989m 6.1g 4.4g S 5.7 6.5 615:28.21 java
$ java -version
java version "1.6.0_21"
Java(TM) SE Runtime Environment (build 1.6.0_21-b06)
Java HotSpot(TM) 64-Bit Server VM (build 17.0-b16, mixed mode)
Maybe I should make my application a Concurrent Data Store app as there is really only one thread doing the writes and reads. But I would like
to understand why my process is spending so much time in locking.
Can I try any other options? How do I prevent such heavy locking from happening? Has anyone seen this kind of behavior? Maybe this is
all normal. I'm pretty new to using BDB.
If there is a way to disable locking that would also work as there is only one thread that's really doing all the job.
Should I disable the file system cache? One thing is that my application does not utilize cache very well as once I visit a key, I don't visit that
key again for a very long time so its very possible that the key has to be read again from the disk.
It is possible that I'm thinking this completely wrong and focussing too much on locking behavior and the problem is else where.
Any thoughts/suggestions etc are welcome. Your help on this is much appreciated.
Thanks,
RamaHi,
Looks like you're using BDB, not BDB JE, and this is the BDB JE forum. Could you please repost here?:
Berkeley DB
Thanks,
mark -
Tuning row lock contention wait events
Hello everyone,
Working on 10g/windows
Top 5 events
EVENT TOTAL_WAITS TIME_WAITED AVG_MS PERCENT
CPU 9462339 48
enq: TX - row lock contention 12531 3660728 2921.34 18
control file parallel write 1300731 3088079 23.74 16
log file parallel write 1510503 1264080 8.37 6
log file sync 1072553 968007 9.03 5
Distribution of row lock wait during the last 4 days in the database server
END_INTERVAL_TIME TOTAL_WAITS TIME_WAITED_MICRO AVG_WAIT_MS
2008-04-01 16:00:58 909 2721008230 2993.41
2008-04-01 15:00:27 50 149941140 2998.82
2008-03-31 12:00:42 193 575595397 2982.36
2008-03-29 23:00:13 172 513058700 2982.9
2008-03-29 22:00:37 164 483940046 2950.85
2008-03-27 22:00:35 565 1667120838 2950.66
2008-03-26 18:00:59 348 1042918982 2996.89
My analysis:
It's obvious that the row lock contention wait time is huge, and this direct me to find out SQL stmt, causing this.
all the SQL statement was SELECT ....... FOR UPDATE stmt.
I was also able to find out locked tables.
My tuning idea:
1. I'm thinking to reorganize hot tables as well as their indexes, but by instinct it seems to not give so much value to avoid the huge row lock wait time.
2. I'm also seeing if I can reduce the number of rows per block, by increasing PCTFREE and diminishing PCTUSED, so the contention will spread over many blocks instead of one heavy block.
Question
As SQL stmt related to those locked tables are select ... for update, how could I tune this kind of stmt?
Does someone have other idea to come up with this row lock contention?
Tanks for your effort and helpTaking another look at your suggested function based index, it depends on the data type of the DEV.POS_FOLIO_ID.POS_FOLIO_ID column. If the column is defined as a number, and it is a primary key, there will already be a usable index on that column.
Yesterday, I wrote this: "Once I understood why or how the sessions were trying to insert duplicate primary key values, I would try to determine why the average number of seconds for the wait event is almost 3 seconds (maybe a timeout)."
After fixing the formatting of the top 5 wait events (total duration unknown):
EVENT TOTAL_WAITS TIME_WAITED AVG_MS PERCENT
CPU 94,623.39 48
enq: TX - row lock contention 12,531 36,607.28 2921.34 18
control file parallel write 1,300,731 30,880.79 23.74 16
log file parallel write 1,510,503 12,640.80 8.37 6
log file sync 1,072,553 9,680.07 9.03 512,531 * 3 second time out = 37,593 seconds = 10.44 hours.
What if the reason for the 3 second average wait time is due to a timeout. I performed a little experiment... I changed a row in a test table and then made a pot of coffee.
In session 1:
CREATE TABLE T1 (
C1 NUMBER(10),
C2 NUMBER(10),
PRIMARY KEY (C1));
INSERT INTO T1
SELECT
ROWNUM,
ROWNUM*10
FROM
DUAL
CONNECT BY
LEVEL<=1000000;
COMMIT;I now have a test table with 1,000,000 rows. I start monitoring the changes in the wait events roughly every 60 seconds, and V$SESSION_WAIT and V$LOCK roughly 4 times per second.
Back in session 1:
UPDATE
T1
SET
C1=-C1
WHERE
C1<=100;I have now modified the first 100 rows that were inserted into the table, time to make the pot of coffee.
In session 2, I try to insert a row with a primary key value of -10:
INSERT INTO T1 VALUES (
-10,
10);Session 2 hangs.
If I take the third 60 second snap of the system wide wait events as the zero point, and the 11th snap as the end point. There were 149 waits on ENQ: TX - ROW LOCK CONTENTION, 148 time outs, 446.62 seconds of total time in the wait event, with an average wait time of 2.997450 seconds.
Rolling down to the session level wait events, SID 208 (my session 2) had 149 waits on ENQ: TX - ROW LOCK CONTENTION, for a total time of 446.61 seconds with an average wait time of 2.997383 seconds. All of the 149 waits and the wait time was in this one session that was locked up for the full duration of this time period because session 1 was making a pot of coffee.
Rolling down to V$SESSION_WAIT (sampled roughly 4 times per second): At the start of the third time interval, SID 208 has been in the ENQ: TX - ROW LOCK CONTENTION wait event for 39 seconds and is actively waiting trying to execute SQL with a hash value of 1001532423, the wait object is -1, wait file is 0, wait block is 0, wait row is 0, P1 is 1415053316, P2 is 196646, P3 is 4754.
At the end of the 11th time interval: , SID 208 has been in the ENQ: TX - ROW LOCK CONTENTION wait event for 483 seconds and is actively waiting trying to execute SQL with a hash value of 1001532423, the wait object is -1, wait file is 0, wait block is 0, wait row is 0, P1 is 1415053316, P2 is 196646, P3 is 4754.
Rolling down to V$LOCK (sampled roughly 4 times per second): I see that SID 214 (session 1) is blocking SID 208 (session 2). SID 214 has a TX lock in mode 6 with ID1 of 196646 and ID2 of 4754. SID 208 is requesting a TX lock in mode 4 with ID1 of 196646 and ID2 of 4754.
So, it seems that I need a faster coffee pot rather than an additional index on my table. It could be that the above process would have found that the application associated with SID 214 was abandoned or crashed and for some reason the lock was not released for a long period of time, a little less than 10.44 hours in your case.
Charles Hooper
IT Manager/Oracle DBA
K&M Machine-Fabricating, Inc. -
Index contention & row lock contention
Hi,
Recently our application loaded a handsome amount of data into our production database. The process took around 5 to 6 hours. During this process, we observed that the performance degraded.
The major wait events are
1) TX index contention (concurrency) and
2) ROW LOCK CONTENTION (application).
The sql which lead to index contention is a simple insert statement into a table x and row lock contention is into a table y.
Both x and y are major detail tables of our application. X has around 4 foreign keys, one foriegn key is referencing table y. the foreign key column x is indexed.
our database is on 10gR2 and using ASSM.
The data block wait event stats during the load process is as follows
select * from dba_hist_waitstat where snap_id between '21454' and '21462' AND WAIT_COUNT>0 and class='data block';
SNAP_ID DBID INSTANCE_NUMBER CLASS WAIT_COUNT TIME
21454 3937665896 1 data block 195089769 33899167
21455 3937665896 1 data block 195095958 33902183
21456 3937665896 1 data block 195096398 33902377
21457 3937665896 1 data block 195097225 33902843
21458 3937665896 1 data block 195628987 34037147
21459 3937665896 1 data block 195944006 34055524
21460 3937665896 1 data block 195944496 34055642
21461 3937665896 1 data block 196183308 34112433
21462 3937665896 1 data block 196213292 34127409
9 rows selected.
select snap_id,eq_type,req_reason, total_req#, total_wait#,failed_req#, cum_wait_time from dba_hist_enqueue_stat where snap_id between '21454' and '21462' and failed_req#>0 and eq_type='TM'
SYS@sqdb AS SYSDBA> /
SNAP_ID EQ REQ_REASON TOTAL_REQ# TOTAL_WAIT# FAILED_REQ# CUM_WAIT_TIME
21454 TM contention 682928465 14 2735 1750
21455 TM contention 682939964 14 2735 1750
21456 TM contention 682950668 14 2735 1750
21457 TM contention 682967980 14 2735 1750
21458 TM contention 682983109 14 2735 1750
21459 TM contention 682998136 14 2735 1750
21460 TM contention 683006816 14 2735 1750
21461 TM contention 683018179 14 2735 1750
21462 TM contention 683432945 14 2735 1750
9 rows selected.
I dont see any significant ITL contention on these objects compared to other contention. (from segment statistics)
SYS@sqdb AS SYSDBA> select b.object_name,a.snap_id, a.physical_writes_total,a.physical_writes_delta,a.physical_reads_total,a.physical_reads_delta, a.itl_waits_total,a.itl_waits_delta from dba_hist_seg_stat a,(select object_id,object_name from dba_objects where owner='ABCD') b where snap_id between '21454' and '21462' and a.obj#= b.object_id and a.itl_waits_delta> 0 order by itl_waits_delta,2,1;
OBJECT_NAME SNAP_ID PHYSICAL_WRITES_TOTAL PHYSICAL_WRITES_DELTA PHYSICAL_READS_TOTAL PHYSICAL_READS_DELTA ITL_WAITS_TOTAL ITL_WAITS_DELTA
IX4_Y 21462 37958 169 105668 174 38 1
IX4_X 21462 102380 1480 147196 5780 56 1
As per the application design, for each row of Y, there are multiple number of rows in X and it varies. What additional information do you think I should be collecting and any tips to fix the issue?
Thanks,
Murthyhi
Are you using in your insert statement something like "SELECT ... FOR UPDATE"
post the result of this query during the error:
SELECT * FROM DBA_BLOCKERS;
SELECT * FROM DBA_WAITERS; -
Reading content during SMB upload
I need some urgent help.
I created the MyFolder class and override the extendedPreAddItem method to perform a validation when a document is uploaded to the folder. Basically, I have to inspect the content of the document being uploaded and, if it is correct, I let it to be uploaded; else, I raise an exception. Everything works fine, except when the file is uploaded through the SMB protocol, which is my main need. With SMB, the getContentSize() always return 0 and the getContentStream() always return null. How could I get the content of the file being uploaded via SMB in an override????? My code follows.
public class S_MyFolder extends S_TieFolder {
public S_MyFolder(S_LibrarySession session,
S_LibraryObjectData data) throws IfsException
super(session, data);
public S_MyFolder(S_LibrarySession session,
Long classId) throws IfsException
super(session, classId);
protected void extendedPreAddItem(
OperationState opState,
S_PublicObject obj,
S_LibraryObjectDefinition def)
throws IfsException
super.extendedPreAddItem(opState, obj, def);
if ((S_Document)obj).getContentSize() > 0) {
// do something with the content
// OK with FTP or Java API
else {
// with SMB, always get here
throw new IfsException(100001);
Please, someone help me. Thanks a lot.
Daniel Viero
iProcess Solucoes em Tecnologia
[email protected]
Porto Alegre, RS - BrazilHello Ben,
I'm sorry that FTP has been proving unstable for you. We are aware of some issues with our current FTP implementation and plan to address them in future versions of Muse. For now, if you are seeing problems with FTP locking up during publish the workaround of exporting the HTML and using a third-party FTP client is what we reccoemend.
Thanks,
Justin -
Row lock contention on WWV_FLOW_DATA
I'm hosting an APEX app on one of my Oracle servers
Oracle 10.1.0.4.0
APEX 3.0.0
The users (and APEX developer) are complaining about slow performance. One of the primary reasons seems to be as a result of row lock contention. And as this is on an APEX table, I think it is fair to think that this is not a finger-up-nose mistake from the APEX developer.
This is what I for example just saw some minutes ago. 3 APEX sessions running for application 103 and page 1 according to V$SESSION.
Wait status for all 3 sessions (ticking over into many minutes) is:
enq: TX - row lock contention
The one session shows this SQL:
INSERT INTO WWV_FLOW_DATA ( FLOW_INSTANCE, ITEM_ID, ITEM_VALUE )
VALUES ( :B4 , :B3 , :B2 ||':'||:B1 ||':0:0')
The other two sessions shows this SQL:
UPDATE WWV_FLOW_DATA SET ITEM_VALUE = NVL(NVL(:B6 ,:B5 ),1)||':'
|| NVL(NVL(:B4 ,:B3 ),15)||':0:0' WHERE ITEM_ID = :B2 AND FLOW_I
NSTANCE = :B1
Fair to assume that all 3 are hitting the same row? (same flow instance and item?)
I've done some searching and it seems that a lot of dead locks have been reported on similar APEX tables. Is this potentially the same underlaying issue, but with just slightly different symptoms?
Some speculation... tabbed browsing is very common with Firefox. I often open multiple tabs on the same APEX app for different pages. What will the back-end impact be if this is done on the same page? Could it be that APEX expects a single-HTTP-request-at-a-time-from-a-web-session?
What does not make sense though is why the update will show row lock contention, assuming all 3 hit the same row. The INSERT should when inserting a row with the same PK as that of an uncommitted inserted row of another session. But why would the UPDATEs in this scenario show row lock enqueues? The INSERT row enqueue points that the row has not yet been committed. If so, the 3rd session's UPDATE should return "no row found" and not be stuck on a row lock enqueue.. right?
This does not make much sense to me.
Any comments, suggestions, and ideas to troubleshoot and resolve this will be much appreciated.> When you and the APEX developer say "slow performance", have you ensured
that this is not logic in the APEX application itself?
The "slow performance" the APEX developer and users mention are usually pages timing out all together. There are no real heavy and complex SQLs executed to my knowledge (none I see on the server side). So I do not think it is an application SQL issue - besides, the "slow performance" on the back-end side, when reported, I seen as sessions spinning on enqueue TX locks.
Not my APEX system so I'm not that familiar with it, but I had a quick look at it. The pages are relatively simplistic as there is no custom stuff (unlike some my APEX apps ;-) ).
Also, this problem does not occur on a specific page, but intermittently on different pages.
> were they all for the same APEX session emanating from the same client?
Unsure.. I kind of expect these will have to be or else the SQL statement will be hitting a different flows row all together (which is why I speculated about the tabbed browser issue). Busy investigating this closer.
>Have statistics been computed recently on the FLOWS_030000 schema?
No. Interesting point as I've (very stupidly) never even considered running DBMS_STATS on the flows schema...
What is interesting btw is that SQL is not static. The latest blocks (same event, same symptoms as described in my original posting) were on this SQL:
UPDATE WWV_FLOW_PREFERENCES$ SET ATTRIBUTE_VALUE = :B5 WHERE USE
R_ID = NVL(:B4 ,NVL(:B3 ,USER)) AND PREFERENCE_NAME = :B2 AND SE
CURITY_GROUP_ID = :B1
It also could be as a result of multiple sessions from the same web browser due to clicking a button/link more than once, or refreshing the browser without waiting at least a few seconds for a response.. (also, the fact that the browser is waiting for a response is not very visual/noticeable to the user)
Am planning to spend some time on this issue tomorrow and will try and provide better technical feedback.
If you have anything specific I can test from my side, please just shout Joel. -
Row lock contention problem on Inventory Management
Hi
My client is product based company and have e-Commerce (online Sale Order) application using Oracle database 10.2.0.5 & Web application deployed on WLS. Client have ~ 90 Warehouse country wide and ~ 200 Products (Items). Each product stock for each warehouse is maintained in Inventory Table (INV_BALANCE) which get updated for each order and their sale's products. We are using NO WAIT function to acquire lock on row to update stock balance. Last few months, client business is growing very fast and during Promotional Events (EXPO), they have sales of ~ 75K orders (~ 250K Items) within 2-3 days which start causing row lock contention problem on INV_BALANCE table. We are seeing lots of "Row Lock Wait" and "enq: TX - row lock contention" event which cause ORA-00060: Deadlock detected and we need to eventually clear blocking sessions or everything just choked.
select statistic_name,value from sys.v_$segment_statistics where object_name = 'INV_BALANCE'
STATISTIC_NAME VALUE
logical reads 12423072
buffer busy waits 3895
db block changes 3516768
physical reads 957
physical writes 12197
row lock waits 49909
space used -52921
Deadlock graph:
---------Blocker(s)-------- ---------Waiter(s)---------
Resource Name process session holds waits process session holds waits
TX-00880017-00002584 321 1675 X 110 1445 X
TX-00b60008-00000741 110 1445 X 337 2158 X
TX-0148000b-0000009e 337 2158 X 378 1525 X
TX-01d50015-0000006f 378 1525 X 363 1842 X
TX-02290012-00000070 363 1842 X 267 1798 X
TX-024a0026-0000006e 267 1798 X 364 2084 X
TX-020a0004-0000006f 364 2084 X 135 2113 X
TX-01dc001f-00000070 135 2113 X 129 1586 X
session 1675: DID 0001-0141-000001CA session 1445: DID 0001-006E-00000202
session 1445: DID 0001-006E-00000202 session 2158: DID 0001-0151-0000026B
session 2158: DID 0001-0151-0000026B session 1525: DID 0001-017A-00000167
session 1525: DID 0001-017A-00000167 session 1842: DID 0001-016B-000002B4
session 1842: DID 0001-016B-000002B4 session 1798: DID 0001-010B-000001F1
Rows waited on:
Session 1445: obj - rowid = 0001AE0E - AAAlK8AAHAAD7rMABY
(dictionary objn - 110094, file - 7, block - 1030860, slot - 88)
Session 2158: obj - rowid = 0001AE0E - AAAlK8AAHAAD7rMACD
(dictionary objn - 110094, file - 7, block - 1030860, slot - 131)
Session 1525: obj - rowid = 0001AE0E - AAAlK8AAHAAD7rOAB2
(dictionary objn - 110094, file - 7, block - 1030862, slot - 118)
Information on the OTHER waiting sessions:
Session 1445:
sid: 1445 ser: 37 audsid: 38316795 user: 69/<none>
program: JDBC Thin Client
application name: JDBC Thin Client, hash value=2546894660
Current SQL Statement:
SELECT ROWID, NVL(QTY_PEND,0)+NVL(:B3 ,0) FROM INV_BALANCE WHERE WHS_ID = :B2 AND STOCK_ID = :B1 FOR UPDATE OF QTY_PEND WAIT 10
Please help me on following
Q1: How can we reduce the "row lock Contention" on INV_BALANCE, all possible ways & best practices?
Q2: How can we change the design to not have "row lock Contention" at all?
Appreciate your help
Thanks
Amit Garg
www.otnblogs.comHi Amit!
I saw you are using FOR UPDATE in your query.
If you want reduce row locks, then you not must use FOR UPDATE.
As you know, FOR UPDATE is locking rows.
Regards
Mahir M. Quluzade -
Enq: TX - row lock contention-Trace file shared.
Hi Team,
I am having enq: TX - row lock contention in top wait event. it is occuring between 10pm - 2am.
We are having sqlloader job running every one hour(conventional path). But for the specific period of time i am getting "Global Enqueue Services Deadlock detected". Between 10-5. I analyzed realated trace file it is make me little confusion.I found there are four insert query culprit for this locking. out of four sql , tow of them are ran by same SID, other two insert ran by same id. I got confused because how same sid locking them self. trace file below. Please help me on this. during this period oracle maintanence window is active.Thanks
Trace file:
*** 2012-10-09 03:40:31.135
user session for deadlock lock 0x15365e060
sid: 1104 ser: 22256 audsid: 8797820 user: 49/iurth flags: 0x45
pid: 71 O/S info: user: oracle, term: UNKNOWN, ospid: 8601
image: oracle@sgh0909
client details:
O/S info: user: user, term: , ospid: 30888
machine: sgj89909 program: sqlldr@sgj89909 (TNS V1-V3)
application name: SQL Loader Conventional Path Load, hash value=1935272243
action name: der/sample.ctl, hash value=3241011749
current SQL:
INSERT INTO sample (ID,SU,TIME,PMAN_S,CV_OS,PRD_US,P_DS,PM_LY_DRD_S,PMP_PY_DIDS,PP_S,PMP_RECV_FRMS) VALUES (:NE,:SID,CAST(TO_DATE(:PNG_TE, 'DDMMYYY,PMP_TRAN_FRMS,PMP_RECV_FRMS) VALUES (:NE_ID,:SUB_ID,CAST(TO_DATE(:POLLING_TIME, 'DDMMYYYYHH24MISS') AS DATE),:PMP_TRAN_OCTS,:PMP_RECV_OCTS,:PMP_DISCARD_US,:PMP_DISCARD_DS,:PMP_POLICY_DISCARD_US,:PMP_POLICY_DISCARD_DS,:PMP_TRAN_FRMS,:PMP_RECV_FRMS) ÷ {+ (Cõ{+ (Cõ{+ ð÷ ð÷ ú;6üΠì/' „P' û[þ‡»q hdšÙÿ – $µ + л{+ XdšÙÿ cšÙÿ л{+ M L šÙ ƒß @ešÙÿ ’Ü cšÙÿ lcšÙÿ M „P' L
*** 2012-10-09 03:40:31.533
Short stack dump:
<-ksedsts()+315<-ksdxfstk()+32<-ksdxcb()+1764<-sspuser()+112<-__restore_rt()<-semtimedop()+10<-sskgpwwait()+265<-skgpwwait()+156<-ksliwat()+1546<-kslwaitctx()+162<-kjusuc()+3868<-ksipgetctx()+1456<-ksqcmi()+22853<-ksqgtlctx()+4311<-ksqgelctx()+601<-ktcwit1()+333<-ktbgtl0()+1321<-kdimod0()+4581<-kdiblLockPiece()+968<-kdiblLockRange()+1639<-kdiblcfls()+189<-kdiblFinish1()+978<-kdibliFinish()+73<-kxibDoFinish()+90<-kxibFinish()+431<-qerltcEndLoadProcessing()+614<-qerltcLoadStateMachine()+858<-qerltcInsertValues()+218<-qerltcFetch()+1373<-insexe()+1095<-opiexe()+5609<-opiodr()+1220<-ttcpip()+1227<-opitsk()+1449<-opiino()+1026<-opiodr()+1220<-opidrv()+580<-sou2o()+90<-opimai_real()+145<-ssthrdmain()+177<-main()+215<-__libc_start_main()+244<-_start()+41
user session for deadlock lock 0x1537feef8
sid: 1107 ser: 44863 audsid: 8797823 user: 49/iurth flags: 0x45
pid: 57 O/S info: user: oracle, term: UNKNOWN, ospid: 4540
image: oracle@sgh0909
client details:
O/S info: user: user, term: , ospid: 7569
machine: sgj89909 program: sqlldr@sgj89909 (TNS V1-V3)
application name: SQL Loader Conventional Path Load, hash value=1935272243
action name: der/sample.ctl, hash value=3241011749
current SQL:
INSERT INTO sample (ID,SU,TIME,PMAN_S,CV_OS,PRD_US,P_DS,PM_LY_DRD_S,PMP_PY_DIDS,PP_S,PMP_RECV_FRMS) VALUES (:NE,:SID,CAST(TO_DATE(:PNG_TE, 'DDMMYYY,PMP_TRAN_FRMS,PMP_RECV_FRMS) VALUES (:NE_ID,:SUB_ID,CAST(TO_DATE(:POLLING_TIME, 'DDMMYYYYHH24MISS') AS DATE),:PMP_TRAN_OCTS,:PMP_RECV_OCTS,:PMP_DISCARD_US,:PMP_DISCARD_DS,:PMP_POLICY_DISCARD_US,:PMP_POLICY_DISCARD_DS,:PMP_TRAN_FRMS,:PMP_RECV_FRMS) ÷ {+ (Cõ{+ (Cõ{+ ð÷ ð÷ –R<üΠì/' „P' hdšÙÿ _ž $µ + л{+ XdšÙÿ cšÙÿ л{+ M L šÙ ƒß @ešÙÿ ’Ü cšÙÿ lcšÙÿ M „P' L
Short stack dump:
<-ksedsts()+315<-ksdxfstk()+32<-ksdxcb()+1764<-sspuser()+112<-__restore_rt()<-semtimedop()+10<-sskgpwwait()+265<-skgpwwait()+156<-ksliwat()+1546<-kslwaitctx()+162<-kjusuc()+3868<-ksipgetctx()+1456<-ksqcmi()+22853<-ksqgtlctx()+4311<-ksqgelctx()+601<-ktcwit1()+333<-ktbgtl0()+1321<-kdimod0()+4581<-kdiblLockPiece()+968<-kdiblLockRange()+1639<-kdiblcfls()+189<-kdiblFinish1()+978<-kdibliFinish()+73<-kxibDoFinish()+90<-kxibFinish()+431<-qerltcEndLoadProcessing()+614<-qerltcLoadStateMachine()+858<-qerltcInsertValues()+218<-qerltcFetch()+1373<-insexe()+1095<-opiexe()+5609<-opiodr()+1220<-ttcpip()+1227<-opitsk()+1449<-opiino()+1026<-opiodr()+1220<-opidrv()+580<-sou2o()+90<-opimai_real()+145<-ssthrdmain()+177<-main()+215<-__libc_start_main()+244<-_start()+41
user session for deadlock lock 0x1528de490
sid: 1107 ser: 44863 audsid: 8797823 user: 49/iurth flags: 0x45
pid: 57 O/S info: user: oracle, term: UNKNOWN, ospid: 4540
image: oracle@sgh0909
client details:
O/S info: user: user, term: , ospid: 7569
machine: sgj89909 program: sqlldr@sgj89909 (TNS V1-V3)
application name: SQL Loader Conventional Path Load, hash value=1935272243
action name: der/sample.ctl, hash value=3241011749
current SQL:
INSERT INTO sample (ID,SU,TIME,PMAN_S,CV_OS,PRD_US,P_DS,PM_LY_DRD_S,PMP_PY_DIDS,PP_S,PMP_RECV_FRMS) VALUES (:NE,:SID,CAST(TO_DATE(:PNG_TE, 'DDMMYYY,PMP_TRAN_FRMS,PMP_RECV_FRMS) VALUES (:NE_ID,:SUB_ID,CAST(TO_DATE(:POLLING_TIME, 'DDMMYYYYHH24MISS') AS DATE),:PMP_TRAN_OCTS,:PMP_RECV_OCTS,:PMP_DISCARD_US,:PMP_DISCARD_DS,:PMP_POLICY_DISCARD_US,:PMP_POLICY_DISCARD_DS,:PMP_TRAN_FRMS,:PMP_RECV_FRMS) ÷ {+ (Cõ{+ (Cõ{+ ð÷ ð÷ ÍïEüΠì/' „P' hdšÙÿ €¦ $µ + л{+ XdšÙÿ cšÙÿ л{+ M L šÙ ƒß @ešÙÿ ’Ü cšÙÿ lcšÙÿ M „P' L
*** 2012-10-09 03:40:32.452
Short stack dump:
<-ksedsts()+315<-ksdxfstk()+32<-ksdxcb()+1764<-sspuser()+112<-__restore_rt()<-semtimedop()+10<-sskgpwwait()+265<-skgpwwait()+156<-ksliwat()+1546<-kslwaitctx()+162<-kjusuc()+3868<-ksipgetctx()+1456<-ksqcmi()+22853<-ksqgtlctx()+4311<-ksqgelctx()+601<-ktcwit1()+333<-ktbgtl0()+1321<-kdimod0()+4581<-kdiblLockPiece()+968<-kdiblLockRange()+1639<-kdiblcfls()+189<-kdiblFinish1()+978<-kdibliFinish()+73<-kxibDoFinish()+90<-kxibFinish()+431<-qerltcEndLoadProcessing()+614<-qerltcLoadStateMachine()+858<-qerltcInsertValues()+218<-qerltcFetch()+1373<-insexe()+1095<-opiexe()+5609<-opiodr()+1220<-ttcpip()+1227<-opitsk()+1449<-opiino()+1026<-opiodr()+1220<-opidrv()+580<-sou2o()+90<-opimai_real()+145<-ssthrdmain()+177<-main()+215<-__libc_start_main()+244<-_start()+41
user session for deadlock lock 0x1528de2e0
sid: 1104 ser: 22256 audsid: 8797820 user: 49/iurth flags: 0x45
pid: 71 O/S info: user: oracle, term: UNKNOWN, ospid: 8601
image: oracle@sgh0909
client details:
O/S info: user: user, term: , ospid: 30888
machine: sgj89909 program: sqlldr@sgj89909 (TNS V1-V3)
application name: SQL Loader Conventional Path Load, hash value=1935272243
action name: der/sample.ctl, hash value=3241011749
current SQL:
INSERT INTO sample (ID,SU,TIME,PMAN_S,CV_OS,PRD_US,P_DS,PM_LY_DRD_S,PMP_PY_DIDS,PP_S,PMP_RECV_FRMS) VALUES (:NE,:SID,CAST(TO_DATE(:PNG_TE, 'DDMMYYY,PMP_TRAN_FRMS,PMP_RECV_FRMS) VALUES (:NE_ID,:SUB_ID,CAST(TO_DATE(:POLLING_TIME, 'DDMMYYYYHH24MISS') AS DATE),:PMP_TRAN_OCTS,:PMP_RECV_OCTS,:PMP_DISCARD_US,:PMP_DISCARD_DS,:PMP_POLICY_DISCARD_US,:PMP_POLICY_DISCARD_DS,:PMP_TRAN_FRMS,:PMP_RECV_FRMS) ÷ {+ (Cõ{+ (Cõ{+ ð÷ ð÷ DlJüΠì/' „P' hdšÙÿ ¾® $µ + л{+ XdšÙÿ cšÙÿ л{+ M L šÙ ƒß @ešÙÿ ’Ü cšÙÿ lcšÙÿ M „P' L
Short stack dump:
<-ksedsts()+315<-ksdxfstk()+32<-ksdxcb()+1764<-sspuser()+112<-__restore_rt()<-semtimedop()+10<-sskgpwwait()+265<-skgpwwait()+156<-ksliwat()+1546<-kslwaitctx()+162<-kjusuc()+3868<-ksipgetctx()+1456<-ksqcmi()+22853<-ksqgtlctx()+4311<-ksqgelctx()+601<-ktcwit1()+333<-ktbgtl0()+1321<-kdimod0()+4581<-kdiblLockPiece()+968<-kdiblLockRange()+1639<-kdiblcfls()+189<-kdiblFinish1()+978<-kdibliFinish()+73<-kxibDoFinish()+90<-kxibFinish()+431<-qerltcEndLoadProcessing()+614<-qerltcLoadStateMachine()+858<-qerltcInsertValues()+218<-qerltcFetch()+1373<-insexe()+1095<-opiexe()+5609<-opiodr()+1220<-ttcpip()+1227<-opitsk()+1449<-opiino()+1026<-opiodr()+1220<-opidrv()+580<-sou2o()+90<-opimai_real()+145<-ssthrdmain()+177<-main()+215<-__libc_start_main()+244<-_start()+41
Global blockers dump start:---------------------------------
DUMP LOCAL BLOCKER/HOLDER: block level 3 res [0x2a001d][0x1cb9d],[TX]
----------resource 0x1341d3c30----------------------
resname : [0x2a001d][0x1cb9d],[TX]
Local node : 4
dir_node : 4
master_node : 4
hv idx : 14
hv last r.inc : 32
current inc : 32
hv status : 0
hv master : 4
open options : dd
grant_bits : KJUSERNL KJUSEREX
grant mode : KJUSERNL KJUSERCR KJUSERCW KJUSERPR KJUSERPW KJUSEREX
count : 3 0 0 0 0 1
val_state : KJUSERVS_NOVALUE
valblk : 0x5f4f4354532c504d505f524543565f4f OCTS,PMPRECV_O
access_node : 4
vbreq_state : 0
state : x0
resp : 0x1341d3c30
On Scan_q? : N
Total accesses: 8568
Imm. accesses: 7210
Granted_locks : 1
Cvting_locks : 3
value_block: 5f 4f 43 54 53 2c 50 4d 50 5f 52 45 43 56 5f 4f
GRANTED_Q :
lp 0x1537feef8 gl KJUSEREX rp 0x1341d3c30 [0x2a001d][0x1cb9d],[TX]
master 4 gl owner 0x154b13128 possible pid 4540 xid 39000-0005-00048D1F bast 0 rseq 1329 mseq 0 history 0x14951495
open opt KJUSERDEADLOCK
CONVERT_Q:
lp 0x1537ff0a8 gl KJUSERNL rl KJUSERPR rp 0x1341d3c30 [0x2a001d][0x1cb9d],[TX]
master 4 gl owner 0x155b0dd28 possible pid 12781 xid 37000-0005-00082EBC bast 0 rseq 1329 mseq 0 history 0x49a5149a
convert opt KJUSERGETVALUE
lp 0x1513ef168 gl KJUSERNL rl KJUSERPR rp 0x1341d3c30 [0x2a001d][0x1cb9d],[TX]
master 4 gl owner 0x159aaf240 possible pid 1347 xid 48000-0005-0001FACA bast 0 rseq 1329 mseq 0 history 0x49a5149a
convert opt KJUSERGETVALUE
lp 0x15365e060 gl KJUSERNL rl KJUSERPR rp 0x1341d3c30 [0x2a001d][0x1cb9d],[TX]
master 4 gl owner 0x154b105e0 possible pid 8601 xid 47000-0005-00024CDE bast 0 rseq 1329 mseq 0 history 0x49a5149a
convert opt KJUSERGETVALUE
----------enqueue 0x1537feef8------------------------
lock version : 2128337
Owner node : 4
grant_level : KJUSEREX
req_level : KJUSEREX
bast_level : KJUSERNL
notify_func : (nil)
resp : 0x1341d3c30
procp : 0x1510e0ee0
pid : 12781
proc version : 10700
oprocp : (nil)
opid : 0
group lock owner : 0x154b13128
possible pid : 4540
xid : 39000-0005-00048D1F
dd_time : 0.0 secs
dd_count : 0
timeout : 0.0 secs
On_timer_q? : N
On_dd_q? : N
lock_state : GRANTED
Open Options : KJUSERDEADLOCK
Convert options : KJUSERNOQUEUE KJUSERNODEADLOCKWAIT
History : 0x14951495
Msg_Seq : 0x0
res_seq : 1329
valblk : 0x2c3a504d505f504f4c4943595f444953 ,:PMP_POLICY_DIS
DUMP LOCAL BLOCKER: initiate state dump for DEADLOCK
possible owner[57.4540] on resource TX-002A001D-0001CB9D
*** 2012-10-09 03:40:32.794
Submitting asynchronized dump request [28]
DUMP LOCAL BLOCKER/HOLDER: block level 3 res [0x2f0003][0x1eee9],[TX]
----------resource 0x1355e6868----------------------
resname : [0x2f0003][0x1eee9],[TX]
Local node : 4
dir_node : 4
master_node : 4
hv idx : 64
hv last r.inc : 32
current inc : 32
hv status : 0
hv master : 4
open options : dd
grant_bits : KJUSERNL KJUSEREX
grant mode : KJUSERNL KJUSERCR KJUSERCW KJUSERPR KJUSERPW KJUSEREX
count : 1 0 0 0 0 1
val_state : KJUSERVS_NOVALUE
valblk : 0x5f4f4354532c504de00e0e5101000000 _OCTS,PMQ
access_node : 4
vbreq_state : 0
state : x0
resp : 0x1355e6868
On Scan_q? : N
Total accesses: 13955
Imm. accesses: 11923
Granted_locks : 1
Cvting_locks : 1
value_block: 5f 4f 43 54 53 2c 50 4d e0 0e 0e 51 01 00 00 00
GRANTED_Q :
lp 0x1528de2e0 gl KJUSEREX rp 0x1355e6868 [0x2f0003][0x1eee9],[TX]
master 4 gl owner 0x154b105e0 possible pid 8601 xid 47000-0005-00024CDE bast 0 rseq 1980 mseq 0 history 0x49a51495
open opt KJUSERDEADLOCK
CONVERT_Q:
lp 0x1528de490 gl KJUSERNL rl KJUSERPR rp 0x1355e6868 [0x2f0003][0x1eee9],[TX]
master 4 gl owner 0x154b13128 possible pid 4540 xid 39000-0005-00048D1F bast 0 rseq 1980 mseq 0 history 0x49a5149a
convert opt KJUSERGETVALUE
----------enqueue 0x1528de2e0------------------------
lock version : 2643245
Owner node : 4
grant_level : KJUSEREX
req_level : KJUSEREX
bast_level : KJUSERNL
notify_func : (nil)
resp : 0x1355e6868
procp : 0x1510f9800
pid : 4540
proc version : 21634
oprocp : (nil)
opid : 0
group lock owner : 0x154b105e0
possible pid : 8601
xid : 47000-0005-00024CDE
dd_time : 0.0 secs
dd_count : 0
timeout : 0.0 secs
On_timer_q? : N
On_dd_q? : N
lock_state : GRANTED
Open Options : KJUSERDEADLOCK
Convert options : KJUSERNOQUEUE KJUSERNODEADLOCKWAIT
History : 0x49a51495
Msg_Seq : 0x0
res_seq : 1980
valblk : 0x2c3a504d505f504f4c4943595f444953 ,:PMP_POLICY_DIS
DUMP LOCAL BLOCKER: initiate state dump for DEADLOCK
possible owner[71.8601] on resource TX-002F0003-0001EEE9
*** 2012-10-09 03:40:32.794
Submitting asynchronized dump request [28]
Global blockers dump end:-----------------------------------
Global Wait-For-Graph(WFG) at ddTS[0.108ea] :
BLOCKED 0x15365e060 3 wq 2 cvtops x1 TX 0x2a001d.0x1cb9d [47000-0005-00024CDE] 4
BLOCKER 0x1537feef8 3 wq 1 cvtops x28 TX 0x2a001d.0x1cb9d [39000-0005-00048D1F] 4
BLOCKED 0x1528de490 3 wq 2 cvtops x1 TX 0x2f0003.0x1eee9 [39000-0005-00048D1F] 4
BLOCKER 0x1528de2e0 3 wq 1 cvtops x28 TX 0x2f0003.0x1eee9 [47000-0005-00024CDE] 4Your scenario involves mode 4 (KJUSERPR) and mode 6 (KJUSEREX) TX locks.
Mode 4 (S) TX locks can have a few causes, including ITLs, unique indexes and bitmap indexes.
It could be a very similar scenario to this recent thread where the situation was primary/unique key collisions from inserts on different nodes:
Re: Cause of Deadlock -
Enq: TX - row lock contention in AWR reports
Dears,
One of my friends asked me to give him a help on analyzing a performance problem they are experimenting from time to time. First of all I am sorry to tell you that he didn’t gave me a lot of information. I will share with you all what I have been sent and would like to have your precious advice according to the available information.
It is a third party software installed on oracle data base 10.2.0.4.0. He said that from time to time the application hangs. He sends me an AWR reports supplied by the local DBA. Unfortunately, the local DBA send only the AWR information he thinks are most important to be looked at (he did not include the load profile and the instance efficiency Percentages part in this AWR for example).
Here below is the available information I can share with you
Snap time sessions cursors/session
Begin Snap 09-mars-11 08:00:03 31 6.5
End Snap 09-mars-11 08:39:49 41 9.8
Elapsed 39.77 (mins)
DB Time 536.65(mins)
Top 5 Timed Events
Event Waits Time(s) Avg Wait(ms) % Total Call Time Wait Class
enq: TX - row lock contention 8,468 25,344 2,993 78.7 Application
read by other session 714,628 4,604 6 14.3 User I/O
db file sequential read 323,264 1,977 6 6.1 User I/O
CPU time 171 .5
db file scattered read 1,885 49 26 .2 User I/OThe AWR reports presents also the following SQL statement as the top SQL of its ‘’SQL ordered by Elapsed Time’’ part
UPDATE xxxx.table1
SET col1 = :1,
col2 = :2,
col3 = :3,
col4 = :4,
WHERE ID = :13
AND colx = :14;And the following sql at its “SQL ordered by Reads’ part
SELECT t1.*
FROM xxxx.table1 t1
LEFT OUTER JOIN xxxx.table2 t2
ON t1.id = t2.id
LEFT OUTER JOIN xxxx.table3 t3
ON t1.id = t3.id
LEFT OUTER JOIN xxxx.table4 t4
ON t1.id = t4.id
WHERE t1.col1 = :1
AND t1.col2 IN (:2,:3, :4);And finally in the Segments by Row Lock Waits part of the AWR I have this:
Owner tablespace name object name object type row lock waits %of capture
xxxx xxxx table1 TABLE 54 100 There are no bitmap indexes on this OLTP data base. There are no unindexed foreign keys and it seems that there are no selects done over a dblink.
I am waiting to have the table table1 script and its indexes
Given those information I think that the locking problem is due to this update on table table1 which is not followed immediately by a commit or a rollback. But, instead, several selects that might be taking a long time are done before reaching the commit that ends the lock on the table table1
What do you think about the possible reason of this lock?
Thanks in advance
Mohamed HouriDears,
I did get the suspected query and asked for its several explain plans thanks to the dbms_xplan.display_awr. The most important thing to point out is that the last where clause of this query seems to be dynamically filled-up
SELECT t1.*
FROM xxxx.table1 t1
LEFT OUTER JOIN xxxx.table2 t2
ON t1.id = t2.id
LEFT OUTER JOIN xxxx.table3 t3
ON t1.id = t3.id
LEFT OUTER JOIN xxxx.table4 t4
ON t1.id = t4.id
WHERE t1.col1 = :1
AND t1.col2 in (:2 , :3 , :4 , :5 , :6 , :7 , :8 )
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
| 0 | SELECT STATEMENT | | | | 14 (100)| |
| 1 | NESTED LOOPS OUTER | | 1 | 280 | 14 (0)| 00:00:01 |
| 2 | NESTED LOOPS OUTER | | 1 | 226 | 9 (0)| 00:00:01 |
| 3 | NESTED LOOPS OUTER | | 1 | 216 | 6 (0)| 00:00:01 |
| 4 | TABLE ACCESS BY INDEX ROWID| TABLE1 | 1 | 141 | 4 (0)| 00:00:01 |
| 5 | INDEX RANGE SCAN | SYS_C0010893 | 1 | | 3 (0)| 00:00:01 |
| 6 | TABLE ACCESS BY INDEX ROWID| TABLE2 | 1 | 75 | 2 (0)| 00:00:01 |
| 7 | INDEX UNIQUE SCAN | SYS_C0010774 | 1 | | 1 (0)| 00:00:01 |
| 8 | INDEX RANGE SCAN | SYS_C0010896 | 3 | 30 | 3 (0)| 00:00:01 |
| 9 | TABLE ACCESS BY INDEX ROWID | TABLE3 | 11 | 594 | 5 (0)| 00:00:01 |
| 10 | INDEX RANGE SCAN | SYS_C0010910 | 11 | | 3 (0)| 00:00:01 |
SELECT t1.*
FROM xxxx.table1 t1
LEFT OUTER JOIN xxxx.table2 t2
ON t1.id = t2.id
LEFT OUTER JOIN xxxx.table3 t3
ON t1.id = t3.id
LEFT OUTER JOIN xxxx.table4 t4
ON t1.id = t4.id
WHERE t1.col1 = :1
AND t1.col2 in (:2 , :3 , :4 , :5 , :6 , :7 , :8 , :9 , :10 , :11 , :12 , :13 , :14 , :15 , :16 ,
:17 , :18 , :19 , :20 , :21 , :22 , :23 , :24 , :25 , :26 , :27 , :28 , :29 , :30 , :31 , :32 ,
:33 , :34 , :35 , :36 , :37 , :38 , :39 , :40 , :41 , :42 , :43 , :44 , :45 , :46 , :47 , :48 ,
:251 , :252 , :253 , :254 , :255 , :256 , :257 ) ;
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
| 0 | SELECT STATEMENT | | | | 5943 (100)| |
| 1 | NESTED LOOPS OUTER | | 9210 | 2536K| 5943 (1)| 00:01:12 |
| 2 | NESTED LOOPS OUTER | | 885 | 195K| 1784 (0)| 00:00:22 |
| 3 | NESTED LOOPS OUTER | | 255 | 55080 | 1019 (0)| 00:00:13 |
| 4 | INLIST ITERATOR | | | | | |
| 5 | TABLE ACCESS BY INDEX ROWID| TABLE1 | 255 | 35955 | 509 (0)| 00:00:07 |
| 6 | INDEX UNIQUE SCAN | SYS_C0010893 | 255 | | 258 (0)| 00:00:04 |
| 7 | TABLE ACCESS BY INDEX ROWID | TABLE2 | 1 | 75 | 2 (0)| 00:00:01 |
| 8 | INDEX UNIQUE SCAN | SYS_C0010774 | 1 | | 1 (0)| 00:00:01 |
| 9 | INDEX RANGE SCAN | SYS_C0010896 | 3 | 30 | 3 (0)| 00:00:01 |
| 10 | TABLE ACCESS BY INDEX ROWID | TABLE3 | 10 | 560 | 5 (0)| 00:00:01 |
| 11 | INDEX RANGE SCAN | FK_TABLE3 | 10 | | 3 (0)| 00:00:01 |
----------------------------------------------------------------------------------------------------------- Suddenly there is an INLIST ITERATOR that pops up and which seems to be started 255 times and the ‘quick’ query which in normal situation (small in list) executes in less than 1 second starts now taking more than 1 minute to complete.
As far as it is a third party software, they can’t change the query.
Do you have any idea on how to solve this problem?
Here below are the optimizer parameters
SQL> show parameter optimizer
NAME TYPE VALUE
optimizer_dynamic_sampling integer 2
optimizer_features_enable string 10.2.0.4
optimizer_index_caching integer 0
optimizer_index_cost_adj integer 100
optimizer_mode string ALL_ROWS
optimizer_secure_view_merging boolean TRUEThanks in advance
Mohamed Houri -
Enq: TX - row lock contention in TOP 5 wait event
DB version:11.1.0.7.0
I am having enq: TX - row lock contention in top 5 wait event.
AWR analyze period - 9-10(pm). During this time only one sql loader is running to insert the data. No other job are running. So there is no chance of other session blocking this session. is there any chance of row lock contention happen by same session.
SQL> SELECT INDEX_NAME,INDEX_TYPE,UNIQUENESS FROM DBA_INDEXES WHERE TABLE_NAME='DATA_DATA';
INDEX_NAME INDEX_TYPE UNIQUENES
CIDX BITMAP NONUNIQUE
VIDX BITMAP NONUNIQUE
Thanks.SQL> SELECT INDEX_NAME,INDEX_TYPE,UNIQUENESS FROM DBA_INDEXES WHERE TABLE_NAME='DATA_DATA';
INDEX_NAME INDEX_TYPE UNIQUENES
CIDX BITMAP NONUNIQUE
VIDX BITMAP NONUNIQUEYou have bitmap indexes here on a table being inserted into. Bitmap Indexes are another source of lock(and deadlock) in OLTP application. You said that the SQLloader was the unique active program but may be you are also triggering another procedure after the load. Procedure in which you might be using also automomous transactions and so on...
Check first if your table is subject to DML operation in a a multi-user concurrent accesss and in which case you have to get rid of those bitmap indexes
http://hourim.wordpress.com/2011/03/14/deadlock-%e2%80%93-part-1-bitmap-index/
Best regards
Mohamed Houri
www.hourim.wordpress.com -
Enq: TX - row lock contention in Select without for update
We have deployed a new Version of our Software on the test-system of our customer.
While the software runs fine on our Systems (Oracle EE 10.2.0.4 as well as Oracle EE 11.2.0.2 on EL 5.4 x86_64) it runs sluggish on our Customer's system (Oracle EE 10.2.0.4 on Solaris 10 x64).
During analysis of the problem I came across this:
SQL ID: Plan Hash: 3934983510
SELECT ts.message_id_t235
FROM t235_time_series ts,
t238_timeseries_reported tsr
WHERE ts.time_series_id_t235 = tsr.timeseries_id_t238
AND tsr.message_id_t238 = :b2
AND ts.current_state_t235 = :b1
AND ROWNUM = 1
call count cpu elapsed disk query current rows
======= ====== ======== ========== ========== ========== ========== ==========
Parse 0 0.00 0.00 0 0 0 0
Execute 8 0.01 *137.55* 0 8 5 0
Fetch 8 0.00 0.00 0 60 0 8
======= ====== ======== ========== ========== ========== ========== ==========
total 16 0.01 137.55 0 68 5 8
Misses in library cache during parse: 0
Optimizer mode: CHOOSE
Parsing user id: 90 (recursive depth: 1)
Elapsed times include waiting on following events:
Event waited on Times Max. Wait Total Waited
======================================== Waited ========== ============
*enq:* *TX* *=* *row* *lock* *contention* 281 0.64 *137.01*
********************************************************************************As you can see the statement is very primitive. The Tables used are very huge (100M+ Rows) but have no bitmap indexes. How can this statement wait on enq: TX = row lock contention?
Thanks in advance!
Best regards,
Sven
Edited by: user12182396 on Feb 4, 2011 5:43 AM
Edited by: user12182396 on Feb 4, 2011 6:02 AMuser12182396 wrote:
We have deployed a new Version of our Software on the test-system of our customer.
While the software runs fine on our Systems (Oracle EE 10.2.0.4 as well as Oracle EE 11.2.0.2 on EL 5.4 x86_64) it runs sluggish on our Customer's system (Oracle EE 10.2.0.4 on Solaris 10 x64).
During analysis of the problem I came across this:
SQL ID: Plan Hash: 3934983510
SELECT ts.message_id_t235
FROM t235_time_series ts,
t238_timeseries_reported tsr
WHERE ts.time_series_id_t235 = tsr.timeseries_id_t238
AND tsr.message_id_t238 = :b2
AND ts.current_state_t235 = :b1
AND ROWNUM = 1
call count cpu elapsed disk query current rows
======= ====== ======== ========== ========== ========== ========== ==========
Parse 0 0.00 0.00 0 0 0 0
Execute 8 0.01 *137.55* 0 8 5 0
Fetch 8 0.00 0.00 0 60 0 8
======= ====== ======== ========== ========== ========== ========== ==========
total 16 0.01 137.55 0 68 5 8
Misses in library cache during parse: 0
Optimizer mode: CHOOSE
Parsing user id: 90 (recursive depth: 1)
Elapsed times include waiting on following events:
Event waited on Times Max. Wait Total Waited
======================================== Waited ========== ============
*enq:* *TX* *=* *row* *lock* *contention* 281 0.64 *137.01*
******************************************************************************** As you can see the statement is very primitive. The Tables used are very huge (100M+ Rows) but have no bitmap indexes. How can this statement wait on enq: TX = row lock contention?
Are either of these tables subject to distributed transactions through database links ? If so then it is possible for a select statement to wait on a TX enqueue in mode 4 if it's trying to read a block which is subject to change by another tranaction that is stuck between the PREPARE and COMMIT phases of a "2 phase commit".
However, I have to say that don't know whether this would show up as: "enq: TX - row lock contention" or "enq: TX - contention". If you monitor the system for a while - checking v$lock for lock types TX, or query v$active_session_history (if you're licensed to use it) you may be able to check the lock mode.
If this is the problem then it's generally indicative of a networking problem of some sort - possibly simple congestion because of excess traffic.
Regards
Jonathan Lewis
http://jonathanlewis.wordpress.com
http://www.jlcomp.demon.co.uk
To post code, statspack/AWR report, execution plans or trace files, START and END the text with the tag {noformat}{noformat} (the word "code" in lowercase, curly brackets, no spaces) so that the text appears in fixed format.
There is a +"Preview"+ tab at the top of the text entry panel. Use this to check what your message will look like before you post the message. If it looks a complete mess you're unlikely to get a response. (Click on the +"Plain text"+ tab if you want to edit the text to tidy it up.) -
Lock contention heavily impacting Oracle DB 10gR1 performance
Hello,
I have put in production a new application using Oracle 10g R1, about a month ago.
Every Sunday since the migration, my application stops working due to Oracle lock contention (proved by Oracle Grid).
I absolutely don't understand why the application runs all the week perfectly (when the volume of data managed by the application if a lot higher) and experiences problems during the week-end. That does not make any sense to me. I have heard people saying that Oracle takes a lot activity time to perform auto-tuning, is that something real? Would anyone know what Oracle could be doing on the Sunday that it would not do the rest of the week. Again, the application is a lot less stressed during the Sunday.
Thanks for your help,
Eric G.Hi Eric,
I absolutely don't understand why the application runs all the week perfectly (when the volume of data managed by the application if a lot higher) and experiences problems during the week-end.Me, I would consider running a STATSPACK orr AWR report, and then you will learn the reason . . . . Here are my notes:
http://www.dba-oracle.com/oracle10g_tuning/t_generating_awr_report.htm
Would anyone know what Oracle could be doing on the Sunday that it would not do the rest of the week.Check your scheduled jobs with dba_schedules:
http://www.rampant-books.com/book_2005_1_scheduling.htm
Hope this helps. . . .
Don Burleson
Oracle Press author
http://www.dba-oracle.com -
Feature restore request. (allow locked and grouped objects to be able to tagged.
In CS2, using the direct selection tool, I'm able to select and tag the image even if it's grouped and locked, no script needed at all, no sweat and fast!
Furthermore this XML element can also be moved freely in the structure hierarchy after tagged and remains grouped and locked.
In CS3 and CS4,
I can't do that.
It will give a warning message "Cannot modify elements that contain locked content, or are contained by locked content. Please unlock or check out the content and try again."
So need to write script to unlock the image's parent first, and then tag and locked the object back automatically.
(This script is essential so as to prevent any unintentional and accidental change on the page layout if it's done manually)
But to make matter worse, grouped objects will need to also find out all these ancestors and have them unlocked first!
It creates alot of hassle! Really being user-unfriendly in CS3/CS4 when doing such a simple xml tagging task.
PLEASE please restore the wonderful CS2 feature back in the next version of Indesign.
MILLIONS of thanks!!I'd suggest you submit your suggestion through the iTunes application feedback page. It's more likely to be seen by the iTunes development group that way.
Maybe you are looking for
-
WE60 - Error while generating the xml schema
Hi Using the transaction WE60 I am trying to generate an XML schema for the BASIC type : DEBMDM06, Segment release - 7 and record type version - 3. Following error message is displayed. "Structure of segment E1T023W is unknown". Kindly let me know ho
-
Firefox crashes, can't create new profile, reinstall doesn't work, can't get to safe mode
Whenever I try to open Firefox, all it does is open the crash reporter. When I press "Restart Firefox", it just opens yet another crash reporter. I'm using a PC. I have tried all of the following, to no avail: 1. Reboot computer 2. System restore 3.
-
Hi Guys, Is anyone having a problem with DB replication on 4.2? I have both my ACS servers added in the network devices with correct keys set. i have set one to send and the other to recieve in the db replication settings, I have also set the partner
-
Problem with spring effect in After Effects
Whey I'm animating an object from A to B using position, thats fine. But When I add one more keyframe that moves it back to A again (or anywhere else for that matter), my problem appears. The object kind of continue to spring forward in the direction
-
How to configure smart card login in sunray 2fs??
Hi all, Please help me to configure smart card login using Sun Ray Server Software 4.0... How to assign a smart card for a particular user? Do I need to flash th smart card for user information or any other method exists?