DeadlockException
Hi, using DPL and 3.1.0 on Win32 platform JDK 1.5.09-b -- simple unit test that does the following:
1) put an object into DPL
2) get it back out and do equality check
3) delete the object from DPL
Step 3 causes a DeadlockException.
Stacktrace follows:
junit.framework.AssertionFailedError: Exception in test: DatabaseException occurred: com.sleepycat.je.DeadlockException: (JE 3.1.0) Lock expired. Locker 14_main_AutoTxn: waited for lock on database=persist#DalStore#FeatureImpl#featureCode node=52 type=WRITE grant=WAIT_NEW timeoutMillis=500 startTime=1161798879187 endTime=1161798880187
Owners: [<LockInfo locker="-1_main_ThreadLocker" type="READ"/>, <LockInfo locker="-1_main_ThreadLocker" type="READ"/>]
Waiters: []
at junit.framework.Assert.fail(Assert.java:47)
at TestBerkeleyCache.testPutFeature(TestBerkeleyCache.java:93)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at junit.framework.TestCase.runTest(TestCase.java:154)
at junit.framework.TestCase.runBare(TestCase.java:127)
at junit.framework.TestResult$1.protect(TestResult.java:106)
at junit.framework.TestResult.runProtected(TestResult.java:124)
at junit.framework.TestResult.run(TestResult.java:109)
at junit.framework.TestCase.run(TestCase.java:118)
at junit.framework.TestSuite.runTest(TestSuite.java:208)
at junit.framework.TestSuite.run(TestSuite.java:203)
at org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference.run(JUnit3TestReference.java:128)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:460)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:673)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:386)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:196)
Here is the method that causes the exception ->
public void remove(long featureId, int featureCode) throws TADalException {
EntityJoin<Long, FeatureImpl> join = new EntityJoin<Long, FeatureImpl>(featureByPrimaryKey);
join.addCondition(featureById, featureId);
join.addCondition(featureByCode, featureCode);
ForwardCursor<Long> cursor = null;
try {
cursor = join.keys();
long id = cursor.next();
featureByPrimaryKey.delete(id);
} catch (DatabaseException e) {
throw new TADalException("DatabaseException occurred: " + e);
} finally {
try {
if (cursor != null) {
cursor.close();
} catch (DatabaseException dbe) {
throw new TADalException("DatabaseException occurred: " + dbe);
}Any ideas on whether I'm doing something I shouldn't here? Thanks in advance...
-davis
Hi Mark --
the problem I have is that EntityCursor is a
sub-interface of ForwardCursor and ForwardCursor has
no delete operation.I'm sorry, I didn't even notice that your cursor was returned with a join.
How to get at delete( ) then?If you want to delete/modify/insert while a join cursor is open, and you want to use a transactional store, then you will need to:
1) Before doing the join, start a transaction by calling Environment.beginTransaction.
2) Pass the transaction to join.keys() so that the transaction is used by the join cursor.
3) Pass the transaction to featureByPrimaryKey.delete() also.
4) When you're done, after closing all cursors, commit the transaction by calling Transaction.commit.
In general, when you are using transactions, if a cursor is open in the current thread and you perform other non-cursor operations in that thread, all operations must use the same transaction. In other words, you cannot use auto-commit for the delete operation when a cursor is open. (Auto-commit is the default when no transaction is passed and the store is transactional.)
Mark
Similar Messages
-
DeadlockException on putNoReturn()
I'm building up a non-transactional data store with multiple worker threads with code that follows the pattern:
PrimaryIndex<Long, BoundObjectImpl> boundObjectById;
boundObjectById.putNoReturn(newBoundObjectInstance);
I'm running this on a box with 8 processors using 15 worker threads to keep the CPUs saturated through I/O blocking as the source data is coming from an Oracle database. I tried increasing the lockTables count to 3, which helped on a dual-processor dev environment, but I'm wondering if I should bump it higher for the 8-way environment?
This is the exception:
DatabaseException occurred: com.sleepycat.je.DeadlockException: (JE
3.1.25 (15233)) Lock expired. Locker -1_pool-1-thread-8_ThreadLocker:
waited for lock on
database=persist#BoundObjectStore#test.boundobject.BoundObjectStoreImpl#targetObjectCode
node=53112855 type=WRITE grant=WAIT_NEW timeoutMillis=1000 startTime=1164987874050 endTime=1164987879589
Also, is there any way to control the number of worker threads spawned by the Berkeley database? It subverts my own thread management if the database is spawning a large number of threads for each worker thread I create.
Thanks,
AndyHello Andy,
I'm running this on a box with 8 processors using 15
worker threads to keep the CPUs saturated through I/O
blocking as the source data is coming from an Oracle
database. I tried increasing the lockTables count to
3, which helped on a dual-processor dev environment,
but I'm wondering if I should bump it higher for the
8-way environment?Yes, I think so, but I'll let others answer that are more knowledgeable than I in this area.
DatabaseException occurred:
com.sleepycat.je.DeadlockException: (JE
3.1.25 (15233)) Lock expired. Locker
-1_pool-1-thread-8_ThreadLocker:
waited for lock on
database=persist#BoundObjectStore#test.boundobject.Bou
ndObjectStoreImpl#targetObjectCode
node=53112855 type=WRITE grant=WAIT_NEW
timeoutMillis=1000 startTime=1164987874050
endTime=1164987879589This exception is not related to the number of lock tables. The number of lock tables is an optimization that does not impact deadlocking.
Deadlocks, how to handle them and how to prevent them, are explained here:
http://www.oracle.com/technology/documentation/berkeley-db/je/TransactionGettingStarted/blocking_deadlocks.html
Also, is there any way to control the number of
worker threads spawned by the Berkeley database? It
subverts my own thread management if the database is
spawning a large number of threads for each worker
thread I create.JE creates a fixed number of threads, it does not create worker threads dynamically. By default it creates three threads: the log cleaner, checkpointer and compressor. Is that too many for your application?
Mark -
Multiple issues, Berkeley JE 3.2.15, 3.2.76
Hi all,
We have been using JE 3.2.15 with great satisfaction for more than a year, but we've been recently running into what seems to be a deadlock with the following stack trace:
#1) "Thread-12" daemon prio=4 tid=0x0b174320 nid=0x72 in Object.wait() [0x208cd000..0x208cdbb8]
at java.lang.Object.wait(Native Method)
- waiting on <0xf4202fc0> (a com.sleepycat.je.txn.Txn)
at com.sleepycat.je.txn.LockManager.lock(LockManager.java:227)
- locked <0xf4202fc0> (a com.sleepycat.je.txn.Txn)
at com.sleepycat.je.txn.Txn.lockInternal(Txn.java:295)
at com.sleepycat.je.txn.Locker.lock(Locker.java:283)
at com.sleepycat.je.dbi.CursorImpl.lockLNDeletedAllowed(CursorImpl.java:2375)
at com.sleepycat.je.dbi.CursorImpl.lockLN(CursorImpl.java:2297)
at com.sleepycat.je.dbi.CursorImpl.searchAndPosition(CursorImpl.java:1983)
at com.sleepycat.je.Cursor.searchInternal(Cursor.java:1188)
at com.sleepycat.je.Cursor.searchAllowPhantoms(Cursor.java:1158)
at com.sleepycat.je.Cursor.search(Cursor.java:1024)
at com.sleepycat.je.Database.get(Database.java:557)
The call does not seem to return, not for as long as 22 hours anyway. Or maybe it's just very slow performing a single get (these are done in loops), giving the appearance that it's stuck. It appears more or less randomly (every couple of months, maybe) at various sites. We have not been able to reproduce it on our test systems.
We recently upgraded to 3.2.76, kinda hoping that this issue had been fixed. While the new version appeared to work fine during internal testing and then for 2 deployments, if failed repeatedly during the third deployment. The original issue just manifested itself again (the above stack trace is from 3.2.76). We also experienced the following:
#2) Environment invalid because of previous exception: com.sleepycat.je.log.DbChecksumException: (JE 3.2.76) Read invalid log entry type: 49
at com.sleepycat.je.log.LogEntryHeader.<init>(LogEntryHeader.java:69)
at com.sleepycat.je.log.LogManager.getLogEntryFromLogSource(LogManager.java:631)
at com.sleepycat.je.log.LogManager.getLogEntry(LogManager.java:597)
at com.sleepycat.je.tree.IN.fetchTarget(IN.java:958)
at com.sleepycat.je.dbi.CursorImpl.searchAndPosition(CursorImpl.java:1963)
at com.sleepycat.je.Cursor.searchInternal(Cursor.java:1188)
at com.sleepycat.je.Cursor.searchAllowPhantoms(Cursor.java:1158)
at com.sleepycat.je.Cursor.search(Cursor.java:1024)
at com.sleepycat.je.Database.get(Database.java:557)
#3) <DaemonThread name="Checkpointer"/> caught exception: com.sleepycat.je.DatabaseException: (JE 3.2.76) fetchTarget of 0x61d/0xa2d452 parent IN=147872417 lastFullVersion=0x69c/0x27e251 parent.getDirty()=true state=0 com.sleepycat.je.log.DbChecksumException: (JE 3.2.76) Read invalid log entry type: 58
com.sleepycat.je.DatabaseException: (JE 3.2.76) fetchTarget of 0x61d/0xa2d452 parent IN=147872417 lastFullVersion=0x69c/0x27e251 parent.getDirty()=true state=0 com.sleepycat.je.log.DbChecksumException: (JE 3.2.76) Read invalid log entry type: 58
at com.sleepycat.je.tree.IN.fetchTarget(IN.java:989)
at com.sleepycat.je.cleaner.Cleaner.migrateLN(Cleaner.java:1100)
at com.sleepycat.je.cleaner.Cleaner.lazyMigrateLNs(Cleaner.java:928)
at com.sleepycat.je.tree.BIN.logInternal(BIN.java:1117)
at com.sleepycat.je.tree.IN.log(IN.java:2657)
at com.sleepycat.je.recovery.Checkpointer.logTargetAndUpdateParent(Checkpointer.java:975)
at com.sleepycat.je.recovery.Checkpointer.flushIN(Checkpointer.java:810)
at com.sleepycat.je.recovery.Checkpointer.flushDirtyNodes(Checkpointer.java:670)
at com.sleepycat.je.recovery.Checkpointer.doCheckpoint(Checkpointer.java:442)
at com.sleepycat.je.recovery.Checkpointer.onWakeup(Checkpointer.java:211)
at com.sleepycat.je.utilint.DaemonThread.run(DaemonThread.java:191)
at java.lang.Thread.run(Thread.java:595)
Caused by: com.sleepycat.je.log.DbChecksumException: (JE 3.2.76) Read invalid log entry type: 58
at com.sleepycat.je.log.LogEntryHeader.<init>(LogEntryHeader.java:69)
at com.sleepycat.je.log.LogManager.getLogEntryFromLogSource(LogManager.java:631)
at com.sleepycat.je.log.LogManager.getLogEntry(LogManager.java:597)
at com.sleepycat.je.tree.IN.fetchTarget(IN.java:958)
... 11 more
* An out of memory condition:
#4) Exception in thread "Cleaner-1" java.lang.OutOfMemoryError: Java heap space
at com.sleepycat.je.log.LogUtils.readByteArray(LogUtils.java:204)
at com.sleepycat.je.log.entry.LNLogEntry.readEntry(LNLogEntry.java:104)
at com.sleepycat.je.log.FileReader.readEntry(FileReader.java:238)
at com.sleepycat.je.log.CleanerFileReader.processEntry(CleanerFileReader.java:140)
at com.sleepycat.je.log.FileReader.readNextEntry(FileReader.java:321)
at com.sleepycat.je.cleaner.FileProcessor.processFile(FileProcessor.java:411)
at com.sleepycat.je.cleaner.FileProcessor.doClean(FileProcessor.java:259)
at com.sleepycat.je.cleaner.FileProcessor.onWakeup(FileProcessor.java:161)
at com.sleepycat.je.utilint.DaemonThread.run(DaemonThread.java:191)
at java.lang.Thread.run(Thread.java:595)
It is possible, of course, that the actual memory issue is somewhere else in the application and Berkeley just happened to be the one getting the error. However the load is very much 24 hours-periodic and the most significant change was the Berkeley upgrade, which does not seem to be running entirely correctly...
I was wondering:
- is issue #1 indeed a deadlock? Has anyone else experienced it? Is there a fix/workaround?
- are #2 and #3 known issues? Note that these happened both while processing 3.2.15 files and, after failure and deletion, pure 3.2.76 files.
- is it possible that an invalid log entry would cause #4? Or is there a memory leak/incorrect allocation in BDB?
If that helps, the standard installation is on Sun x86 hardware running Solaris and (Sun) Java 1.5.0_11. Berkeley is run as an XA resource. Like I said we haven't seen any of these issues on our internal test systems. The customer that experienced #2, #3 and #4 has since been rolled back to 3.2.15, and #1 is rather infrequent. So that's pretty much all the information we have and are going to get, but if there's anything else you'd like to know...
Thanks in advance,
Matthieu BentotHello Matthieu,
Thank you for your clear explanation.
On #1, the deadlock, I don't see any known JE bug that could cause this, in the current release or that has been fixed since 3.2.15. The thread dump you sent implies that a thread is waiting to acquire a record lock, which means that some other thread or transaction holds the record lock at that time. There are a few things that come to mind:
1) If you are setting the lock timeout to a large value, or to zero which means "wait forever", this kind of problem could occur as the result of normal record lock contention. I assume you're not doing this or you would have mentioned it, but I thought I should ask. Are you calling EnvironmentConfig or TransactionConfig.setLockTimeout, or setting the lock timeout via the je.properties file?
2) Are you performing retries when DeadlockException is thrown (as suggested in our documentation)? By retry I mean that when you catch DeadlockException, you close any open cursors, abort the transaction (if you're using transactions), and then start the operation or transaction from the beginning. One possibility is that two threads are continuously retrying, both trying to access the same record(s). If so, one possible solution is to delay for a small time interval before retrying, to give other threads a chance to finish their operation or transaction.
3) The most common reason for the problem you're seeing is that a cursor or transaction is accidentally left open. A cursor or transaction holds record locks until it is closed (or committed or aborted in the case of a transaction). If you have a cursor or transaction that has "leaked" -- been discarded without being closed -- or you have a thread that keeps a cursor or transaction open for a long period, another thread trying to access the record will be unable to lock it. Normally, DeadlockException will be thrown in one thread or the other. But if you are setting the lock timeout (1) or retrying operations (2), this could cause the problem you're seeing. Are you seeing DeadlockException?
If you can describe more about how you handle DeadlockException, send your environment, database, cursor and transaction configuration settings, and describe how you close cursors and commit/abort transactions, hopefully we will be able to narrow this down.
Your #2 and #3 are the same problem, which is very likely a known problem that we're currently focussed on. It is interesting that you never saw this problem in 3.2.15 -- that could be a clue for us.
When you started seeing the problem, were there any changes other than moving to JE 3.2.76? Did you start using a network file system of some kind? Or use different hardware that provides more concurrency? Or update your application to perform with more concurrency?
You mentioned that this occurred in a deployment but not in your test environment. If you have the JE log files from that deployment and/or you can reproduce the problem there, we would very much like to work with you to resolve this. If so, please send me an email -- mark.hayes at the obvious .com -- and I'll ask you to send log files and/or run with a debugging JE jar file when trying to reproduce the problem.
It is possible that #4 is a side effect of the earlier problems, but it's very difficult to say for sure. I don't see any known bugs since 3.2.76 that could cause an OutOfMemoryError.
Please send us more information on #1 as I mentioned above, and send me email on the checksum problem (#2 and #3).
Thanks,
--mark -
Suspicious error, maybe 3.2.68 related?
This is very tentative, but wanted to relay details in case it meant anything to the BDB team:
One of the users of our crawler tried dropping in 3.2.68 as a replacement for 3.2.44, and after about one weeks' runtime hit similar, unexpected OutOfMemoryError stacks on two separate machines. A representative stack:
Exception in thread "ToeThread #87: " java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2882)
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
at java.lang.StringBuffer.append(StringBuffer.java:224)
at com.sleepycat.je.DatabaseException.<init>(DatabaseException.java:25)
at com.sleepycat.je.DeadlockException.<init>(DeadlockException.java:29)
at com.sleepycat.je.txn.LockManager.makeTimeoutMsgInternal(LockManager.java:432)
at com.sleepycat.je.txn.SyncedLockManager.makeTimeoutMsg(SyncedLockManager.java:76)
at com.sleepycat.je.txn.LockManager.lock(LockManager.java:262)
at com.sleepycat.je.txn.BasicLocker.lockInternal(BasicLocker.java:94)
at com.sleepycat.je.txn.Locker.lock(Locker.java:257)
at com.sleepycat.je.dbi.CursorImpl.lockLNDeletedAllowed(CursorImpl.java:2375)
at com.sleepycat.je.dbi.CursorImpl.lockLN(CursorImpl.java:2297)
at com.sleepycat.je.dbi.CursorImpl.searchAndPosition(CursorImpl.java:1983)
at com.sleepycat.je.dbi.DbTree.modifyDbRoot(DbTree.java:336)
at com.sleepycat.je.recovery.Checkpointer.flushIN(Checkpointer.java:750)
at com.sleepycat.je.recovery.Checkpointer.flushDirtyNodes(Checkpointer.java:669)
at com.sleepycat.je.recovery.Checkpointer.syncDatabase(Checkpointer.java:571)
at com.sleepycat.je.dbi.DatabaseImpl.sync(DatabaseImpl.java:545)
at com.sleepycat.je.Database.sync(Database.java:322)
Of course the real cause of the memory depletion may be elsewhere, and indeed in our project's code, but that this BDB-JE operation triggered heap depletion twice was at least a little suspicious. (Might the DeadlockException message length grow to extremes in certain situations?)
Our users' original comment is viewable at http://webteam.archive.org/jira/browse/HER-1387 .
- Gordon @ IAI've asked our user about the je.txn.dumpLocks value, as well as a couple other questions to try to narrow down what happened around the error.
It's still very possible something in our code is largely to blame, and I'd have to relay the patch to the 3rd party for testing as we haven't yet seen this here -- so a patch isn't needed, yet, but thanks for the quick offer.
I'll update with anything else I learn. -
Non-transactional cursor writes block reads
I'm opening a cursor without using a transaction like so:
CursorConfig conf = new CursorConfig();
Cursor cursor = db.openCursor(null, conf);
I then iterate over the items, sometimes replacing a value. I want to be able to read (and maybe write) to any value from another thread while the cursor is still open, but gets result in the following error:
com.sleepycat.je.DeadlockException: (JE 3.3.74) Lock expired. Locker 32098350 -1_NioProcessor-3_ThreadLocker: waited for lock on database=settings LockAddr:1537969 node=2623139 type=READ grant=WAIT_NEW timeoutMillis=500 startTime=1229438453225 endTime=1229438453730
Owners: [<LockInfo locker="11743647 -1_NioProcessor-1_ThreadLocker" type="WRITE"/>]
Waiters: []
I would have expected that outside a transaction any update made through a cursor would be applied immediately, but instead the record is locked until the cursor is closed. This happens whether or not je.env.isTransactional is set to true.
Setting je.env.isLocking to false gives me the cursor behaviour that I want, but I'm not sure what else is affected by this setting, and the javadoc comment is just vaguely worrying rather than helpful ("This property should be set to false only in special circumstances when it is safe to run without record locking.")
I am not planning on using any transactions in this application.Cormac,
I was looking through the documentation for a reference page to direct you to, and I realized that probably the clearest explanation in the javadoc is in the com.sleepycat.je.LockMode page here: [http://www.oracle.com/technology/documentation/berkeley-db/je/java/com/sleepycat/je/LockMode.html]. There is a reference to it from the CursorConfig page, but I can see that the link between the two pages is not that clear.
From the LockMode javadoc, there is this paragraph.
bq. Locking Rules \\ Together with CursorConfig, TransactionConfig and EnvironmentConfig settings, lock mode parameters determine how records are \\ locked during read operations. Record locking is used to enforce the \\ isolation modes that are configured. Record locking is summarized below for \\ read and write operations. For more information on isolation levels and \\ transactions, see [Writing Transactional Applications|http://www.oracle.com/technology/documentation/berkeley-db/je/TransactionGettingStarted/index.html]. \\ With one exception, a record lock is always acquired when a record is \\ read or written, and a cursor will always hold the lock as long as it is \\ positioned on the record. The exception is when [READ_UNCOMMITTED|http://www.oracle.com/technology/documentation/berkeley-db/je/java/com/sleepycat/je/LockMode.html#READ_UNCOMMITTED] \\ is specified, which allows a record to be read without any locking.
Transactional cursors, by default, hold all locks until the cursor is closed. In your case, you are using a non transactional cursor, and locks are not collected the same way. But a lock for modified record is still taken in a non transactional cursor, and is held as long as the cursor is positioned at that record, to provide cursor stability. I am guessing that is what is happening in your case. If you close the cursor or move to a different position, you will be releasing the lock.
In a second posting, you comment on your experiences using the je.env.locking property. That property is really only to be used in cases where the application is very constrained and has high performance requirements; frankly, that was put in for a particular set of power users and we really don't recommend it for general purpose use. (We should doc it better too). One characteristic is that it disables log cleaning, because the lack of locks makes it impossible for us to properly coordinate access to the data from the log cleaning threads.
Regards,
Linda
Edited by: Linda Lee on Dec 16, 2008 2:19 PM (changed " link between the two pages is that clear" -> not that clear) -
Put failure behavior in a non-transactional db?
Hi all,
I'm still getting to know the ropes here, so hopefully this isn't a stupid question. I'm still prototyping an application, and as such I haven't set my database to be transactional (learning one thing at a time). The app runs for quite a while, but eventually it tries to put an entity into a primary index that violates one of the secondary key constraints (uniqueness). The put() call throws an exception (properly showing KEYEXIST) but then all subsequent operations that I try to perform fail with a lock timeout (DeadlockException).
My guess is that some resource wasn't cleaned up after the exception since I'm not running transactionally (so no auto-commit), but I can't find any documentation to back that up. Am I likely to avoid this problem if I turn on transactions? If I don't, is there something in the API I can do to clean up after an exception in a non-transactional put?
Thanks!
Jeff AlexanderYou may want to check to make sure that you're not leaving a cursor open when an exception occurs. An open cursor will hold a lock. I recommend closing cursors in a finally block.
If a lock is held by mistake (either your mistake such as not closing a cursor, or a bug in JE), a deadlock may not be caused immediately because other operations may not need to lock the same record. To debug this, perhaps after receiving the KEYEXIST exception and closing cursors properly, you could try closing all Databases and the Environment. Environment.close will throw an exception if any locks were not released.
It would be nice if there were a way to query the number of outstanding locks, but I don't think we have a way to do that currently, without reaching into JE internals.
--mark -
Berkeley DB and DB optimization
Hi,
I have been testing BerkeleyDB-4.7.25 with 16 *.bdb files using BTREE on a Linux server 64bits. Each *.bdb file reaches approximately a size of 3.2 GB.
I have run a set of operations that include puts/gets/updates/deletes
I would like to ask a couple of questions, please:
1)
Is there is any Berkeley DB tool/function to optimize the *.bdb files for/after deletion.
2)
I have been running dbstat -e (please find the output of db_stat below) trying to improve some of the DB_CONFIG parameters.
set_flags DB_TXN_WRITE_NOSYNC
set_cachesize 0 2147483648 1
mutex_set_max 1000000
set_tx_max 500000
set_lg_regionmax 524288
set_lg_bsize 4194304
set_lg_max 20971520
set_lk_max_locks 10000
set_lk_max_lockers 10000
set_lk_max_objects 10000
I have increased the cache size, but it does not seem to be helping to improve the operation response times.
I would really appreciate any help.
Would the use of DB_SYSTEM_MEM (create the shared regions in system shared memory) help ?
Would the preallocation of the db files help ?
Would the increase of the log buffer help ?
Would the size of the log help (based on the values related to data written since last checkpoint in db_stat) ?
Could you please help ?
Thanks,
Mariella
This is the output of db_stat -e:
0x40988 Log magic number
14 Log version number
4MB Log record cache size
0 Log file mode
20Mb Current log file size
72M Records entered into the log (72944260)
92GB 761MB 385KB 636B Log bytes written
1GB 805MB 40KB 747B Log bytes written since last checkpoint
6596982 Total log file I/O writes
0 Total log file I/O writes due to overflow
7295 Total log file flushes
39228 Total log file I/O reads
4749 Current log file number
18526992 Current log file offset
4748 On-disk log file number
20970984 On-disk log file offset
1 Maximum commits in a log flush
1 Minimum commits in a log flush
4MB 512KB Log region size
303613 The number of region locks that required waiting (0%)
100 Last allocated locker ID
0x7fffffff Current maximum unused locker ID
9 Number of lock modes
10000 Maximum number of locks possible
10000 Maximum number of lockers possible
10000 Maximum number of lock objects possible
40 Number of lock object partitions
16 Number of current locks
274 Maximum number of locks at any one time
7 Maximum number of locks in any one bucket
0 Maximum number of locks stolen by for an empty partition
0 Maximum number of locks stolen for any one partition
100 Number of current lockers
108 Maximum number of lockers at any one time
16 Number of current lock objects
176 Maximum number of lock objects at any one time
4 Maximum number of lock objects in any one bucket
0 Maximum number of objects stolen by for an empty partition
0 Maximum number of objects stolen for any one partition
118M Total number of locks requested (118356655)
118M Total number of locks released (118356639)
119802 Total number of locks upgraded
16 Total number of locks downgraded
20673 Lock requests not available due to conflicts, for which we waited
0 Lock requests not available due to conflicts, for which we did not wait
0 Number of deadlocks
0 Lock timeout value
0 Number of locks that have timed out
500000 Transaction timeout value
0 Number of transactions that have timed out
7MB 768KB The size of the lock region
5019 The number of partition locks that required waiting (0%)
328 The maximum number of times any partition lock was waited for (0%)
0 The number of object queue operations that required waiting (0%)
280 The number of locker allocations that required waiting (0%)
958 The number of region locks that required waiting (0%)
4 Maximum hash bucket length
2GB Total cache size
1 Number of caches
1 Maximum number of caches
2GB Pool individual cache size
0 Maximum memory-mapped file size
0 Maximum open file descriptors
0 Maximum sequential buffer writes
0 Sleep after writing maximum sequential buffers
0 Requested pages mapped into the process' address space
150M Requested pages found in the cache (92%)
12M Requested pages not found in the cache (12855704)
8449044 Pages created in the cache
12M Pages read into the cache (12855704)
20M Pages written from the cache to the backing file (20044721)
32M Clean pages forced from the cache (32698230)
1171137 Dirty pages forced from the cache
9227380 Dirty pages written by trickle-sync thread
505880 Current total page count
356352 Current clean page count
149528 Current dirty page count
262147 Number of hash buckets used for page location
184M Total number of times hash chains searched for a page (184542797)
34 The longest hash chain searched for a page
945M Total number of hash chain entries checked for page (945465289)
430 The number of hash bucket locks that required waiting (0%)
34 The maximum number of times any hash bucket lock was waited for (0%)
5595 The number of region locks that required waiting (0%)
0 The number of buffers frozen
0 The number of buffers thawed
0 The number of frozen buffers freed
34M The number of page allocations (34375350)
76M The number of hash buckets examined during allocations (76979039)
18 The maximum number of hash buckets examined for an allocation
33M The number of pages examined during allocations (33869157)
4 The max number of pages examined for an allocation
2 Threads waited on page I/O
Pool File: file_p10.bdb
4096 Page size
0 Requested pages mapped into the process' address space
9376233 Requested pages found in the cache (92%)
800764 Requested pages not found in the cache
4096 Page size
0 Requested pages mapped into the process' address space
9376233 Requested pages found in the cache (92%)
800764 Requested pages not found in the cache
526833 Pages created in the cache
800764 Pages read into the cache
1179504 Pages written from the cache to the backing file
Pool File: file_p3.bdb
4096 Page size
4658/8873223 File/offset for last checkpoint LSN
Thu Apr 30 22:00:23 2009 Checkpoint timestamp
0x806584b8 Last transaction ID allocated
500000 Maximum number of active transactions configured
0 Active transactions
8 Maximum active transactions
6653112 Number of transactions begun
60327 Number of transactions aborted
6592785 Number of transactions committed
144048 Snapshot transactions
257302 Maximum snapshot transactions
0 Number of transactions restored
185MB 24KB Transaction region size
90116 The number of region locks that required waiting (0%)
Active transactions:
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
129MB 720KB Mutex region size
108 The number of region locks that required waiting (0%)
4 Mutex alignment
200 Mutex test-and-set spins
1000000 Mutex total count
331261 Mutex free count
668739 Mutex in-use count
781915 Mutex maximum in-use count
Mutex counts
331259 Unallocated
16 db handle
1 env dblist
2 env handle
1 env region
43 lock region
274 logical lock
1 log filename
1 log flush
2 log region
16 mpoolfile handle
16 mpool filehandle
17 mpool file bucket
1 mpool handle
262147 mpool hash bucket
262147 mpool buffer I/O
1 mpool region
1 mutex region
1 twister
1 txn active list
1 transaction checkpoint
144050 txn mvcc
1 txn regionuser11096811 wrote:
i have same questionWhat is the question exactly? What DB release are you using?
user11096811 wrote:
the app throws com.sleepycat.db.LockNotGrantedException .what should i do?The LockNotGrantedException exception being thrown is a subclass of DeadlockException.
A LockNotGrantedException is thrown when a lock requested using the Environment.getLock or Environment.lockVector methods, where the noWait flag or lock timers were configured, could not be granted before the wait-time expired.
Additionally, LockNotGrantedException is thrown when a Concurrent Data Store database environment configured for lock timeouts was unable to grant a lock in the allowed time.
Additionally, LockNotGrantedException is thrown when lock or transaction timeouts have been configured and a database operation has timed out. Applications can handle all deadlocks by
catching the DeadlockException. You can read more on how to configure the locking subsystem and resolve the deadlocks at [The Locking Subsystem|http://www.oracle.com/technology/documentation/berkeley-db/db/gsg_txn/JAVA/lockingsubsystem.html].
Thanks,
Bogdan Coman -
Hi All,
I have two processes, one creates environement in RW mode, creates a transactional database(s), writes n records, and closes it.
Another process opens the same env (by this I means same directory), in RO mode (I understand this gives snapshot view as of open time), finds all databases in that environment (getDatabaseNames), if works on that data put in by process 1, and truncates the db. Truncate throws foll. error.
com.sleepycat.je.DeadlockException: (JE 3.2.13) Lock expired. Locker 250_main_Txn: waited for lock on database=_jeNameMap node=34 type=WRITE grant=WAIT_PROMOTION timeoutMillis=500 startTime=1187817739998 endTime=1187817740998
Owners: [<LockInfo locker="250_main_Txn" type="READ"/>, <LockInfo locker="-1_main_BasicLocker" type="READ"/>]
Waiters: []
Transaction 250_main_Txn owns 34 <LockInfo locker="250_main_Txn" type="READ"/>
Transaction 250_main_Txn waits for node 34
How to address this ?
Logically this is what I am trying to do. Process 1 accumulates new data and Process 2 processes the data once a number of new records created by P1 reach a threshold. Once processed, it wants to drop those records. I thought easiest way is to do truncate.
P1 uses databases in a circular fashion (much like redo logs). Say there are 5 databases to start with, P1 places records into DB1, then DB2, ..and uses DB1 again after DB5. In the mean time, P2 should have emptied records from DB1 etc when P1 returns to it. P2 cannot open the databases in RW mode for obvious reasons, so it cant delete records. P1 closes db soon it writes threshold # of records.
Any suggestions on how I might go about this ?
Thanks
VissuHi Vissu,
Truncating a database is a write operation, so you can't do it in a read-only environment.
If you must write the data in one process and read/truncate it in another, then the only ways I think of doing that are:
1) The writer process must close the environment while the reader/truncator process truncates it. This is difficult since it requires coordinating the two processes.
2) Instead of rotating through 5 databases, rotate through 5 environments (each with a single database). That way, the reader/truncator can open the environment read-write, after it has been closed by the writer.
I recommend trying (2). Of course, things will be much simpler if you can use a single process for reading, reading and truncating.
--mark -
Serious lock problem - transaction has active cursors
Hi,
i use BDB C via the Java-API (4.6.21, developing with WindowsXP). Currently i am having serious problems with LOCK_NOT_GRANTED_EXCEPTIONS.
My application is a queue implementation. One push objects into the database, another one reads them and a third one deletes entries with an outdated TTL (time-to-live). All of these processes are done by multiple dedicated threads what leads to highly concurrent situations.
The TTL cleaning uses a tx-cursor which deletes entries that are too old. The inserter simply uses put(). The reader uses a cursor too, which starts with a pointer (via getSearchKey) or at the beginning of the database (if no start could be found, the pointer is then outdated as it was deleted by TTL cleaning run).
The isolation level is at READ_COMMITTED. Sometimes i use RMW locks for reading and updating status information.
So i implemented a lock detection which retries at least the inserts if a lock fails. The reads can fail and report their exceptions to the outside world. This seems to work fine. Sometimes i get some LOCK_NOT_GRANTED_EXCEPTIONS and the recovery should work. But it works only sometimes.
BDB-C complains with "transaction has active cursors" (see txn.c - line 1186). All subsequent calls to rollback or commit or any other method of BDB-C handles fails then with a RunRecoveryException. I am absolutely sure, i have no open cursors, as i close them implicitly before tx.commit() or tx.abort(). I wrote a reference counter to ensure that and it shows always a count of 0 open cursors as expected.
The whole process is of course quite heavy stress for the DB - i am actually read, write, delete and checkpoint the db at the same position at the same time.
I am tending to introduce MVCC to avoid those locking issues. As i understood concurrent updates of the same db page with multiversion transactions may also cause DeadLockExceptions. Can i use RMW locks to avoid concurrent updates on the same pages with MVCC?
Some hints?
TIA
DirkHi,
I have created a patch for Berkeley DB 4.6.21, which outputs information about all cursor open and close operations. This should help you see which cursor is not being closed.
If you apply the patch below to your Berkeley DB tree, and rebuild.
Then add setVerbose(VerboseConfig.CURSOR, true), to your EnvironmentConfig initialization.
It will send log messages to the output stream configured via EnvironmentConfig.setMessageStream(). The default is System.out.
There will likely be a lot of output.
Please let me know if this helps you resolve the problem. Otherwise, ask more questions ;)
Regards,
Alex Gorrod
diff -rc db-4.6.21.orig/build_windows/db.h db-4.6.21/build_windows/db.h
*** db-4.6.21.orig/build_windows/db.h Fri Sep 28 01:32:08 2007
--- db-4.6.21/build_windows/db.h Mon May 12 17:29:52 2008
*** 60,66 ****
#define DB_VERSION_MAJOR 4
#define DB_VERSION_MINOR 6
#define DB_VERSION_PATCH 21
! #define DB_VERSION_STRING "Berkeley DB 4.6.21: (September 27, 2007)"
--- 60,66 ----
#define DB_VERSION_MAJOR 4
#define DB_VERSION_MINOR 6
#define DB_VERSION_PATCH 21
! #define DB_VERSION_STRING "Berkeley DB 4.6.21: (May 12, 2008)"
*** 2133,2138 ****
--- 2133,2139 ----
#define DB_VERB_REGISTER 0x0010 /* Dump waits-for table. */
#define DB_VERB_REPLICATION 0x0020 /* Replication information. */
#define DB_VERB_WAITSFOR 0x0040 /* Dump waits-for table. */
+ #define DB_VERB_CURSOR 0x0080 /* Cursor open/close information. */
u_int32_t verbose; /* Verbose output. */
void app_private; / Application-private handle. */
diff -rc db-4.6.21.orig/db/db_am.c db-4.6.21/db/db_am.c
*** db-4.6.21.orig/db/db_am.c Thu Jun 14 04:21:30 2007
--- db-4.6.21/db/db_am.c Mon May 12 17:32:08 2008
*** 283,288 ****
--- 283,291 ----
TAILQ_INSERT_TAIL(&dbp->active_queue, dbc, links);
F_SET(dbc, DBC_ACTIVE);
MUTEX_UNLOCK(dbenv, dbp->mutex);
+ if (FLD_ISSET(dbenv->verbose, DB_VERB_CURSOR))
+ __db_msg(dbenv,
+ "Cursor (%p) opened, associated txn: %p", dbc, dbc->txn);
*dbcp = dbc;
return (0);
diff -rc db-4.6.21.orig/db/db_cam.c db-4.6.21/db/db_cam.c
*** db-4.6.21.orig/db/db_cam.c Tue Jun 5 21:46:24 2007
--- db-4.6.21/db/db_cam.c Mon May 12 17:32:25 2008
*** 71,76 ****
--- 71,79 ----
opd = cp->opd;
ret = 0;
+ if (FLD_ISSET(dbenv->verbose, DB_VERB_CURSOR))
+ __db_msg(dbenv,
+ "Cursor (%p) closed, associated txn: %p", dbc, dbc->txn);
* Remove the cursor(s) from the active queue. We may be closing two
* cursors at once here, a top-level one and a lower-level, off-page
diff -rc db-4.6.21.orig/dbinc/db.in db-4.6.21/dbinc/db.in
*** db-4.6.21.orig/dbinc/db.in Fri Jun 29 00:23:35 2007
--- db-4.6.21/dbinc/db.in Mon May 12 17:23:20 2008
*** 2106,2111 ****
--- 2106,2112 ----
#define DB_VERB_REGISTER 0x0010 /* Dump waits-for table. */
#define DB_VERB_REPLICATION 0x0020 /* Replication information. */
#define DB_VERB_WAITSFOR 0x0040 /* Dump waits-for table. */
+ #define DB_VERB_CURSOR 0x0080 /* Cursor open/close information. */
u_int32_t verbose; /* Verbose output. */
void app_private; / Application-private handle. */
diff -rc db-4.6.21.orig/dist/pubdef.in db-4.6.21/dist/pubdef.in
*** db-4.6.21.orig/dist/pubdef.in Fri Jul 6 10:22:52 2007
--- db-4.6.21/dist/pubdef.in Mon May 12 17:21:48 2008
*** 432,437 ****
--- 432,438 ----
DB_USERCOPY_SETDATA * I N
DB_USE_ENVIRON D I J
DB_USE_ENVIRON_ROOT D I J
+ DB_VERB_CURSOR D I J
DB_VERB_DEADLOCK D I J
DB_VERB_FILEOPS D I J
DB_VERB_FILEOPS_ALL D I J
diff -rc db-4.6.21.orig/env/env_method.c db-4.6.21/env/env_method.c
*** db-4.6.21.orig/env/env_method.c Fri May 18 01:15:11 2007
--- db-4.6.21/env/env_method.c Mon May 12 17:28:57 2008
*** 1070,1075 ****
--- 1070,1076 ----
int *onoffp;
switch (which) {
+ case DB_VERB_CURSOR:
case DB_VERB_DEADLOCK:
case DB_VERB_FILEOPS:
case DB_VERB_FILEOPS_ALL:
*** 1098,1103 ****
--- 1099,1105 ----
int on;
switch (which) {
+ case DB_VERB_CURSOR:
case DB_VERB_DEADLOCK:
case DB_VERB_FILEOPS:
case DB_VERB_FILEOPS_ALL:
diff -rc db-4.6.21.orig/java/src/com/sleepycat/db/EnvironmentConfig.java db-4.6.21/java/src/com/sleepycat/db/EnvironmentConfig.java
*** db-4.6.21.orig/java/src/com/sleepycat/db/EnvironmentConfig.java Fri Jul 6 10:22:54 2007
--- db-4.6.21/java/src/com/sleepycat/db/EnvironmentConfig.java Mon May 12 17:25:22 2008
*** 114,119 ****
--- 114,120 ----
private boolean yieldCPU = false;
/* Verbose Flags */
+ private boolean verboseCursor = false;
private boolean verboseDeadlock = false;
private boolean verboseFileops = false;
private boolean verboseFileopsAll = false;
*** 846,851 ****
--- 847,855 ----
public void setVerbose(final VerboseConfig flag, boolean enable) {
int iflag = flag.getInternalFlag();
switch (iflag) {
+ case DbConstants.DB_VERB_CURSOR:
+ verboseCursor = enable;
+ break;
case DbConstants.DB_VERB_DEADLOCK:
verboseDeadlock = enable;
break;
*** 876,881 ****
--- 880,887 ----
public boolean getVerbose(final VerboseConfig flag) {
int iflag = flag.getInternalFlag();
switch (iflag) {
+ case DbConstants.DB_VERB_CURSOR:
+ return verboseCursor;
case DbConstants.DB_VERB_DEADLOCK:
return verboseDeadlock;
case DbConstants.DB_VERB_FILEOPS:
*** 1151,1156 ****
--- 1157,1167 ----
dbenv.set_flags(offFlags, false);
/* Verbose flags */
+ if (verboseCursor && !oldConfig.verboseCursor)
+ dbenv.set_verbose(DbConstants.DB_VERB_CURSOR, true);
+ if (!verboseCursor && oldConfig.verboseCursor)
+ dbenv.set_verbose(DbConstants.DB_VERB_CURSOR, false);
+
if (verboseDeadlock && !oldConfig.verboseDeadlock)
dbenv.set_verbose(DbConstants.DB_VERB_DEADLOCK, true);
if (!verboseDeadlock && oldConfig.verboseDeadlock)
*** 1342,1347 ****
--- 1353,1359 ----
yieldCPU = ((envFlags & DbConstants.DB_YIELDCPU) != 0);
/* Verbose flags */
+ verboseCursor = dbenv.get_verbose(DbConstants.DB_VERB_CURSOR);
verboseDeadlock = dbenv.get_verbose(DbConstants.DB_VERB_DEADLOCK);
verboseFileops = dbenv.get_verbose(DbConstants.DB_VERB_FILEOPS);
verboseFileopsAll = dbenv.get_verbose(DbConstants.DB_VERB_FILEOPS_ALL);
diff -rc db-4.6.21.orig/java/src/com/sleepycat/db/VerboseConfig.java db-4.6.21/java/src/com/sleepycat/db/VerboseConfig.java
*** db-4.6.21.orig/java/src/com/sleepycat/db/VerboseConfig.java Fri Jul 6 10:22:54 2007
--- db-4.6.21/java/src/com/sleepycat/db/VerboseConfig.java Mon May 12 17:28:07 2008
*** 12,17 ****
--- 12,19 ----
import com.sleepycat.db.internal.DbEnv;
public final class VerboseConfig {
+ public static final VerboseConfig CURSOR =
+ new VerboseConfig("CURSOR", DbConstants.DB_VERB_CURSOR);
public static final VerboseConfig DEADLOCK =
new VerboseConfig("DEADLOCK", DbConstants.DB_VERB_DEADLOCK);
public static final VerboseConfig FILEOPS =
diff -rc db-4.6.21.orig/java/src/com/sleepycat/db/internal/DbConstants.java db-4.6.21/java/src/com/sleepycat/db/internal/DbConstants.java
*** db-4.6.21.orig/java/src/com/sleepycat/db/internal/DbConstants.java Fri Sep 28 01:32:07 2007
--- db-4.6.21/java/src/com/sleepycat/db/internal/DbConstants.java Mon May 12 17:29:17 2008
*** 194,199 ****
--- 194,200 ----
int DB_UPGRADE = 0x0000001;
int DB_USE_ENVIRON = 0x0004000;
int DB_USE_ENVIRON_ROOT = 0x0008000;
+ int DB_VERB_CURSOR = 0x0080;
int DB_VERB_DEADLOCK = 0x0001;
int DB_VERB_FILEOPS = 0x0002;
int DB_VERB_FILEOPS_ALL = 0x0004; -
Deadlocks and more on a large database.
Hi,
We are running a multithreaded server. The database is performing as cache, which is queried/updated about 500 times/sec by about 500 threads.
Most queries are misses.
We are experiencing a large number of deadlocks in the database, and sometime get/put operations fails to complete even after 20 retries.
We would please like to know the following (not all questions related to the deadlock issue):
1. How can we reduce the number of deadlocks ?
2. Is using deferred mode favorable in this use case ?
3. How do we find the optimal log file size ?
4. Are there any more configuration properties we need to consider ?
Thanks,
Lior've tried changing the lock timeout to the default,
as specified in the je.properties file. (50000).
however, it seems the change has no effect, and the
deadlocks occur in the same frequency as before.Ok.
we've also tried without much success:
envConfig.setConfigParam("je.env.sharedLatches",
"true");
envConfig.setConfigParam("je.lock.nLockTables",
"401");The above params have no impact on deadlocks.
Is there any other configuration properties I can
change?No, except perhaps to increase the cache size so that operations complete more quickly. An operation that does a read from disk will take much longer than when the record is in cache.
Here's the stack trace, after modifying the lock
timeout property:
2008-07-23 16:55:54,977 WARN
[com.flash.http.postOptCache.service.PostOptCacheFacad
eImpl] com.sleepycat.je.DeadlockException: (JE
3.2.68) Lock expired. Locker
-1_HPI-executor707_ThreadLocker: waited for lock on
database=postoptcache node=4211902 type=WRITE
grant=WAIT_NEW timeoutMillis=500
startTime=1216821354475 endTime=1216821354977
Owners: [<LockInfo
locker="-1_HPI-executor825_ThreadLocker"
type="WRITE"/>]
Waiters: []
com.sleepycat.je.DeadlockException: (JE 3.2.68) Lock
expired. Locker -1_HPI-executor707_ThreadLocker:
waited for lock on database=postoptcache node=4211902
type=WRITE grant=WAIT_NEW timeoutMillis=500
startTime=1216821354475 endTime=1216821354977This information tells us that thread HPI-executor707 is trying to perform a write operation and times out after 500 millis. The write lock on the record is not granted because thread HPI-executor825 owns the write lock on that record.
Thread HPI-executor707 is performing a Database.delete. We don't know what thread HPI-executor825 is doing, but we can assume that is holding the record lock for more than 500 millis.
You have a couple options:
1) Find out why HPI-executor825 is holding the lock for so long, and try to correct that problem. For example, if it is keeping a cursor open, try to close the cursor earlier.
2) If you're sure that no thread is keeping a cursor open for any longer than necessary, then it is possible that something else is slowing things down. Is I/O very slow on this system? Are full GCs taking a long time? If so, you can try to correct these problems, but you can also increase the lock timeout to a value larger than any delay caused by I/O or GC.
Basically, you need to find out why the record lock is being held by thread HPI-executor825 for more than 500 millis. Then either correct that situation so that the lock is not held for so long, or increase the lock timeout if there is no way to reduce the time that the lock is held.
--mark -
Concurrently write and read caused program hang infinitely.
Hi, everyone
I used BDB XML 2.4.11
os : windows xp service pack 2
I have set the java option -Xms64M -Xmx256M
I used a util class for retrieving the XmlManager and XmlContainer handle, the code is below
import java.io.File;
import java.io.FileNotFoundException;
import java.util.HashMap;
import java.util.Iterator;
import java.util.Map;
import com.sleepycat.db.CheckpointConfig;
import com.sleepycat.db.DatabaseException;
import com.sleepycat.db.Environment;
import com.sleepycat.db.EnvironmentConfig;
import com.sleepycat.db.LockDetectMode;
import com.sleepycat.dbxml.XmlContainer;
import com.sleepycat.dbxml.XmlContainerConfig;
import com.sleepycat.dbxml.XmlException;
import com.sleepycat.dbxml.XmlManager;
import com.sleepycat.dbxml.XmlManagerConfig;
public class BdbXmlUtil {
private static Map<String, XmlContainer> openedContainer = new HashMap<String, XmlContainer>();
private static XmlManager xmlManager = null;
public static synchronized XmlManager getXmlManager() {
if (xmlManager == null) {
EnvironmentConfig envConf = new EnvironmentConfig();
envConf.setAllowCreate(true);
envConf.setInitializeCache(true);
envConf.setInitializeLocking(true);
envConf.setInitializeLogging(true);
envConf.setRunRecovery(true);
envConf.setTransactional(true);
envConf.setLockDetectMode(LockDetectMode.MINWRITE);
envConf.setCacheSize(256 * 1024 * 1024);
envConf.setTxnMaxActive(5000);
envConf.setMultiversion(true);
envConf.setTemporaryDirectory(new File("E:/temp/bdb-xml/tmp"));
envConf.setErrorStream(System.err);
// for performance
// The setTxnNoWait has to set ture, only the transaction set true will not take effect.
envConf.setTxnNoSync(true);
//envConf.setTxnNoWait(true);
Environment environment = null;
try {
environment = new Environment(new File("E:/temp/bdb-xml/bdb_xml_testenv"), envConf);
CheckpointConfig checkpointConf = new CheckpointConfig();
checkpointConf.setKBytes(1024);
environment.checkpoint(checkpointConf);
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (DatabaseException e) {
e.printStackTrace();
XmlManagerConfig managerConfig = new XmlManagerConfig();
managerConfig.setAdoptEnvironment(true);
try {
xmlManager = new XmlManager(environment, managerConfig);
} catch (XmlException e) {
e.printStackTrace();
return xmlManager;
public static synchronized XmlContainer getXmlContainer(String xmlContainerName) {
XmlContainer xmlContainer = null;
if (openedContainer.containsKey(xmlContainerName)) {
xmlContainer =
(XmlContainer) openedContainer.get(xmlContainerName);
} else {
XmlContainerConfig containerConf = new XmlContainerConfig();
containerConf.setTransactional(true);
containerConf.setAllowCreate(true);
containerConf.setNodeContainer(false);
containerConf.setMultiversion(true);
/*containerConf.setReadUncommitted(true);*/
try {
xmlContainer =
xmlManager.openContainer(
xmlContainerName, containerConf);
} catch (XmlException e) {
e.printStackTrace();
openedContainer.put(xmlContainerName, xmlContainer);
return xmlContainer;
public static synchronized void closeEnv() {
try {
Iterator<String> containerNames = openedContainer.keySet().iterator();
while (containerNames.hasNext()) {
XmlContainer container = openedContainer.get(containerNames.next());
container.close();
if (xmlManager != null) {
xmlManager.close();
} catch (Exception e) {
e.printStackTrace();
}I used two classes to test the concurrency, one is responsible for put document to database, and the other is used to query database while the put operation is being performed. The run method of the two classes is below:
put document process code:
boolean retry = true;
int retry_count = 0;
// while loop is used for deadlock retries
while (retry) {
XmlTransaction txn = null;
try {
XmlUpdateContext context = xmlManager.createUpdateContext();
TransactionConfig tc = new TransactionConfig();
tc.setNoSync(true);
txn = xmlManager.createTransaction(null, tc);
String doc = generateId() + ".xml";
xmlContainer.putDocument(txn, doc, "<user><name>hello world</name></user>", context);
txn.commit();
txn = null;
System.out.println("created : " + doc);
retry = false;
} catch (XmlException xe) {
retry = false;
if (xe.getDatabaseException() instanceof DeadlockException) {
System.out.println(getName()
+ " got deadlock exception!");
if (retry_count < 20) {
System.err.println(getName() + " : Retrying operation.");
retry = true;
retry_count++;
try {
Thread.sleep(100);
} catch (InterruptedException e) {}
} else {
System.err.println(getName()
+ " : out of retries. Giving up.");
} else {
System.err.println("XmlException during concurrent-test: "
+ xe.getMessage());
} finally {
if (txn != null) {
try {
txn.abort();
} catch (Exception e) {
System.err.println("Error aborting txn in concurrent-test: "
+ e.toString());
e.printStackTrace();
}the query process code :
XmlManager xmlManager = BdbXmlUtil.getXmlManager();
XmlResults res = null;
XmlQueryExpression expr = null;
XmlTransaction txn = null;
try {
BdbXmlUtil.getXmlContainer("entry.dbxml");
XmlQueryContext qc = xmlManager.createQueryContext();
expr = xmlManager.prepare("doc('dbxml:/entry.dbxml/1.xml')//user/name", qc);
TransactionConfig tc = new TransactionConfig();
tc.setSnapshot(true);
txn = xmlManager.createTransaction(null, tc);
res = expr.execute(txn, qc);
System.out.println("The query, '" + expr.getQuery() +
"'\n\t returned " + res.size() + " result(s)");
while (res.hasNext()) {
XmlValue value = res.next();
System.out.println(value.asString());
txn.commit();
txn = null;
increaseInvokeCount();
} catch (XmlException e) {
e.printStackTrace();
} finally {
if (res != null) {
res.delete();
if (expr != null) {
expr.delete();
if (txn != null) {
try {
txn.abort();
} catch (Exception e) {
System.err.println("Error aborting txn in query test: "
+ e.toString());
e.printStackTrace();
}in the main class, I have generated 100 threads to run the put document process and 30 threads to run the query process. After insert several documents to database, the whole process hangs infinitely. I think it is caused by the query process. When i set the query thread number to 1 or disable the query process, the put document process ended with success.
I think it is caused by the lock. I try to execute the query in Snapshot isolation, but it doesn't works. I also tried the No Wait On Blocks, it seem ok, but alway throws DeadLockException, which can not bear. I doubt about that the lock granularity is per page, and in my program I used the Wholedoc Containers. The Query Process just take action in one document, which is 1.xml, but the lock seem that the whole container is locked. Could it be the page is so large that it covers the whole container? I referenced the document Getting Started with Transaction Processing For Java, but find no hint.
The BDB XML is excellent in performance, and I want use it in my program. I really hope that this problem can be solved. I don't know whether it is a bug or my reason.Any suggestion will be very helpful, thanks advance.
Jhon Kao
Message was edited to better format the Java code:
gmfeinberg
added some more info to describe the problem:
Excalibur
nullDear gmfeinberg,
You are so kind. I appreciate very much for your approval. You are alway very warm-hearted to help others. Thanks very much.
I followed your advice and changed the prepare() to add the transaction as a parameter. The complete code I have already posted above. But unfortunately the program hangs also. The complete output of the db_stat -CA when the hang happens is below:
Default locking region information:
395 Last allocated locker ID
0x7fffffff Current maximum unused locker ID
9 Number of lock modes
1000 Maximum number of locks possible
1000 Maximum number of lockers possible
1000 Maximum number of lock objects possible
132 Number of current locks
134 Maximum number of locks at any one time
544 Number of current lockers
582 Maximum number of lockers at any one time
12 Number of current lock objects
19 Maximum number of lock objects at any one time
3935 Total number of locks requested
3697 Total number of locks released
0 Total number of locks upgraded
58 Total number of locks downgraded
257 Lock requests not available due to conflicts, for which we waited
101 Lock requests not available due to conflicts, for which we did not wait
0 Number of deadlocks
0 Lock timeout value
0 Number of locks that have timed out
0 Transaction timeout value
0 Number of transactions that have timed out
472KB The size of the lock region
938 The number of region locks that required waiting (3%)
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Lock REGINFO information:
Lock Region type
5 Region ID
__db.005 Region name
0x720000 Original region address
0x720000 Region address
0x720048 Region primary address
0 Region maximum allocation
0 Region allocated
Region allocations: 3005 allocations, 0 failures, 0 frees, 1 longest
REGION_JOIN_OK Region flags
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Lock region parameters:
65571 Lock region region mutex [938/28484 3% 4244/7164]
1031 locker table size
1031 object table size
448 obj_off
45880 locker_off
0 need_dd
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Lock conflict matrix:
0 0 0 0 0 0 0 0 0
0 0 1 0 1 0 1 0 1
0 1 1 1 1 1 1 1 1
0 0 0 0 0 0 0 0 0
0 1 1 0 0 0 0 1 1
0 0 1 0 0 0 0 0 1
0 1 1 0 0 0 0 1 1
0 0 1 0 1 0 1 0 0
0 1 1 0 1 1 1 0 1
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Locks grouped by lockers:
Locker Mode Count Status ----------------- Object ---------------
2 dd=464 locks held 2 write locks 0 pid/thread 1704/5836
2 READ 1 HELD entry.dbxml handle 2
2 READ 1 HELD entry.dbxml handle 0
5 dd=463 locks held 0 write locks 0 pid/thread 1704/5836
6 dd=462 locks held 1 write locks 0 pid/thread 1704/5836
6 READ 1 HELD entry.dbxml handle 4
9 dd=461 locks held 0 write locks 0 pid/thread 1704/5836
a dd=460 locks held 1 write locks 0 pid/thread 1704/5836
a READ 1 HELD entry.dbxml handle 6
d dd=459 locks held 0 write locks 0 pid/thread 1704/5836
e dd=458 locks held 1 write locks 0 pid/thread 1704/5836
e READ 1 HELD entry.dbxml handle 8
11 dd=457 locks held 0 write locks 0 pid/thread 1704/5836
12 dd=456 locks held 1 write locks 0 pid/thread 1704/5836
12 READ 1 HELD entry.dbxml handle 10
15 dd=455 locks held 0 write locks 0 pid/thread 1704/5836
16 dd=454 locks held 2 write locks 0 pid/thread 1704/5836
16 READ 1 HELD entry.dbxml handle 12
16 READ 5 HELD entry.dbxml handle 0
19 dd=453 locks held 0 write locks 0 pid/thread 1704/7764
1d dd=452 locks held 1 write locks 0 pid/thread 1704/5836
1d READ 1 HELD entry.dbxml handle 14
20 dd=451 locks held 0 write locks 0 pid/thread 1704/6460
21 dd=450 locks held 2 write locks 0 pid/thread 1704/5836
21 READ 1 HELD entry.dbxml handle 16
21 READ 2 HELD entry.dbxml handle 0
24 dd=449 locks held 0 write locks 0 pid/thread 1704/5836
52 dd=448 locks held 2 write locks 0 pid/thread 1704/5836
52 READ 1 HELD entry.dbxml handle 18
52 READ 1 HELD entry.dbxml handle 0
55 dd=447 locks held 0 write locks 0 pid/thread 1704/5836
56 dd=446 locks held 1 write locks 0 pid/thread 1704/1168
56 READ 1 HELD entry.dbxml page 13
57 dd=335 locks held 0 write locks 0 pid/thread 1704/4692
58 dd=334 locks held 0 write locks 0 pid/thread 1704/4692
59 dd=333 locks held 0 write locks 0 pid/thread 1704/4692
5a dd=332 locks held 0 write locks 0 pid/thread 1704/4692
5b dd=331 locks held 0 write locks 0 pid/thread 1704/4692
5c dd=330 locks held 0 write locks 0 pid/thread 1704/4692
5d dd=327 locks held 0 write locks 0 pid/thread 1704/7184
5e dd=324 locks held 0 write locks 0 pid/thread 1704/1168
5f dd=320 locks held 0 write locks 0 pid/thread 1704/6876
60 dd=319 locks held 0 write locks 0 pid/thread 1704/6380
61 dd=318 locks held 0 write locks 0 pid/thread 1704/6752
62 dd=317 locks held 0 write locks 0 pid/thread 1704/7348
63 dd=316 locks held 0 write locks 0 pid/thread 1704/1844
64 dd=315 locks held 0 write locks 0 pid/thread 1704/5224
65 dd=314 locks held 0 write locks 0 pid/thread 1704/5668
66 dd=313 locks held 0 write locks 0 pid/thread 1704/8164
67 dd=312 locks held 0 write locks 0 pid/thread 1704/6272
68 dd=310 locks held 0 write locks 0 pid/thread 1704/6268
69 dd=309 locks held 0 write locks 0 pid/thread 1704/7148
6a dd=308 locks held 0 write locks 0 pid/thread 1704/7184
6b dd=307 locks held 0 write locks 0 pid/thread 1704/6596
6c dd=306 locks held 0 write locks 0 pid/thread 1704/6044
6d dd=302 locks held 0 write locks 0 pid/thread 1704/6476
6e dd=301 locks held 0 write locks 0 pid/thread 1704/6916
6f dd=300 locks held 0 write locks 0 pid/thread 1704/4784
70 dd=299 locks held 0 write locks 0 pid/thread 1704/7444
71 dd=298 locks held 0 write locks 0 pid/thread 1704/5544
72 dd=297 locks held 0 write locks 0 pid/thread 1704/7140
73 dd=296 locks held 0 write locks 0 pid/thread 1704/5724
74 dd=295 locks held 0 write locks 0 pid/thread 1704/956
75 dd=294 locks held 0 write locks 0 pid/thread 1704/7916
76 dd=293 locks held 0 write locks 0 pid/thread 1704/6116
77 dd=292 locks held 0 write locks 0 pid/thread 1704/7928
78 dd=291 locks held 0 write locks 0 pid/thread 1704/6116
79 dd=290 locks held 0 write locks 0 pid/thread 1704/7160
7a dd=289 locks held 0 write locks 0 pid/thread 1704/6656
7b dd=288 locks held 0 write locks 0 pid/thread 1704/7212
7c dd=287 locks held 0 write locks 0 pid/thread 1704/6116
7d dd=286 locks held 0 write locks 0 pid/thread 1704/5820
7e dd=285 locks held 0 write locks 0 pid/thread 1704/5088
7f dd=284 locks held 0 write locks 0 pid/thread 1704/1108
80 dd=283 locks held 0 write locks 0 pid/thread 1704/7372
81 dd=282 locks held 0 write locks 0 pid/thread 1704/4692
82 dd=281 locks held 0 write locks 0 pid/thread 1704/7460
83 dd=280 locks held 0 write locks 0 pid/thread 1704/5528
84 dd=279 locks held 0 write locks 0 pid/thread 1704/7160
85 dd=278 locks held 0 write locks 0 pid/thread 1704/1168
86 dd=275 locks held 0 write locks 0 pid/thread 1704/5464
87 dd=274 locks held 0 write locks 0 pid/thread 1704/5652
88 dd=273 locks held 0 write locks 0 pid/thread 1704/6628
89 dd=272 locks held 0 write locks 0 pid/thread 1704/5164
8a dd=271 locks held 0 write locks 0 pid/thread 1704/7372
8b dd=270 locks held 0 write locks 0 pid/thread 1704/5236
8c dd=269 locks held 0 write locks 0 pid/thread 1704/6532
8d dd=268 locks held 0 write locks 0 pid/thread 1704/4656
8e dd=267 locks held 0 write locks 0 pid/thread 1704/7708
8f dd=266 locks held 0 write locks 0 pid/thread 1704/5112
90 dd=265 locks held 0 write locks 0 pid/thread 1704/7484
91 dd=259 locks held 0 write locks 0 pid/thread 1704/7352
92 dd=258 locks held 0 write locks 0 pid/thread 1704/7408
93 dd=257 locks held 0 write locks 0 pid/thread 1704/5472
94 dd=256 locks held 0 write locks 0 pid/thread 1704/7372
95 dd=255 locks held 0 write locks 0 pid/thread 1704/7552
96 dd=252 locks held 0 write locks 0 pid/thread 1704/5112
97 dd=249 locks held 0 write locks 0 pid/thread 1704/5472
98 dd=248 locks held 0 write locks 0 pid/thread 1704/5472
99 dd=247 locks held 0 write locks 0 pid/thread 1704/6460
9a dd=246 locks held 0 write locks 0 pid/thread 1704/5432
9b dd=245 locks held 0 write locks 0 pid/thread 1704/7220
9c dd=244 locks held 0 write locks 0 pid/thread 1704/7768
9d dd=243 locks held 0 write locks 0 pid/thread 1704/5088
9e dd=242 locks held 0 write locks 0 pid/thread 1704/5544
9f dd=241 locks held 0 write locks 0 pid/thread 1704/5272
a0 dd=240 locks held 0 write locks 0 pid/thread 1704/7284
a1 dd=239 locks held 0 write locks 0 pid/thread 1704/3884
a2 dd=238 locks held 0 write locks 0 pid/thread 1704/4552
a3 dd=237 locks held 0 write locks 0 pid/thread 1704/6460
a4 dd=236 locks held 0 write locks 0 pid/thread 1704/6324
a5 dd=235 locks held 0 write locks 0 pid/thread 1704/6272
a6 dd=234 locks held 0 write locks 0 pid/thread 1704/7276
a7 dd=233 locks held 0 write locks 0 pid/thread 1704/4552
a8 dd=232 locks held 0 write locks 0 pid/thread 1704/6876
a9 dd=231 locks held 0 write locks 0 pid/thread 1704/5296
aa dd=230 locks held 0 write locks 0 pid/thread 1704/5668
ab dd=229 locks held 0 write locks 0 pid/thread 1704/8164
ac dd=228 locks held 0 write locks 0 pid/thread 1704/5444
ad dd=227 locks held 0 write locks 0 pid/thread 1704/7916
ae dd=226 locks held 0 write locks 0 pid/thread 1704/1944
af dd=225 locks held 0 write locks 0 pid/thread 1704/7040
b0 dd=224 locks held 0 write locks 0 pid/thread 1704/5548
b1 dd=223 locks held 0 write locks 0 pid/thread 1704/3880
b2 dd=222 locks held 0 write locks 0 pid/thread 1704/5428
b3 dd=221 locks held 0 write locks 0 pid/thread 1704/5224
b4 dd=220 locks held 0 write locks 0 pid/thread 1704/6420
b5 dd=219 locks held 0 write locks 0 pid/thread 1704/7104
b6 dd=218 locks held 0 write locks 0 pid/thread 1704/5944
b7 dd=217 locks held 0 write locks 0 pid/thread 1704/7728
b8 dd=216 locks held 0 write locks 0 pid/thread 1704/7632
b9 dd=215 locks held 0 write locks 0 pid/thread 1704/7436
ba dd=214 locks held 0 write locks 0 pid/thread 1704/5472
bb dd=213 locks held 0 write locks 0 pid/thread 1704/7148
bc dd=212 locks held 0 write locks 0 pid/thread 1704/4200
bd dd=211 locks held 0 write locks 0 pid/thread 1704/4716
be dd=210 locks held 0 write locks 0 pid/thread 1704/5652
bf dd=209 locks held 0 write locks 0 pid/thread 1704/7012
c0 dd=208 locks held 0 write locks 0 pid/thread 1704/6532
c1 dd=207 locks held 0 write locks 0 pid/thread 1704/4692
c2 dd=206 locks held 0 write locks 0 pid/thread 1704/7936
c3 dd=205 locks held 0 write locks 0 pid/thread 1704/656
c4 dd=204 locks held 0 write locks 0 pid/thread 1704/6460
c5 dd=203 locks held 0 write locks 0 pid/thread 1704/7348
c6 dd=202 locks held 0 write locks 0 pid/thread 1704/4768
c7 dd=201 locks held 0 write locks 0 pid/thread 1704/6080
c8 dd=200 locks held 0 write locks 0 pid/thread 1704/6380
c9 dd=199 locks held 0 write locks 0 pid/thread 1704/376
ca dd=198 locks held 0 write locks 0 pid/thread 1704/5468
cb dd=197 locks held 0 write locks 0 pid/thread 1704/7128
cc dd=196 locks held 0 write locks 0 pid/thread 1704/6408
cd dd=195 locks held 0 write locks 0 pid/thread 1704/7396
ce dd=194 locks held 0 write locks 0 pid/thread 1704/7972
cf dd=193 locks held 0 write locks 0 pid/thread 1704/5088
d0 dd=192 locks held 0 write locks 0 pid/thread 1704/7848
d1 dd=191 locks held 0 write locks 0 pid/thread 1704/5912
d2 dd=190 locks held 0 write locks 0 pid/thread 1704/5820
d3 dd=189 locks held 0 write locks 0 pid/thread 1704/4552
d4 dd=188 locks held 0 write locks 0 pid/thread 1704/7140
d5 dd=187 locks held 0 write locks 0 pid/thread 1704/5116
d6 dd=186 locks held 0 write locks 0 pid/thread 1704/4784
d7 dd=185 locks held 0 write locks 0 pid/thread 1704/5472
d8 dd=184 locks held 0 write locks 0 pid/thread 1704/5756
d9 dd=183 locks held 0 write locks 0 pid/thread 1704/6584
da dd=182 locks held 0 write locks 0 pid/thread 1704/5236
db dd=181 locks held 0 write locks 0 pid/thread 1704/7444
dc dd=180 locks held 0 write locks 0 pid/thread 1704/7928
dd dd=179 locks held 0 write locks 0 pid/thread 1704/5724
de dd=178 locks held 0 write locks 0 pid/thread 1704/6476
df dd=177 locks held 0 write locks 0 pid/thread 1704/7768
e0 dd=176 locks held 0 write locks 0 pid/thread 1704/5444
e1 dd=175 locks held 0 write locks 0 pid/thread 1704/1944
e2 dd=174 locks held 0 write locks 0 pid/thread 1704/1448
e3 dd=173 locks held 0 write locks 0 pid/thread 1704/5088
e4 dd=172 locks held 0 write locks 0 pid/thread 1704/6420
e5 dd=171 locks held 0 write locks 0 pid/thread 1704/6532
e6 dd=170 locks held 0 write locks 0 pid/thread 1704/5088
e7 dd=169 locks held 0 write locks 0 pid/thread 1704/5272
e8 dd=168 locks held 0 write locks 0 pid/thread 1704/6896
e9 dd=167 locks held 0 write locks 0 pid/thread 1704/5652
ea dd=166 locks held 0 write locks 0 pid/thread 1704/7040
eb dd=165 locks held 0 write locks 0 pid/thread 1704/6272
ec dd=164 locks held 0 write locks 0 pid/thread 1704/3880
ed dd=163 locks held 0 write locks 0 pid/thread 1704/5236
ee dd=162 locks held 0 write locks 0 pid/thread 1704/6532
ef dd=161 locks held 0 write locks 0 pid/thread 1704/656
f0 dd=160 locks held 0 write locks 0 pid/thread 1704/6836
f1 dd=159 locks held 0 write locks 0 pid/thread 1704/7916
f2 dd=158 locks held 0 write locks 0 pid/thread 1704/6324
f3 dd=157 locks held 0 write locks 0 pid/thread 1704/3884
f4 dd=156 locks held 0 write locks 0 pid/thread 1704/7220
f5 dd=155 locks held 0 write locks 0 pid/thread 1704/6188
f6 dd=154 locks held 0 write locks 0 pid/thread 1704/7960
f7 dd=153 locks held 0 write locks 0 pid/thread 1704/7396
f8 dd=152 locks held 0 write locks 0 pid/thread 1704/7632
f9 dd=151 locks held 0 write locks 0 pid/thread 1704/5088
fa dd=150 locks held 0 write locks 0 pid/thread 1704/7728
fb dd=149 locks held 0 write locks 0 pid/thread 1704/5428
fc dd=148 locks held 0 write locks 0 pid/thread 1704/5468
fd dd=147 locks held 0 write locks 0 pid/thread 1704/7276
fe dd=146 locks held 0 write locks 0 pid/thread 1704/4200
ff dd=145 locks held 0 write locks 0 pid/thread 1704/6352
100 dd=144 locks held 0 write locks 0 pid/thread 1704/4896
101 dd=143 locks held 0 write locks 0 pid/thread 1704/5548
102 dd=142 locks held 0 write locks 0 pid/thread 1704/7128
103 dd=141 locks held 0 write locks 0 pid/thread 1704/5088
104 dd=140 locks held 0 write locks 0 pid/thread 1704/5088
105 dd=139 locks held 0 write locks 0 pid/thread 1704/6460
106 dd=138 locks held 0 write locks 0 pid/thread 1704/376
107 dd=137 locks held 0 write locks 0 pid/thread 1704/6956
108 dd=136 locks held 0 write locks 0 pid/thread 1704/7848
109 dd=135 locks held 0 write locks 0 pid/thread 1704/5432
10a dd=134 locks held 0 write locks 0 pid/thread 1704/5912
10b dd=133 locks held 0 write locks 0 pid/thread 1704/6876
10c dd=132 locks held 0 write locks 0 pid/thread 1704/5896
10d dd=131 locks held 0 write locks 0 pid/thread 1704/5156
10e dd=130 locks held 0 write locks 0 pid/thread 1704/6260
10f dd=129 locks held 0 write locks 0 pid/thread 1704/7232
110 dd=128 locks held 0 write locks 0 pid/thread 1704/4784
111 dd=127 locks held 0 write locks 0 pid/thread 1704/6896
112 dd=126 locks held 0 write locks 0 pid/thread 1704/7116
113 dd=125 locks held 0 write locks 0 pid/thread 1704/6476
114 dd=124 locks held 0 write locks 0 pid/thread 1704/5756
115 dd=123 locks held 0 write locks 0 pid/thread 1704/4768
116 dd=122 locks held 0 write locks 0 pid/thread 1704/5944
117 dd=121 locks held 0 write locks 0 pid/thread 1704/7444
118 dd=120 locks held 0 write locks 0 pid/thread 1704/7436
119 dd=119 locks held 0 write locks 0 pid/thread 1704/5236
11a dd=118 locks held 0 write locks 0 pid/thread 1704/5116
11b dd=117 locks held 0 write locks 0 pid/thread 1704/7104
11c dd=116 locks held 0 write locks 0 pid/thread 1704/8144
11d dd=115 locks held 0 write locks 0 pid/thread 1704/5224
11e dd=114 locks held 0 write locks 0 pid/thread 1704/8164
11f dd=113 locks held 0 write locks 0 pid/thread 1704/5544
120 dd=112 locks held 0 write locks 0 pid/thread 1704/5724
121 dd=111 locks held 0 write locks 0 pid/thread 1704/7972
122 dd=110 locks held 1 write locks 0 pid/thread 1704/1108
122 READ 1 HELD entry.dbxml page 13
123 dd=109 locks held 0 write locks 0 pid/thread 1704/5296
124 dd=108 locks held 0 write locks 0 pid/thread 1704/7348
125 dd=107 locks held 0 write locks 0 pid/thread 1704/6080
126 dd=106 locks held 0 write locks 0 pid/thread 1704/7184
126 READ 1 WAIT entry.dbxml page 13
127 dd=105 locks held 0 write locks 0 pid/thread 1704/6408
128 dd=104 locks held 0 write locks 0 pid/thread 1704/7284
129 dd=103 locks held 0 write locks 0 pid/thread 1704/6584
12a dd=102 locks held 0 write locks 0 pid/thread 1704/6216
12b dd=101 locks held 0 write locks 0 pid/thread 1704/6380
12c dd=100 locks held 0 write locks 0 pid/thread 1704/5668
12d dd=99 locks held 0 write locks 0 pid/thread 1704/4120
12e dd=98 locks held 0 write locks 0 pid/thread 1704/5804
12f dd=97 locks held 0 write locks 0 pid/thread 1704/7140
130 dd=96 locks held 0 write locks 0 pid/thread 1704/6712
131 dd=95 locks held 0 write locks 0 pid/thread 1704/6564
132 dd=94 locks held 0 write locks 0 pid/thread 1704/7808
133 dd=93 locks held 0 write locks 0 pid/thread 1704/1844
134 dd=92 locks held 0 write locks 0 pid/thread 1704/4024
135 dd=91 locks held 0 write locks 0 pid/thread 1704/6900
136 dd=90 locks held 0 write locks 0 pid/thread 1704/6928
137 dd=89 locks held 0 write locks 0 pid/thread 1704/7764
138 dd=88 locks held 0 write locks 0 pid/thread 1704/7320
139 dd=87 locks held 0 write locks 0 pid/thread 1704/6752
13a dd=86 locks held 0 write locks 0 pid/thread 1704/6728
13b dd=85 locks held 0 write locks 0 pid/thread 1704/1864
13c dd=84 locks held 0 write locks 0 pid/thread 1704/956
13d dd=83 locks held 0 write locks 0 pid/thread 1704/1168
13d READ 1 WAIT entry.dbxml page 11
13e dd=82 locks held 0 write locks 0 pid/thread 1704/7328
13f dd=81 locks held 0 write locks 0 pid/thread 1704/7496
140 dd=80 locks held 0 write locks 0 pid/thread 1704/6776
141 dd=79 locks held 0 write locks 0 pid/thread 1704/6916
142 dd=78 locks held 0 write locks 0 pid/thread 1704/6948
143 dd=77 locks held 0 write locks 0 pid/thread 1704/4236
144 dd=76 locks held 0 write locks 0 pid/thread 1704/7056
145 dd=75 locks held 0 write locks 0 pid/thread 1704/4176
146 dd=74 locks held 0 write locks 0 pid/thread 1704/6676
147 dd=73 locks held 0 write locks 0 pid/thread 1704/7764
148 dd=72 locks held 0 write locks 0 pid/thread 1704/1108
148 READ 1 WAIT entry.dbxml page 11
149 dd=71 locks held 0 write locks 0 pid/thread 1704/5100
14a dd=70 locks held 0 write locks 0 pid/thread 1704/4724
14b dd=69 locks held 0 write locks 0 pid/thread 1704/7236
14c dd=67 locks held 0 write locks 0 pid/thread 1704/3940
14d dd=66 locks held 0 write locks 0 pid/thread 1704/5688
14e dd=65 locks held 0 write locks 0 pid/thread 1704/6616
14f dd=64 locks held 0 write locks 0 pid/thread 1704/7764
150 dd=63 locks held 0 write locks 0 pid/thread 1704/7456
151 dd=62 locks held 0 write locks 0 pid/thread 1704/7428
152 dd=61 locks held 0 write locks 0 pid/thread 1704/6064
153 dd=60 locks held 0 write locks 0 pid/thread 1704/5464
154 dd=59 locks held 0 write locks 0 pid/thread 1704/7428
155 dd=58 locks held 0 write locks 0 pid/thread 1704/7148
155 READ 1 WAIT entry.dbxml page 13
156 dd=57 locks held 0 write locks 0 pid/thread 1704/7148
157 dd=56 locks held 0 write locks 0 pid/thread 1704/5620
158 dd=55 locks held 0 write locks 0 pid/thread 1704/3716
159 dd=54 locks held 0 write locks 0 pid/thread 1704/7640
15a dd=53 locks held 0 write locks 0 pid/thread 1704/4692
15a READ 1 WAIT entry.dbxml page 13
15b dd=52 locks held 0 write locks 0 pid/thread 1704/4692
15c dd=51 locks held 0 write locks 0 pid/thread 1704/5164
15c READ 1 WAIT entry.dbxml page 13
15d dd=50 locks held 0 write locks 0 pid/thread 1704/5164
15e dd=49 locks held 0 write locks 0 pid/thread 1704/5820
15e READ 1 WAIT entry.dbxml page 13
15f dd=48 locks held 0 write locks 0 pid/thread 1704/5820
160 dd=47 locks held 0 write locks 0 pid/thread 1704/6496
161 dd=46 locks held 0 write locks 0 pid/thread 1704/7624
162 dd=45 locks held 0 write locks 0 pid/thread 1704/4772
163 dd=44 locks held 0 write locks 0 pid/thread 1704/7552
163 READ 1 WAIT entry.dbxml page 13
164 dd=43 locks held 0 write locks 0 pid/thread 1704/7552
165 dd=42 locks held 0 write locks 0 pid/thread 1704/7372
166 dd=41 locks held 0 write locks 0 pid/thread 1704/5536
167 dd=40 locks held 0 write locks 0 pid/thread 1704/6796
168 dd=39 locks held 0 write locks 0 pid/thread 1704/5268
169 dd=38 locks held 0 write locks 0 pid/thread 1704/8024
16a dd=37 locks held 0 write locks 0 pid/thread 1704/6460
16a READ 1 WAIT entry.dbxml page 13
16b dd=36 locks held 0 write locks 0 pid/thread 1704/6460
16c dd=35 locks held 0 write locks 0 pid/thread 1704/4368
16d dd=34 locks held 0 write locks 0 pid/thread 1704/4368
16e dd=33 locks held 0 write locks 0 pid/thread 1704/6016
16f dd=32 locks held 0 write locks 0 pid/thread 1704/6908
170 dd=31 locks held 0 write locks 0 pid/thread 1704/7640
171 dd=30 locks held 0 write locks 0 pid/thread 1704/6016
80000018 dd=445 locks held 0 write locks 0 pid/thread 1704/1944
172 dd=29 locks held 0 write locks 0 pid/thread 1704/6532
80000019 dd=444 locks held 0 write locks 0 pid/thread 1704/6728
173 dd=28 locks held 0 write locks 0 pid/thread 1704/6588
8000001a dd=443 locks held 0 write locks 0 pid/thread 1704/7916
174 dd=27 locks held 0 write locks 0 pid/thread 1704/7420
8000001b dd=442 locks held 0 write locks 0 pid/thread 1704/4200
175 dd=26 locks held 0 write locks 0 pid/thread 1704/6336
8000001c dd=441 locks held 0 write locks 0 pid/thread 1704/5548
176 dd=25 locks held 0 write locks 0 pid/thread 1704/5088
8000001d dd=440 locks held 0 write locks 0 pid/thread 1704/1844
177 dd=24 locks held 0 write locks 0 pid/thread 1704/7212
8000001e dd=439 locks held 0 write locks 0 pid/thread 1704/6876
178 dd=23 locks held 0 write locks 0 pid/thread 1704/7212
8000001f dd=438 locks held 0 write locks 0 pid/thread 1704/5224
179 dd=22 locks held 0 write locks 0 pid/thread 1704/5088
80000020 dd=437 locks held 0 write locks 0 pid/thread 1704/5668
17a dd=20 locks held 0 write locks 0 pid/thread 1704/7660
80000021 dd=436 locks held 0 write locks 0 pid/thread 1704/7348
17b dd=19 locks held 0 write locks 0 pid/thread 1704/7160
80000022 dd=435 locks held 0 write locks 0 pid/thread 1704/8164
17c dd=18 locks held 0 write locks 0 pid/thread 1704/6656
80000023 dd=434 locks held 0 write locks 0 pid/thread 1704/6420
17d dd=17 locks held 0 write locks 0 pid/thread 1704/7740
80000024 dd=433 locks held 0 write locks 0 pid/thread 1704/4120
17e dd=16 locks held 0 write locks 0 pid/thread 1704/5652
17e READ 1 WAIT entry.dbxml page 13
80000025 dd=432 locks held 0 write locks 0 pid/thread 1704/3884
17f dd=15 locks held 0 write locks 0 pid/thread 1704/5652
80000026 dd=431 locks held 0 write locks 0 pid/thread 1704/6272
180 dd=13 locks held 0 write locks 0 pid/thread 1704/7928
180 READ 1 WAIT entry.dbxml page 13
80000027 dd=430 locks held 0 write locks 0 pid/thread 1704/5688
181 dd=12 locks held 0 write locks 0 pid/thread 1704/7928
80000028 dd=429 locks held 0 write locks 0 pid/thread 1704/6380
182 dd=11 locks held 0 write locks 0 pid/thread 1704/6596
182 READ 1 WAIT entry.dbxml page 13
80000029 dd=428 locks held 0 write locks 0 pid/thread 1704/6752
183 dd=10 locks held 0 write locks 0 pid/thread 1704/6596
8000002a dd=427 locks held 0 write locks 0 pid/thread 1704/6584
184 dd= 8 locks held 0 write locks 0 pid/thread 1704/5088
184 READ 1 WAIT entry.dbxml page 13
8000002b dd=426 locks held 0 write locks 0 pid/thread 1704/7328
185 dd= 7 locks held 0 write locks 0 pid/thread 1704/5088
8000002c dd=425 locks held 0 write locks 0 pid/thread 1704/7232
186 dd= 6 locks held 0 write locks 0 pid/thread 1704/5236
186 READ 1 WAIT entry.dbxml page 13
8000002d dd=424 locks held 0 write locks 0 pid/thread 1704/5756
187 dd= 5 locks held 0 write locks 0 pid/thread 1704/5236
8000002e dd=423 locks held 0 write locks 0 pid/thread 1704/5896
188 dd= 4 locks held 0 write locks 0 pid/thread 1704/6532
188 READ 1 WAIT entry.dbxml page 13
8000002f dd=422 locks held 0 write locks 0 pid/thread 1704/7276
189 dd= 3 locks held 0 write locks 0 pid/thread 1704/6532
80000030 dd=421 locks held 0 write locks 0 pid/thread 1704/4176
18a dd= 1 locks held 0 write locks 0 pid/thread 1704/5472
18a READ 1 WAIT entry.dbxml page 13
80000031 dd=420 locks held 0 write locks 0 pid/thread 1704/1448
18b dd= 0 locks held 0 write locks 0 pid/thread 1704/5472
80000032 dd=419 locks held 0 write locks 0 pid/thread 1704/4024
80000033 dd=418 locks held 0 write locks 0 pid/thread 1704/376
80000034 dd=417 locks held 0 write locks 0 pid/thread 1704/7284
80000035 dd=416 locks held 0 write locks 0 pid/thread 1704/5432
80000036 dd=415 locks held 0 write locks 0 pid/thread 1704/6956
80000037 dd=414 locks held 0 write locks 0 pid/thread 1704/6080
80000038 dd=413 locks held 0 write locks 0 pid/thread 1704/6776
80000039 dd=412 locks held 0 write locks 0 pid/thread 1704/5912
8000003a dd=411 locks held 0 write locks 0 pid/thread 1704/5544
8000003b dd=410 locks held 0 write locks 0 pid/thread 1704/656
8000003c dd=409 locks held 0 write locks 0 pid/thread 1704/7220
8000003d dd=408 locks held 0 write locks 0 pid/thread 1704/4768
8000003f dd=407 locks held 0 write locks 0 pid/thread 1704/6092
80000040 dd=406 locks held 0 write locks 0 pid/thread 1704/5804
80000041 dd=405 locks held 0 write locks 0 pid/thread 1704/5272
80000042 dd=404 locks held 0 write locks 0 pid/thread 1704/6712
80000043 dd=403 locks held 0 write locks 0 pid/thread 1704/7056
80000044 dd=402 locks held 0 write locks 0 pid/thread 1704/7104
80000045 dd=401 locks held 0 write locks 0 pid/thread 1704/7040
80000046 dd=400 locks held 0 write locks 0 pid/thread 1704/5116
80000047 dd=399 locks held 0 write locks 0 pid/thread 1704/7728
80000048 dd=398 locks held 0 write locks 0 pid/thread 1704/6900
80000049 dd=397 locks held 0 write locks 0 pid/thread 1704/7936
8000004a dd=396 locks held 0 write locks 0 pid/thread 1704/7236
8000004b dd=395 locks held 0 write locks 0 pid/thread 1704/8144
8000004c dd=394 locks held 0 write locks 0 pid/thread 1704/7140
8000004d dd=393 locks held 0 write locks 0 pid/thread 1704/6324
8000004e dd=392 locks held 0 write locks 0 pid/thread 1704/4552
8000004f dd=391 locks held 0 write locks 0 pid/thread 1704/5944
80000050 dd=390 locks held 0 write locks 0 pid/thread 1704/956
80000051 dd=389 locks held 0 write locks 0 pid/thread 1704/5156
80000052 dd=388 locks held 0 write locks 0 pid/thread 1704/5444
80000053 dd=387 locks held 0 write locks 0 pid/thread 1704/4784
80000054 dd=386 locks held 0 write locks 0 pid/thread 1704/4724
80000055 dd=385 locks held 0 write locks 0 pid/thread 1704/5100
80000056 dd=384 locks held 0 write locks 0 pid/thread 1704/3940
80000057 dd=383 locks held 0 write locks 0 pid/thread 1704/6676
80000058 dd=382 locks held 0 write locks 0 pid/thread 1704/7436
80000059 dd=381 locks held 0 write locks 0 pid/thread 1704/7768
8000005a dd=380 locks held 0 write locks 0 pid/thread 1704/7496
8000005b dd=379 locks held 0 write locks 0 pid/thread 1704/6476
8000005c dd=378 locks held 0 write locks 0 pid/thread 1704/6352
8000005d dd=377 locks held 0 write locks 0 pid/thread 1704/6928
8000005e dd=376 locks held 0 write locks 0 pid/thread 1704/6916
8000005f dd=375 locks held 0 write locks 0 pid/thread 1704/7128
80000060 dd=374 locks held 0 write locks 0 pid/thread 1704/5468
80000061 dd=373 locks held 0 write locks 0 pid/thread 1704/6408
80000062 dd=372 locks held 0 write locks 0 pid/thread 1704/6260
80000063 dd=371 locks held 0 write locks 0 pid/thread 1704/6948
80000064 dd=370 locks held 0 write locks 0 pid/thread 1704/1864
80000065 dd=369 locks held 0 write locks 0 pid/thread 1704/5296
80000066 dd=368 locks held 0 write locks 0 pid/thread 1704/7848
80000067 dd=367 locks held 0 write locks 0 pid/thread 1704/7444
80000068 dd=366 locks held 0 write locks 0 pid/thread 1704/7320
80000069 dd=365 locks held 0 write locks 0 pid/thread 1704/7972
8000006a dd=364 locks held 0 write locks 0 pid/thread 1704/5724
8000006b dd=363 locks held 0 write locks 0 pid/thread 1704/7456
8000006c dd=362 locks held 0 write locks 0 pid/thread 1704/6896
8000006d dd=361 locks held 0 write locks 0 pid/thread 1704/6616
8000006e dd=360 locks held 0 write locks 0 pid/thread 1704/7632
8000006f dd=359 locks held 0 write locks 0 pid/thread 1704/6216
80000070 dd=358 locks held 0 write locks 0 pid/thread 1704/7116
80000071 dd=357 locks held 0 write locks 0 pid/thread 1704/7396
80000072 dd=356 locks held 0 write locks 0 pid/thread 1704/4896
80000073 dd=355 locks held 0 write locks 0 pid/thread 1704/5428
80000074 dd=354 locks held 0 write locks 0 pid/thread 1704/6564
80000075 dd=353 locks held 0 write locks 0 pid/thread 1704/7960
80000076 dd=352 locks held 0 write locks 0 pid/thread 1704/6188
80000077 dd=351 locks held 0 write locks 0 pid/thread 1704/3880
80000078 dd=350 locks held 0 write locks 0 pid/thread 1704/6836
80000079 dd=349 locks held 0 write locks 0 pid/thread 1704/4236
8000007a dd=348 locks held 0 write locks 0 pid/thread 1704/7808
8000007b dd=347 locks held 0 write locks 0 pid/thread 1704/1168
8000007c dd=346 locks held 0 write locks 0 pid/thread 1704/5236
8000007d dd=345 locks held 0 write locks 0 pid/thread 1704/5088
8000007e dd=344 locks held 0 write locks 0 pid/thread 1704/6460
8000007f dd=343 locks held 0 write locks 0 pid/thread 1704/4692
80000081 dd=341 locks held 0 write locks 0 pid/thread 1704/5472
80000082 dd=340 locks held 0 write locks 0 pid/thread 1704/1108
80000083 dd=339 locks held 0 write locks 0 pid/thread 1704/5164
80000084 dd=338 locks held 0 write locks 0 pid/thread 1704/7184
80000085 dd=337 locks held 0 write locks 0 pid/thread 1704/7552
80000086 dd=336 locks held 0 write locks 0 pid/thread 1704/7928
80000092 dd=329 locks held 0 write locks 0 pid/thread 1704/5652
80000094 dd=328 locks held 0 write locks 0 pid/thread 1704/6532
80000096 dd=326 locks held 0 write locks 0 pid/thread 1704/6596
80000098 dd=325 locks held 0 write locks 0 pid/thread 1704/7148
8000009b dd=323 locks held 0 write locks 0 pid/thread 1704/5820
800000a2 dd=434 locks held 0 write locks 0 pid/thread 1704/6420
800000a2 WRITE 1 WAIT entry.dbxml page 11
800000a3 dd=433 locks held 0 write locks 0 pid/thread 1704/4120
800000a3 WRITE 1 WAIT entry.dbxml page 11
800000a4 dd=439 locks held 0 write locks 0 pid/thread 1704/6876
800000a4 WRITE 1 WAIT entry.dbxml page 11
800000a5 dd=429 locks held 0 write locks 0 pid/thread 1704/6380
800000a5 WRITE 1 WAIT entry.dbxml page 11
800000a6 dd=428 locks held 0 write locks 0 pid/thread 1704/6752
800000a6 WRITE 1 WAIT entry.dbxml page 11
800000a7 dd=436 locks held 0 write locks 0 pid/thread 1704/7348
800000a7 WRITE 1 WAIT entry.dbxml page 11
800000a8 dd=435 locks held 0 write locks 0 pid/thread 1704/8164
800000a8 WRITE 1 WAIT entry.dbxml page 11
800000a9 dd=440 locks held 0 write locks 0 pid/thread 1704/1844
800000a9 WRITE 1 WAIT entry.dbxml page 11
800000aa dd=438 locks held 0 write locks 0 pid/thread 1704/5224
800000aa WRITE 1 WAIT entry.dbxml page 11
800000ab dd=437 locks held 0 write locks 0 pid/thread 1704/5668
800000ab WRITE 1 WAIT entry.dbxml page 11
800000ac dd=431 locks held 0 write locks 0 pid/thread 1704/6272
800000ac WRITE 1 WAIT entry.dbxml page 11
800000b9 dd=412 locks held 0 write locks 0 pid/thread 1704/5912
800000b9 WRITE 1 WAIT entry.dbxml page 11
800000bd dd=416 locks held 0 write locks 0 pid/thread 1704/5432
800000bd WRITE 1 WAIT entry.dbxml page 11
800000be dd=409 locks held 0 write locks 0 pid/thread 1704/7220
800000be WRITE 1 WAIT entry.dbxml page 11
800000bf dd=422 locks held 0 write locks 0 pid/thread 1704/7276
800000bf WRITE 1 WAIT entry.dbxml page 11
800000c0 dd=424 locks held 0 write locks 0 pid/thread 1704/5756
800000c0 WRITE 1 WAIT entry.dbxml page 11
800000c1 dd=410 locks held 0 write locks 0 pid/thread 1704/656
800000c1 WRITE 1 WAIT entry.dbxml page 11
800000c5 dd=420 locks held 0 write locks 0 pid/thread 1704/1448
800000c5 WRITE 1 WAIT entry.dbxml page 11
800000c6 dd=417 locks held 0 write locks 0 pid/thread 1704/7284
800000c6 WRITE 1 WAIT entry.dbxml page 11
800000c7 dd=415 locks held 0 write locks 0 pid/thread 1704/6956
800000c7 WRITE 1 WAIT entry.dbxml page 11
800000c8 dd=423 locks held 0 write locks 0 pid/thread 1704/5896
800000c8 WRITE 1 WAIT entry.dbxml page 11
800000c9 dd=401 locks held 0 write locks 0 pid/thread 1704/7040
800000c9 WRITE 1 WAIT entry.dbxml page 11
800000ca dd=418 locks held 0 write locks 0 pid/thread 1704/376
800000ca WRITE 1 WAIT entry.dbxml page 11
800000cb dd=432 locks held 0 write locks 0 pid/thread 1704/3884
800000cb WRITE 1 WAIT entry.dbxml page 11
800000cd dd=393 locks held 0 write locks 0 pid/thread 1704/6324
800000cd WRITE 1 WAIT entry.dbxml page 11
800000d0 dd=427 locks held 0 write locks 0 pid/thread 1704/6584
800000d0 WRITE 1 WAIT entry.dbxml page 11
800000d2 dd=408 locks held 0 write locks 0 pid/thread 1704/4768
800000d2 WRITE 1 WAIT entry.dbxml page 11
800000d3 dd=406 locks held 0 write locks 0 pid/thread 1704/5804
800000d3 WRITE 1 WAIT entry.dbxml page 11
800000d4 dd=405 locks held 0 write locks 0 pid/thread 1704/5272
800000d4 WRITE 1 WAIT entry.dbxml page 11
800000d5 dd=373 locks held 0 write locks 0 pid/thread 1704/6408
800000d5 WRITE 1 WAIT entry.dbxml page 11
800000d6 dd=369 locks held 0 write locks 0 pid/thread 1704/5296
800000d6 WRITE 1 WAIT entry.dbxml page 11
800000d7 dd=389 locks held 0 write locks 0 pid/thread 1704/5156
800000d7 WRITE 1 WAIT entry.dbxml page 11
800000d8 dd=370 locks held 0 write locks 0 pid/thread 1704/1864
800000d8 READ 1 WAIT entry.dbxml page 11
800000d9 dd=411 locks held 0 write locks 0 pid/thread 1704/5544
800000d9 WRITE 1 WAIT entry.dbxml page 11
800000da dd=365 locks held 0 write locks 0 pid/thread 1704/7972
800000da WRITE 1 WAIT entry.dbxml page 11
800000de dd=381 locks held 0 write locks 0 pid/thread 1704/7768
800000de WRITE 1 WAIT entry.dbxml page 11
800000e0 dd=395 locks held 0 write locks 0 pid/thread 1704/8144
800000e0 WRITE 1 WAIT entry.dbxml page 11
800000e1 dd=384 locks held 0 write locks 0 pid/thread 1704/3940
800000e1 READ 1 WAIT entry.dbxml page 11
800000e2 dd=385 locks held 0 write locks 0 pid/thread 1704/5100
800000e2 READ 1 WAIT entry.dbxml page 11
800000e3 dd=396 locks held 0 write locks 0 pid/thread 1704/7236
800000e3 READ 1 WAIT entry.dbxml page 11
800000e4 dd=383 locks held 0 write locks 0 pid/thread 1704/6676
800000e4 READ 1 WAIT entry.dbxml page 11
800000e5 dd=407 locks held 1 write locks 1 pid/thread 1704/6092
800000e5 WRITE 1 WAIT entry.dbxml page 13
800000e5 WRITE 1 HELD entry.dbxml page 11
800000e6 dd=392 locks held 0 write locks 0 pid/thread 1704/4552
800000e6 WRITE 1 WAIT entry.dbxml page 11
800000e7 dd=386 locks held 0 write locks 0 pid/thread 1704/4724
800000e7 READ 1 WAIT entry.dbxml page 11
800000e8 dd=402 locks held 0 write locks 0 pid/thread 1704/7104
800000e8 WRITE 1 WAIT entry.dbxml page 11
800000e9 dd=376 locks held 0 write locks 0 pid/thread 1704/6916
800000e9 READ 1 WAIT entry.dbxml page 11
800000ea dd=379 locks held 0 write locks 0 pid/thread 1704/6476
800000ea WRITE 1 WAIT entry.dbxml page 11
800000eb dd=387 locks held 0 write locks 0 pid/thread 1704/4784
800000eb WRITE 1 WAIT entry.dbxml page 11
800000ec dd=367 locks held 0 write locks 0 pid/thread 1704/7444
800000ec WRITE 1 WAIT entry.dbxml page 11
800000ed dd=394 locks held 0 write locks 0 pid/thread 1704/7140
800000ed WRITE 1 WAIT entry.dbxml page 11
800000ef dd=390 locks held 0 write locks 0 pid/thread 1704/956
800000ef READ 1 WAIT entry.dbxml page 11
800000f0 dd=364 locks held 0 write locks 0 pid/thread 1704/5724
800000f0 WRITE 1 WAIT entry.dbxml page 11
800000f1 dd=443 locks held 0 write locks 0 pid/thread 1704/7916
800000f1 WRITE 1 WAIT entry.dbxml page 11
800000f2 dd=377 locks held 0 write locks 0 pid/thread 1704/6928
800000f2 WRITE 1 WAIT entry.dbxml page 11
800000f4 dd=398 locks held 0 write locks 0 pid/thread 1704/6900
800000f4 WRITE 1 WAIT entry.dbxml page 11
800000f7 dd=374 locks held 0 write locks 0 pid/thread 1704/5468
800000f7 WRITE 1 WAIT entry.dbxml page 11
800000f8 dd=348 locks held 0 write locks 0 pid/thread 1704/7808
800000f8 WRITE 1 WAIT entry.dbxml page 11
800000f9 dd=378 locks held 0 write locks 0 pid/thread 1704/6352
800000f9 WRITE 1 WAIT entry.dbxml page 11
800000fa dd=375 locks held 0 write locks 0 pid/thread 1704/7128
800000fa WRITE 1 WAIT entry.dbxml page 11
800000fb dd=388 locks held 0 write locks 0 pid/thread 1704/5444
800000fb WRITE 1 WAIT entry.dbxml page 11
800000fc dd=400 locks held 0 write locks 0 pid/thread 1704/5116
800000fc WRITE 1 WAIT entry.dbxml page 11
800000fe dd=352 locks held 0 write locks 0 pid/thread 1704/6188
800000fe WRITE 1 WAIT entry.dbxml page 11
80000100 dd=349 locks held 0 write locks 0 pid/thread 1704/4236
80000100 READ 1 WAIT entry.dbxml page 11
80000101 dd=356 locks held 0 write locks 0 pid/thread 1704/4896
80000101 WRITE 1 WAIT entry.dbxml page 11
80000102 dd=444 locks held 0 write locks 0 pid/thread 1704/6728
80000102 WRITE 1 WAIT entry.dbxml page 11
80000103 dd=421 locks held 0 write locks 0 pid/thread 1704/4176
80000103 READ 1 WAIT entry.dbxml page 11
80000104 dd=355 locks held 0 write locks 0 pid/thread 1704/5428
80000104 WRITE 1 WAIT entry.dbxml page 11
80000105 dd=430 locks held 0 write locks 0 pid/thread 1704/5688
80000105 READ 1 WAIT entry.dbxml page 11
80000108 dd=413 locks held 0 write locks 0 pid/thread 1704/6776
80000108 READ 1 WAIT entry.dbxml page 11
8000010b dd=380 locks held 0 write locks 0 pid/thread 1704/7496
8000010b READ 1 WAIT entry.dbxml page 11
8000010c dd=357 locks held 0 write locks 0 pid/thread 1704/7396
8000010c WRITE 1 WAIT entry.dbxml page 11
80000119 dd=445 locks held 0 write locks 0 pid/thread 1704/1944
80000119 WRITE 1 WAIT entry.dbxml page 11
8000011a dd=360 locks held 0 write locks 0 pid/thread 1704/7632
8000011a WRITE 1 WAIT entry.dbxml page 11
8000011b dd=351 locks held 0 write locks 0 pid/thread 1704/3880
8000011b WRITE 1 WAIT entry.dbxml page 11
8000011c dd=441 locks held 0 write locks 0 pid/thread 1704/5548
8000011c WRITE 1 WAIT entry.dbxml page 11
8000011d dd=362 locks held 0 write locks 0 pid/thread 1704/6896
8000011d WRITE 1 WAIT entry.dbxml page 11
8000011e dd=366 locks held 0 write locks 0 pid/thread 1704/7320
8000011e WRITE 1 WAIT entry.dbxml page 11
8000011f dd=414 locks held 0 write locks 0 pid/thread 1704/6080
8000011f WRITE 1 WAIT entry.dbxml page 11
80000120 dd=350 locks held 0 write locks 0 pid/thread 1704/6836
80000120 WRITE 1 WAIT entry.dbxml page 11
80000121 dd=403 locks held 0 write locks 0 pid/thread 1704/7056
80000121 READ 1 WAIT entry.dbxml page 11
80000122 dd=391 locks held 0 write locks 0 pid/thread 1704/5944
80000122 WRITE 1 WAIT entry.dbxml page 11
80000125 dd=425 locks held 0 write locks 0 pid/thread 1704/7232
80000125 WRITE 1 WAIT entry.dbxml page 11
80000126 dd=404 locks held 0 write locks 0 pid/thread 1704/6712
80000126 WRITE 1 WAIT entry.dbxml page 11
80000127 dd=397 locks held 0 write locks 0 pid/thread 1704/7936
80000127 WRITE 1 WAIT entry.dbxml page 11
80000129 dd=399 locks held 0 write locks 0 pid/thread 1704/7728
80000129 WRITE 1 WAIT entry.dbxml page 11
8000012a dd=371 locks held 0 write locks 0 pid/thread 1704/6948
8000012a READ 1 WAIT entry.dbxml page 11
8000012b dd=372 locks held 0 write locks 0 pid/thread 1704/6260
8000012b WRITE 1 WAIT entry.dbxml page 11
8000012e dd=354 locks held 0 write locks 0 pid/thread 1704/6564
8000012e WRITE 1 WAIT entry.dbxml page 11
80000131 dd=353 locks held 0 write locks 0 pid/thread 1704/7960
80000131 WRITE 1 WAIT entry.dbxml page 11
80000132 dd=426 locks held 0 write locks 0 pid/thread 1704/7328
80000132 READ 1 WAIT entry.dbxml page 11
80000134 dd=442 locks held 0 write locks 0 pid/thread 1704/4200
80000134 WRITE 1 WAIT entry.dbxml page 11
80000135 dd=382 locks held 0 write locks 0 pid/thread 1704/7436
80000135 WRITE 1 WAIT entry.dbxml page 11
80000136 dd=419 locks held 0 write locks 0 pid/thread 1704/4024
80000136 WRITE 1 WAIT entry.dbxml page 11
80000137 dd=359 locks held 0 write locks 0 pid/thread 1704/6216
80000137 WRITE 1 WAIT entry.dbxml page 11
8000013a dd=368 locks held 0 write locks 0 pid/thread 1704/7848
8000013a WRITE 1 WAIT entry.dbxml page 11
8000013c dd=361 locks held 0 write locks 0 pid/thread 1704/6616
8000013c READ 1 WAIT entry.dbxml page 11
8000013d dd=358 locks held 0 write locks 0 pid/thread 1704/7116
8000013d WRITE -
Deadlock issues with one thread?
I've been trying to nail down a deadlock issue for quite a while now and I'm just at a loss. My application basically has one main thread of execution which does get(), put(), and delete() on a database, and another background thread that goes through the database periodically (e.g. every 5 minutes) and prunes the db. The pruning thread works by using a cursor to iterate over the db and evaluates each record and, if it's expired, deletes it, stopping when it reaches the end of the DB or when it's deleted 100,000 records. I had been getting DeadlockExceptions and figured it was a conflict between the main thread and the pruning thread competing for locks on the same objects, so I modified my code so that when the pruning thread wakes up and starts pruning, it makes the main thread wait(), so theoretically the two threads should never be doing db operations at the same time. However, this doesn't seem to have resolved the issue, as I still get DeadlockExceptions. I've raised the lock timeout to 1000 ms and set it to retry up to 20x to no avail. I've even tried it with the pruning threads completely disabled and I still got it, so it seems the deadlock is occurring even within the single thread. Any ideas? I'm not using transactions, if that matters.
Also, on this page - http://www.oracle.com/technology/documentation/berkeley-db/db/gsg_txn/JAVA/lockingsubsystem.html#deadlockresolve - there's reference to "EnvironmentConfig.setLockDetectMode()". I'm using JE 3.2.23 and there doesn't appear to be any such function. Is this out-of-date documentation or am I using and old version?The first step is to take a look at the information in the DeadlockException -- it will tell you which threads are conflicting. Please post the stack traces and I'll be happy to help interpret them.
Also please take a look at this FAQ:
http://www.oracle.com/technology/products/berkeley-db/faq/je_faq.html#23
If you set je.txn.deadlockStackTrace and je.txn.dumpLocks as described, you'll get more information when the deadlock occurs.
Also, on this page -
http://www.oracle.com/technology/documentation/berkeley-db/db/gsg_txn/JAVA/lockingsubsystem.html#deadlockresolve
- there's reference to "EnvironmentConfig.setLockDetectMode()". I'm using JE 3.2.23 and
there doesn't appear to be any such function. Is this out-of-date documentation or am I
using and old version?The link above is for documentation in Berkeley DB the C Edition, not Berkeley DB Java Edition (JE) -- you're reading the doc for the wrong product. The JE doc is here:
http://www.oracle.com/technology/documentation/berkeley-db/je/TransactionGettingStarted/jelock.html
There is no lock detect mode in JE, it always uses a timeout of the oldest locker.
Mark -
Reducing number of deadlock exceptions
com.sleepycat.je.DeadlockException: (JE 3.2.13) Lock expired. Locker 15347458_pool-1-thread-10_Txn: waited for lock on database=tasks13587 node=73038 type=WRITE grant=WAIT_NEW timeoutMillis=500 startTime=1170088002362 endTime=1170088002866
Owners: [<LockInfo locker="15347467_pool-1-thread-8_Txn" type="WRITE"/>]
Waiters: []
at com.sleepycat.je.txn.LockManager.lock(LockManager.java:266)
at com.sleepycat.je.txn.Txn.lockInternal(Txn.java:296)
at com.sleepycat.je.txn.Locker.lock(Locker.java:257)
at com.sleepycat.je.dbi.CursorImpl.lockLNDeletedAllowed(CursorImpl.java:2349)
at com.sleepycat.je.dbi.CursorImpl.lockLN(CursorImpl.java:2271)
at com.sleepycat.je.dbi.CursorImpl.searchAndPosition(CursorImpl.java:1963)
at com.sleepycat.je.Cursor.searchInternal(Cursor.java:1178)
at com.sleepycat.je.Cursor.searchAllowPhantoms(Cursor.java:1148)
at com.sleepycat.je.Cursor.search(Cursor.java:1014)
at com.sleepycat.je.Cursor.getSearchKey(Cursor.java:556)
at com.redwerk.webcrawler.storage.url.BDBJEStorage.update(BDBJEStorage.java:298)
at com.redwerk.webcrawler.filter.record.UrlProcessedFilter._accept(UrlProcessedFilter.java:36)
at com.redwerk.webcrawler.filter.record.ChainedFilter.accept(ChainedFilter.java:23)
at com.redwerk.webcrawler.filter.record.ChainedFilter.accept(ChainedFilter.java:25)
at com.redwerk.webcrawler.Crawler.addToQueue(Crawler.java:372)
at com.redwerk.webcrawler.ConsumerThread.processNestedURLs(ConsumerThread.java:343)
at com.redwerk.webcrawler.ConsumerThread.run(ConsumerThread.java:145)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
at java.util.concurrent.FutureTask.run(FutureTask.java:123)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)
such exception occurs about 400 times on each 1500 database operations
the method, which throws such exception, looks like this:
* @see UrlStorageInterface#update(String, Record)
public void update(String key, Record url) throws RetryRequiredException,
RecoveryRequiredException, Exception {
key = UrlNormalizer.toNormalcyForm(key);
if (log.isDebugEnabled())
log.debug("Updating " + url.getUrl());
//put(key, url);
Transaction txn = null;
Record record = null;
TupleBinding valueBinding = new RecordBinding();
DatabaseEntry entry = new DatabaseEntry();
Cursor cursor = null;
try {
txn = env.beginTransaction(null, null);
cursor = tasksDB.openCursor(txn, null);
if (cursor.getSearchKey(new DatabaseEntry(key.getBytes()), entry,
LockMode.RMW) == OperationStatus.SUCCESS) {
record = (Record) valueBinding.entryToObject(entry);
url.cloneInfo(record);
valueBinding.objectToEntry(record, entry);
cursor.putCurrent(entry);
} else {
valueBinding.objectToEntry(url, entry);
tasksDB.put(txn, new DatabaseEntry(key.getBytes()), entry);
cursor.close();
cursor = null;
txn.commit();
txn = null;
} catch (DeadlockException e) {
log.error(e,e);
throw new RetryRequiredException();
} catch (RunRecoveryException e) {
log.error(e,e);
throw new RecoveryRequiredException();
} catch (DatabaseException e) {
log.error(e, e);
} finally {
if (cursor != null)
try {
cursor.close();
} catch (DatabaseException e) {
log.error(e, e);
if (txn != null)
try {
txn.abort();
} catch (DatabaseException e) {
log.error(e, e);
}Probably because many threads are trying to access
the same record, as you say. This will cause lots of
contention (blocking) in any system. Blocking will
cause DeadlockException if the lock timeout is
exceeded.That's true, as far as I can see in logs, it is possible several threads are accessing same record at a time.
May be due to large amount of data records I needto
increase deadlock detection timeout or somethinglike
this?If you increase the lock timeout, you will probably
see less deadlocks. But there will be roughly the
same amount of contention. But you should try
increasing the lock timeout and measure the
performance impact. The number of deadlocks is only
one measurement. The measurement that matters most
is throughput (ops/sec).Looks like I can increase a timeout for all databases only, thus with another database (queue) I will see some issues because there are 2 secondary databases attacted to a queue database.
In a crawler application do many crawling threads
update the same URLs in a real world data set? Yes, imagine you have standard header and footer for a page, and you have several hundreds of thousands pages (online store). Our application is fetching the URLs from the page (including urls from header, which are common for entire website), then, before adding the url to queue, it checks - if the url is downloaded and parsed yet, if so - it updates the record in processed urls database (adds a new parent and probably some another information about the page). So several threads can update same record at a time.
Or does this only occur in your test data set? In fact, it occurs only with live data (customer's website) :(
Whenever many threads are competing for exclusive
access to the same resource (a database record, or
any other shared resource), the only way to reduce
contention is to change the application so that this
is not true.
For example, you can partition the key space so that
each thread that writes to the database works on a
separate key range (where keys are URLs in your
case). This means operations would have to be queued
by crawler threads and dequeued by database threads,
partitioning the keys by thread during the queuing
process. That's one idea. The best approach depends
on trade-offs in your application.Thank you, I was thinking about a bit different idea (which I was implementing earlier, and rejected to place in production) - utilize java.util.concurrent package with it's nice queues. So the storage implementation will be wrapped with decorator, which will intercept put, update and push operations and instead of accessing database - will just add records to queue. Separate thread will populate records from a queue and update database. To avoid case when such improvement is nothing than synchronized operations, it might be useful to add some kind of in-memory updating, so when a record is added to a queue, if record with similar key is already exists in a queue - it will be updated within queue.
Separate thread will fetch records from queue and update respective databases.
Maybe you are looking for
-
Emptied cache and relaunched Safari. The pdf I click on that shows a document goes blank when I click on print. I did learn that other documents in Safari will print after all, but not this one. Enabled cookies to no avail. Firefox works. Must be som
-
Hi All, I have 3 years of Experience in SAP ABAP. Now i got an oppertunity to work on SAP IS oil and Gas. I don't have any Idea on IS oil and gas. how to start learn and is i am the suitable person to learn with ABAP knowledge. I have only one advant
-
Hi, We are looking at generating 1099 forms for previous years i.e. 2011 and 2012. The problem is every year the 1099 form is updated by a SAP Note as per changes made by IRS and whenever I print 2011 data, I get it in 2014 format. Has someone faced
-
CD drive makes a lot of noise when I insert a CD
Whenever I insert a CD into my slot it makes a noise that sounds like its eating my CD -- Not sure if there is always a noise or if i have a problem with my computer.
-
What has happened to the graphics in my emails??
What has happened to all the graphics in my emails?