DeadlockException

Hi, using DPL and 3.1.0 on Win32 platform JDK 1.5.09-b -- simple unit test that does the following:
1) put an object into DPL
2) get it back out and do equality check
3) delete the object from DPL
Step 3 causes a DeadlockException.
Stacktrace follows:
junit.framework.AssertionFailedError: Exception in test: DatabaseException occurred: com.sleepycat.je.DeadlockException: (JE 3.1.0) Lock expired. Locker 14_main_AutoTxn: waited for lock on database=persist#DalStore#FeatureImpl#featureCode node=52 type=WRITE grant=WAIT_NEW timeoutMillis=500 startTime=1161798879187 endTime=1161798880187
Owners: [<LockInfo locker="-1_main_ThreadLocker" type="READ"/>, <LockInfo locker="-1_main_ThreadLocker" type="READ"/>]
Waiters: []
     at junit.framework.Assert.fail(Assert.java:47)
     at TestBerkeleyCache.testPutFeature(TestBerkeleyCache.java:93)
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
     at java.lang.reflect.Method.invoke(Unknown Source)
     at junit.framework.TestCase.runTest(TestCase.java:154)
     at junit.framework.TestCase.runBare(TestCase.java:127)
     at junit.framework.TestResult$1.protect(TestResult.java:106)
     at junit.framework.TestResult.runProtected(TestResult.java:124)
     at junit.framework.TestResult.run(TestResult.java:109)
     at junit.framework.TestCase.run(TestCase.java:118)
     at junit.framework.TestSuite.runTest(TestSuite.java:208)
     at junit.framework.TestSuite.run(TestSuite.java:203)
     at org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference.run(JUnit3TestReference.java:128)
     at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
     at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:460)
     at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:673)
     at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:386)
     at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:196)
Here is the method that causes the exception ->
public void remove(long featureId, int featureCode) throws TADalException {
     EntityJoin<Long, FeatureImpl> join = new EntityJoin<Long, FeatureImpl>(featureByPrimaryKey);
     join.addCondition(featureById, featureId);
     join.addCondition(featureByCode, featureCode);
     ForwardCursor<Long> cursor = null;
     try {
         cursor = join.keys();
         long id = cursor.next();
         featureByPrimaryKey.delete(id);
     } catch (DatabaseException e) {
         throw new TADalException("DatabaseException occurred: " + e);
     } finally {
         try {
          if (cursor != null) {
              cursor.close();
         } catch (DatabaseException dbe) {
          throw new TADalException("DatabaseException occurred: " + dbe);
    }Any ideas on whether I'm doing something I shouldn't here? Thanks in advance...
-davis

Hi Mark --
the problem I have is that EntityCursor is a
sub-interface of ForwardCursor and ForwardCursor has
no delete operation.I'm sorry, I didn't even notice that your cursor was returned with a join.
How to get at delete( ) then?If you want to delete/modify/insert while a join cursor is open, and you want to use a transactional store, then you will need to:
1) Before doing the join, start a transaction by calling Environment.beginTransaction.
2) Pass the transaction to join.keys() so that the transaction is used by the join cursor.
3) Pass the transaction to featureByPrimaryKey.delete() also.
4) When you're done, after closing all cursors, commit the transaction by calling Transaction.commit.
In general, when you are using transactions, if a cursor is open in the current thread and you perform other non-cursor operations in that thread, all operations must use the same transaction. In other words, you cannot use auto-commit for the delete operation when a cursor is open. (Auto-commit is the default when no transaction is passed and the store is transactional.)
Mark

Similar Messages

  • DeadlockException on putNoReturn()

    I'm building up a non-transactional data store with multiple worker threads with code that follows the pattern:
    PrimaryIndex<Long, BoundObjectImpl> boundObjectById;
    boundObjectById.putNoReturn(newBoundObjectInstance);
    I'm running this on a box with 8 processors using 15 worker threads to keep the CPUs saturated through I/O blocking as the source data is coming from an Oracle database. I tried increasing the lockTables count to 3, which helped on a dual-processor dev environment, but I'm wondering if I should bump it higher for the 8-way environment?
    This is the exception:
    DatabaseException occurred: com.sleepycat.je.DeadlockException: (JE
    3.1.25 (15233)) Lock expired. Locker -1_pool-1-thread-8_ThreadLocker:
    waited for lock on
    database=persist#BoundObjectStore#test.boundobject.BoundObjectStoreImpl#targetObjectCode
    node=53112855 type=WRITE grant=WAIT_NEW timeoutMillis=1000 startTime=1164987874050 endTime=1164987879589
    Also, is there any way to control the number of worker threads spawned by the Berkeley database? It subverts my own thread management if the database is spawning a large number of threads for each worker thread I create.
    Thanks,
    Andy

    Hello Andy,
    I'm running this on a box with 8 processors using 15
    worker threads to keep the CPUs saturated through I/O
    blocking as the source data is coming from an Oracle
    database. I tried increasing the lockTables count to
    3, which helped on a dual-processor dev environment,
    but I'm wondering if I should bump it higher for the
    8-way environment?Yes, I think so, but I'll let others answer that are more knowledgeable than I in this area.
    DatabaseException occurred:
    com.sleepycat.je.DeadlockException: (JE
    3.1.25 (15233)) Lock expired. Locker
    -1_pool-1-thread-8_ThreadLocker:
    waited for lock on
    database=persist#BoundObjectStore#test.boundobject.Bou
    ndObjectStoreImpl#targetObjectCode
    node=53112855 type=WRITE grant=WAIT_NEW
    timeoutMillis=1000 startTime=1164987874050
    endTime=1164987879589This exception is not related to the number of lock tables. The number of lock tables is an optimization that does not impact deadlocking.
    Deadlocks, how to handle them and how to prevent them, are explained here:
    http://www.oracle.com/technology/documentation/berkeley-db/je/TransactionGettingStarted/blocking_deadlocks.html
    Also, is there any way to control the number of
    worker threads spawned by the Berkeley database? It
    subverts my own thread management if the database is
    spawning a large number of threads for each worker
    thread I create.JE creates a fixed number of threads, it does not create worker threads dynamically. By default it creates three threads: the log cleaner, checkpointer and compressor. Is that too many for your application?
    Mark

  • Multiple issues, Berkeley JE 3.2.15, 3.2.76

    Hi all,
    We have been using JE 3.2.15 with great satisfaction for more than a year, but we've been recently running into what seems to be a deadlock with the following stack trace:
    #1) "Thread-12" daemon prio=4 tid=0x0b174320 nid=0x72 in Object.wait() [0x208cd000..0x208cdbb8]
    at java.lang.Object.wait(Native Method)
    - waiting on <0xf4202fc0> (a com.sleepycat.je.txn.Txn)
    at com.sleepycat.je.txn.LockManager.lock(LockManager.java:227)
    - locked <0xf4202fc0> (a com.sleepycat.je.txn.Txn)
    at com.sleepycat.je.txn.Txn.lockInternal(Txn.java:295)
    at com.sleepycat.je.txn.Locker.lock(Locker.java:283)
    at com.sleepycat.je.dbi.CursorImpl.lockLNDeletedAllowed(CursorImpl.java:2375)
    at com.sleepycat.je.dbi.CursorImpl.lockLN(CursorImpl.java:2297)
    at com.sleepycat.je.dbi.CursorImpl.searchAndPosition(CursorImpl.java:1983)
    at com.sleepycat.je.Cursor.searchInternal(Cursor.java:1188)
    at com.sleepycat.je.Cursor.searchAllowPhantoms(Cursor.java:1158)
    at com.sleepycat.je.Cursor.search(Cursor.java:1024)
    at com.sleepycat.je.Database.get(Database.java:557)
    The call does not seem to return, not for as long as 22 hours anyway. Or maybe it's just very slow performing a single get (these are done in loops), giving the appearance that it's stuck. It appears more or less randomly (every couple of months, maybe) at various sites. We have not been able to reproduce it on our test systems.
    We recently upgraded to 3.2.76, kinda hoping that this issue had been fixed. While the new version appeared to work fine during internal testing and then for 2 deployments, if failed repeatedly during the third deployment. The original issue just manifested itself again (the above stack trace is from 3.2.76). We also experienced the following:
    #2) Environment invalid because of previous exception: com.sleepycat.je.log.DbChecksumException: (JE 3.2.76) Read invalid log entry type: 49
    at com.sleepycat.je.log.LogEntryHeader.<init>(LogEntryHeader.java:69)
    at com.sleepycat.je.log.LogManager.getLogEntryFromLogSource(LogManager.java:631)
    at com.sleepycat.je.log.LogManager.getLogEntry(LogManager.java:597)
    at com.sleepycat.je.tree.IN.fetchTarget(IN.java:958)
    at com.sleepycat.je.dbi.CursorImpl.searchAndPosition(CursorImpl.java:1963)
    at com.sleepycat.je.Cursor.searchInternal(Cursor.java:1188)
    at com.sleepycat.je.Cursor.searchAllowPhantoms(Cursor.java:1158)
    at com.sleepycat.je.Cursor.search(Cursor.java:1024)
    at com.sleepycat.je.Database.get(Database.java:557)
    #3) <DaemonThread name="Checkpointer"/> caught exception: com.sleepycat.je.DatabaseException: (JE 3.2.76) fetchTarget of 0x61d/0xa2d452 parent IN=147872417 lastFullVersion=0x69c/0x27e251 parent.getDirty()=true state=0 com.sleepycat.je.log.DbChecksumException: (JE 3.2.76) Read invalid log entry type: 58
    com.sleepycat.je.DatabaseException: (JE 3.2.76) fetchTarget of 0x61d/0xa2d452 parent IN=147872417 lastFullVersion=0x69c/0x27e251 parent.getDirty()=true state=0 com.sleepycat.je.log.DbChecksumException: (JE 3.2.76) Read invalid log entry type: 58
    at com.sleepycat.je.tree.IN.fetchTarget(IN.java:989)
    at com.sleepycat.je.cleaner.Cleaner.migrateLN(Cleaner.java:1100)
    at com.sleepycat.je.cleaner.Cleaner.lazyMigrateLNs(Cleaner.java:928)
    at com.sleepycat.je.tree.BIN.logInternal(BIN.java:1117)
    at com.sleepycat.je.tree.IN.log(IN.java:2657)
    at com.sleepycat.je.recovery.Checkpointer.logTargetAndUpdateParent(Checkpointer.java:975)
    at com.sleepycat.je.recovery.Checkpointer.flushIN(Checkpointer.java:810)
    at com.sleepycat.je.recovery.Checkpointer.flushDirtyNodes(Checkpointer.java:670)
    at com.sleepycat.je.recovery.Checkpointer.doCheckpoint(Checkpointer.java:442)
    at com.sleepycat.je.recovery.Checkpointer.onWakeup(Checkpointer.java:211)
    at com.sleepycat.je.utilint.DaemonThread.run(DaemonThread.java:191)
    at java.lang.Thread.run(Thread.java:595)
    Caused by: com.sleepycat.je.log.DbChecksumException: (JE 3.2.76) Read invalid log entry type: 58
    at com.sleepycat.je.log.LogEntryHeader.<init>(LogEntryHeader.java:69)
    at com.sleepycat.je.log.LogManager.getLogEntryFromLogSource(LogManager.java:631)
    at com.sleepycat.je.log.LogManager.getLogEntry(LogManager.java:597)
    at com.sleepycat.je.tree.IN.fetchTarget(IN.java:958)
    ... 11 more
    * An out of memory condition:
    #4) Exception in thread "Cleaner-1" java.lang.OutOfMemoryError: Java heap space
    at com.sleepycat.je.log.LogUtils.readByteArray(LogUtils.java:204)
    at com.sleepycat.je.log.entry.LNLogEntry.readEntry(LNLogEntry.java:104)
    at com.sleepycat.je.log.FileReader.readEntry(FileReader.java:238)
    at com.sleepycat.je.log.CleanerFileReader.processEntry(CleanerFileReader.java:140)
    at com.sleepycat.je.log.FileReader.readNextEntry(FileReader.java:321)
    at com.sleepycat.je.cleaner.FileProcessor.processFile(FileProcessor.java:411)
    at com.sleepycat.je.cleaner.FileProcessor.doClean(FileProcessor.java:259)
    at com.sleepycat.je.cleaner.FileProcessor.onWakeup(FileProcessor.java:161)
    at com.sleepycat.je.utilint.DaemonThread.run(DaemonThread.java:191)
    at java.lang.Thread.run(Thread.java:595)
    It is possible, of course, that the actual memory issue is somewhere else in the application and Berkeley just happened to be the one getting the error. However the load is very much 24 hours-periodic and the most significant change was the Berkeley upgrade, which does not seem to be running entirely correctly...
    I was wondering:
    - is issue #1 indeed a deadlock? Has anyone else experienced it? Is there a fix/workaround?
    - are #2 and #3 known issues? Note that these happened both while processing 3.2.15 files and, after failure and deletion, pure 3.2.76 files.
    - is it possible that an invalid log entry would cause #4? Or is there a memory leak/incorrect allocation in BDB?
    If that helps, the standard installation is on Sun x86 hardware running Solaris and (Sun) Java 1.5.0_11. Berkeley is run as an XA resource. Like I said we haven't seen any of these issues on our internal test systems. The customer that experienced #2, #3 and #4 has since been rolled back to 3.2.15, and #1 is rather infrequent. So that's pretty much all the information we have and are going to get, but if there's anything else you'd like to know...
    Thanks in advance,
    Matthieu Bentot

    Hello Matthieu,
    Thank you for your clear explanation.
    On #1, the deadlock, I don't see any known JE bug that could cause this, in the current release or that has been fixed since 3.2.15. The thread dump you sent implies that a thread is waiting to acquire a record lock, which means that some other thread or transaction holds the record lock at that time. There are a few things that come to mind:
    1) If you are setting the lock timeout to a large value, or to zero which means "wait forever", this kind of problem could occur as the result of normal record lock contention. I assume you're not doing this or you would have mentioned it, but I thought I should ask. Are you calling EnvironmentConfig or TransactionConfig.setLockTimeout, or setting the lock timeout via the je.properties file?
    2) Are you performing retries when DeadlockException is thrown (as suggested in our documentation)? By retry I mean that when you catch DeadlockException, you close any open cursors, abort the transaction (if you're using transactions), and then start the operation or transaction from the beginning. One possibility is that two threads are continuously retrying, both trying to access the same record(s). If so, one possible solution is to delay for a small time interval before retrying, to give other threads a chance to finish their operation or transaction.
    3) The most common reason for the problem you're seeing is that a cursor or transaction is accidentally left open. A cursor or transaction holds record locks until it is closed (or committed or aborted in the case of a transaction). If you have a cursor or transaction that has "leaked" -- been discarded without being closed -- or you have a thread that keeps a cursor or transaction open for a long period, another thread trying to access the record will be unable to lock it. Normally, DeadlockException will be thrown in one thread or the other. But if you are setting the lock timeout (1) or retrying operations (2), this could cause the problem you're seeing. Are you seeing DeadlockException?
    If you can describe more about how you handle DeadlockException, send your environment, database, cursor and transaction configuration settings, and describe how you close cursors and commit/abort transactions, hopefully we will be able to narrow this down.
    Your #2 and #3 are the same problem, which is very likely a known problem that we're currently focussed on. It is interesting that you never saw this problem in 3.2.15 -- that could be a clue for us.
    When you started seeing the problem, were there any changes other than moving to JE 3.2.76? Did you start using a network file system of some kind? Or use different hardware that provides more concurrency? Or update your application to perform with more concurrency?
    You mentioned that this occurred in a deployment but not in your test environment. If you have the JE log files from that deployment and/or you can reproduce the problem there, we would very much like to work with you to resolve this. If so, please send me an email -- mark.hayes at the obvious .com -- and I'll ask you to send log files and/or run with a debugging JE jar file when trying to reproduce the problem.
    It is possible that #4 is a side effect of the earlier problems, but it's very difficult to say for sure. I don't see any known bugs since 3.2.76 that could cause an OutOfMemoryError.
    Please send us more information on #1 as I mentioned above, and send me email on the checksum problem (#2 and #3).
    Thanks,
    --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           

  • Suspicious error, maybe 3.2.68 related?

    This is very tentative, but wanted to relay details in case it meant anything to the BDB team:
    One of the users of our crawler tried dropping in 3.2.68 as a replacement for 3.2.44, and after about one weeks' runtime hit similar, unexpected OutOfMemoryError stacks on two separate machines. A representative stack:
    Exception in thread "ToeThread #87: " java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Arrays.java:2882)
    at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
    at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
    at java.lang.StringBuffer.append(StringBuffer.java:224)
    at com.sleepycat.je.DatabaseException.<init>(DatabaseException.java:25)
    at com.sleepycat.je.DeadlockException.<init>(DeadlockException.java:29)
    at com.sleepycat.je.txn.LockManager.makeTimeoutMsgInternal(LockManager.java:432)
    at com.sleepycat.je.txn.SyncedLockManager.makeTimeoutMsg(SyncedLockManager.java:76)
    at com.sleepycat.je.txn.LockManager.lock(LockManager.java:262)
    at com.sleepycat.je.txn.BasicLocker.lockInternal(BasicLocker.java:94)
    at com.sleepycat.je.txn.Locker.lock(Locker.java:257)
    at com.sleepycat.je.dbi.CursorImpl.lockLNDeletedAllowed(CursorImpl.java:2375)
    at com.sleepycat.je.dbi.CursorImpl.lockLN(CursorImpl.java:2297)
    at com.sleepycat.je.dbi.CursorImpl.searchAndPosition(CursorImpl.java:1983)
    at com.sleepycat.je.dbi.DbTree.modifyDbRoot(DbTree.java:336)
    at com.sleepycat.je.recovery.Checkpointer.flushIN(Checkpointer.java:750)
    at com.sleepycat.je.recovery.Checkpointer.flushDirtyNodes(Checkpointer.java:669)
    at com.sleepycat.je.recovery.Checkpointer.syncDatabase(Checkpointer.java:571)
    at com.sleepycat.je.dbi.DatabaseImpl.sync(DatabaseImpl.java:545)
    at com.sleepycat.je.Database.sync(Database.java:322)
    Of course the real cause of the memory depletion may be elsewhere, and indeed in our project's code, but that this BDB-JE operation triggered heap depletion twice was at least a little suspicious. (Might the DeadlockException message length grow to extremes in certain situations?)
    Our users' original comment is viewable at http://webteam.archive.org/jira/browse/HER-1387 .
    - Gordon @ IA

    I've asked our user about the je.txn.dumpLocks value, as well as a couple other questions to try to narrow down what happened around the error.
    It's still very possible something in our code is largely to blame, and I'd have to relay the patch to the 3rd party for testing as we haven't yet seen this here -- so a patch isn't needed, yet, but thanks for the quick offer.
    I'll update with anything else I learn.

  • Non-transactional cursor writes block reads

    I'm opening a cursor without using a transaction like so:
    CursorConfig conf = new CursorConfig();
    Cursor cursor = db.openCursor(null, conf);
    I then iterate over the items, sometimes replacing a value. I want to be able to read (and maybe write) to any value from another thread while the cursor is still open, but gets result in the following error:
    com.sleepycat.je.DeadlockException: (JE 3.3.74) Lock expired. Locker 32098350 -1_NioProcessor-3_ThreadLocker: waited for lock on database=settings LockAddr:1537969 node=2623139 type=READ grant=WAIT_NEW timeoutMillis=500 startTime=1229438453225 endTime=1229438453730
    Owners: [<LockInfo locker="11743647 -1_NioProcessor-1_ThreadLocker" type="WRITE"/>]
    Waiters: []
    I would have expected that outside a transaction any update made through a cursor would be applied immediately, but instead the record is locked until the cursor is closed. This happens whether or not je.env.isTransactional is set to true.
    Setting je.env.isLocking to false gives me the cursor behaviour that I want, but I'm not sure what else is affected by this setting, and the javadoc comment is just vaguely worrying rather than helpful ("This property should be set to false only in special circumstances when it is safe to run without record locking.")
    I am not planning on using any transactions in this application.

    Cormac,
    I was looking through the documentation for a reference page to direct you to, and I realized that probably the clearest explanation in the javadoc is in the com.sleepycat.je.LockMode page here: [http://www.oracle.com/technology/documentation/berkeley-db/je/java/com/sleepycat/je/LockMode.html]. There is a reference to it from the CursorConfig page, but I can see that the link between the two pages is not that clear.
    From the LockMode javadoc, there is this paragraph.
    bq. Locking Rules \\ Together with CursorConfig, TransactionConfig and EnvironmentConfig settings, lock mode parameters determine how records are \\ locked during read operations. Record locking is used to enforce the \\ isolation modes that are configured. Record locking is summarized below for \\ read and write operations. For more information on isolation levels and \\ transactions, see [Writing Transactional Applications|http://www.oracle.com/technology/documentation/berkeley-db/je/TransactionGettingStarted/index.html]. \\ With one exception, a record lock is always acquired when a record is \\ read or written, and a cursor will always hold the lock as long as it is \\ positioned on the record. The exception is when [READ_UNCOMMITTED|http://www.oracle.com/technology/documentation/berkeley-db/je/java/com/sleepycat/je/LockMode.html#READ_UNCOMMITTED] \\ is specified, which allows a record to be read without any locking.
    Transactional cursors, by default, hold all locks until the cursor is closed. In your case, you are using a non transactional cursor, and locks are not collected the same way. But a lock for modified record is still taken in a non transactional cursor, and is held as long as the cursor is positioned at that record, to provide cursor stability. I am guessing that is what is happening in your case. If you close the cursor or move to a different position, you will be releasing the lock.
    In a second posting, you comment on your experiences using the je.env.locking property. That property is really only to be used in cases where the application is very constrained and has high performance requirements; frankly, that was put in for a particular set of power users and we really don't recommend it for general purpose use. (We should doc it better too). One characteristic is that it disables log cleaning, because the lack of locks makes it impossible for us to properly coordinate access to the data from the log cleaning threads.
    Regards,
    Linda
    Edited by: Linda Lee on Dec 16, 2008 2:19 PM (changed " link between the two pages is that clear" -> not that clear)

  • Put failure behavior in a non-transactional db?

    Hi all,
    I'm still getting to know the ropes here, so hopefully this isn't a stupid question. I'm still prototyping an application, and as such I haven't set my database to be transactional (learning one thing at a time). The app runs for quite a while, but eventually it tries to put an entity into a primary index that violates one of the secondary key constraints (uniqueness). The put() call throws an exception (properly showing KEYEXIST) but then all subsequent operations that I try to perform fail with a lock timeout (DeadlockException).
    My guess is that some resource wasn't cleaned up after the exception since I'm not running transactionally (so no auto-commit), but I can't find any documentation to back that up. Am I likely to avoid this problem if I turn on transactions? If I don't, is there something in the API I can do to clean up after an exception in a non-transactional put?
    Thanks!
    Jeff Alexander

    You may want to check to make sure that you're not leaving a cursor open when an exception occurs. An open cursor will hold a lock. I recommend closing cursors in a finally block.
    If a lock is held by mistake (either your mistake such as not closing a cursor, or a bug in JE), a deadlock may not be caused immediately because other operations may not need to lock the same record. To debug this, perhaps after receiving the KEYEXIST exception and closing cursors properly, you could try closing all Databases and the Environment. Environment.close will throw an exception if any locks were not released.
    It would be nice if there were a way to query the number of outstanding locks, but I don't think we have a way to do that currently, without reaching into JE internals.
    --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               

  • Berkeley DB and DB optimization

    Hi,
    I have been testing BerkeleyDB-4.7.25 with 16 *.bdb files using BTREE on a Linux server 64bits. Each *.bdb file reaches approximately a size of 3.2 GB.
    I have run a set of operations that include puts/gets/updates/deletes
    I would like to ask a couple of questions, please:
    1)
    Is there is any Berkeley DB tool/function to optimize the *.bdb files for/after deletion.
    2)
    I have been running dbstat -e (please find the output of db_stat below) trying to improve some of the DB_CONFIG parameters.
    set_flags DB_TXN_WRITE_NOSYNC
    set_cachesize 0 2147483648 1
    mutex_set_max 1000000
    set_tx_max 500000
    set_lg_regionmax 524288
    set_lg_bsize 4194304
    set_lg_max 20971520
    set_lk_max_locks 10000
    set_lk_max_lockers 10000
    set_lk_max_objects 10000
    I have increased the cache size, but it does not seem to be helping to improve the operation response times.
    I would really appreciate any help.
    Would the use of DB_SYSTEM_MEM (create the shared regions in system shared memory) help ?
    Would the preallocation of the db files help ?
    Would the increase of the log buffer help ?
    Would the size of the log help (based on the values related to data written since last checkpoint in db_stat) ?
    Could you please help ?
    Thanks,
    Mariella
    This is the output of db_stat -e:
    0x40988 Log magic number
    14 Log version number
    4MB Log record cache size
    0 Log file mode
    20Mb Current log file size
    72M Records entered into the log (72944260)
    92GB 761MB 385KB 636B Log bytes written
    1GB 805MB 40KB 747B Log bytes written since last checkpoint
    6596982 Total log file I/O writes
    0 Total log file I/O writes due to overflow
    7295 Total log file flushes
    39228 Total log file I/O reads
    4749 Current log file number
    18526992 Current log file offset
    4748 On-disk log file number
    20970984 On-disk log file offset
    1 Maximum commits in a log flush
    1 Minimum commits in a log flush
    4MB 512KB Log region size
    303613 The number of region locks that required waiting (0%)
    100 Last allocated locker ID
    0x7fffffff Current maximum unused locker ID
    9 Number of lock modes
    10000 Maximum number of locks possible
    10000 Maximum number of lockers possible
    10000 Maximum number of lock objects possible
    40 Number of lock object partitions
    16 Number of current locks
    274 Maximum number of locks at any one time
    7 Maximum number of locks in any one bucket
    0 Maximum number of locks stolen by for an empty partition
    0 Maximum number of locks stolen for any one partition
    100 Number of current lockers
    108 Maximum number of lockers at any one time
    16 Number of current lock objects
    176 Maximum number of lock objects at any one time
    4 Maximum number of lock objects in any one bucket
    0 Maximum number of objects stolen by for an empty partition
    0 Maximum number of objects stolen for any one partition
    118M Total number of locks requested (118356655)
    118M Total number of locks released (118356639)
    119802 Total number of locks upgraded
    16 Total number of locks downgraded
    20673 Lock requests not available due to conflicts, for which we waited
    0 Lock requests not available due to conflicts, for which we did not wait
    0 Number of deadlocks
    0 Lock timeout value
    0 Number of locks that have timed out
    500000 Transaction timeout value
    0 Number of transactions that have timed out
    7MB 768KB The size of the lock region
    5019 The number of partition locks that required waiting (0%)
    328 The maximum number of times any partition lock was waited for (0%)
    0 The number of object queue operations that required waiting (0%)
    280 The number of locker allocations that required waiting (0%)
    958 The number of region locks that required waiting (0%)
    4 Maximum hash bucket length
    2GB Total cache size
    1 Number of caches
    1 Maximum number of caches
    2GB Pool individual cache size
    0 Maximum memory-mapped file size
    0 Maximum open file descriptors
    0 Maximum sequential buffer writes
    0 Sleep after writing maximum sequential buffers
    0 Requested pages mapped into the process' address space
    150M Requested pages found in the cache (92%)
    12M Requested pages not found in the cache (12855704)
    8449044 Pages created in the cache
    12M Pages read into the cache (12855704)
    20M Pages written from the cache to the backing file (20044721)
    32M Clean pages forced from the cache (32698230)
    1171137 Dirty pages forced from the cache
    9227380 Dirty pages written by trickle-sync thread
    505880 Current total page count
    356352 Current clean page count
    149528 Current dirty page count
    262147 Number of hash buckets used for page location
    184M Total number of times hash chains searched for a page (184542797)
    34 The longest hash chain searched for a page
    945M Total number of hash chain entries checked for page (945465289)
    430 The number of hash bucket locks that required waiting (0%)
    34 The maximum number of times any hash bucket lock was waited for (0%)
    5595 The number of region locks that required waiting (0%)
    0 The number of buffers frozen
    0 The number of buffers thawed
    0 The number of frozen buffers freed
    34M The number of page allocations (34375350)
    76M The number of hash buckets examined during allocations (76979039)
    18 The maximum number of hash buckets examined for an allocation
    33M The number of pages examined during allocations (33869157)
    4 The max number of pages examined for an allocation
    2 Threads waited on page I/O
    Pool File: file_p10.bdb
    4096 Page size
    0 Requested pages mapped into the process' address space
    9376233 Requested pages found in the cache (92%)
    800764 Requested pages not found in the cache
    4096 Page size
    0 Requested pages mapped into the process' address space
    9376233 Requested pages found in the cache (92%)
    800764 Requested pages not found in the cache
    526833 Pages created in the cache
    800764 Pages read into the cache
    1179504 Pages written from the cache to the backing file
    Pool File: file_p3.bdb
    4096 Page size
    4658/8873223 File/offset for last checkpoint LSN
    Thu Apr 30 22:00:23 2009 Checkpoint timestamp
    0x806584b8 Last transaction ID allocated
    500000 Maximum number of active transactions configured
    0 Active transactions
    8 Maximum active transactions
    6653112 Number of transactions begun
    60327 Number of transactions aborted
    6592785 Number of transactions committed
    144048 Snapshot transactions
    257302 Maximum snapshot transactions
    0 Number of transactions restored
    185MB 24KB Transaction region size
    90116 The number of region locks that required waiting (0%)
    Active transactions:
    =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
    129MB 720KB Mutex region size
    108 The number of region locks that required waiting (0%)
    4 Mutex alignment
    200 Mutex test-and-set spins
    1000000 Mutex total count
    331261 Mutex free count
    668739 Mutex in-use count
    781915 Mutex maximum in-use count
    Mutex counts
    331259 Unallocated
    16 db handle
    1 env dblist
    2 env handle
    1 env region
    43 lock region
    274 logical lock
    1 log filename
    1 log flush
    2 log region
    16 mpoolfile handle
    16 mpool filehandle
    17 mpool file bucket
    1 mpool handle
    262147 mpool hash bucket
    262147 mpool buffer I/O
    1 mpool region
    1 mutex region
    1 twister
    1 txn active list
    1 transaction checkpoint
    144050 txn mvcc
    1 txn region

    user11096811 wrote:
    i have same questionWhat is the question exactly? What DB release are you using?
    user11096811 wrote:
    the app throws com.sleepycat.db.LockNotGrantedException .what should i do?The LockNotGrantedException exception being thrown is a subclass of DeadlockException.
    A LockNotGrantedException is thrown when a lock requested using the Environment.getLock or Environment.lockVector methods, where the noWait flag or lock timers were configured, could not be granted before the wait-time expired.
    Additionally, LockNotGrantedException is thrown when a Concurrent Data Store database environment configured for lock timeouts was unable to grant a lock in the allowed time.
    Additionally, LockNotGrantedException is thrown when lock or transaction timeouts have been configured and a database operation has timed out. Applications can handle all deadlocks by
    catching the DeadlockException. You can read more on how to configure the locking subsystem and resolve the deadlocks at [The Locking Subsystem|http://www.oracle.com/technology/documentation/berkeley-db/db/gsg_txn/JAVA/lockingsubsystem.html].
    Thanks,
    Bogdan Coman

  • Truncate db throws exception

    Hi All,
    I have two processes, one creates environement in RW mode, creates a transactional database(s), writes n records, and closes it.
    Another process opens the same env (by this I means same directory), in RO mode (I understand this gives snapshot view as of open time), finds all databases in that environment (getDatabaseNames), if works on that data put in by process 1, and truncates the db. Truncate throws foll. error.
    com.sleepycat.je.DeadlockException: (JE 3.2.13) Lock expired. Locker 250_main_Txn: waited for lock on database=_jeNameMap node=34 type=WRITE grant=WAIT_PROMOTION timeoutMillis=500 startTime=1187817739998 endTime=1187817740998
    Owners: [<LockInfo locker="250_main_Txn" type="READ"/>, <LockInfo locker="-1_main_BasicLocker" type="READ"/>]
    Waiters: []
    Transaction 250_main_Txn owns 34 <LockInfo locker="250_main_Txn" type="READ"/>
    Transaction 250_main_Txn waits for node 34
    How to address this ?
    Logically this is what I am trying to do. Process 1 accumulates new data and Process 2 processes the data once a number of new records created by P1 reach a threshold. Once processed, it wants to drop those records. I thought easiest way is to do truncate.
    P1 uses databases in a circular fashion (much like redo logs). Say there are 5 databases to start with, P1 places records into DB1, then DB2, ..and uses DB1 again after DB5. In the mean time, P2 should have emptied records from DB1 etc when P1 returns to it. P2 cannot open the databases in RW mode for obvious reasons, so it cant delete records. P1 closes db soon it writes threshold # of records.
    Any suggestions on how I might go about this ?
    Thanks
    Vissu

    Hi Vissu,
    Truncating a database is a write operation, so you can't do it in a read-only environment.
    If you must write the data in one process and read/truncate it in another, then the only ways I think of doing that are:
    1) The writer process must close the environment while the reader/truncator process truncates it. This is difficult since it requires coordinating the two processes.
    2) Instead of rotating through 5 databases, rotate through 5 environments (each with a single database). That way, the reader/truncator can open the environment read-write, after it has been closed by the writer.
    I recommend trying (2). Of course, things will be much simpler if you can use a single process for reading, reading and truncating.
    --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               

  • Serious lock problem - transaction has active cursors

    Hi,
    i use BDB C via the Java-API (4.6.21, developing with WindowsXP). Currently i am having serious problems with LOCK_NOT_GRANTED_EXCEPTIONS.
    My application is a queue implementation. One push objects into the database, another one reads them and a third one deletes entries with an outdated TTL (time-to-live). All of these processes are done by multiple dedicated threads what leads to highly concurrent situations.
    The TTL cleaning uses a tx-cursor which deletes entries that are too old. The inserter simply uses put(). The reader uses a cursor too, which starts with a pointer (via getSearchKey) or at the beginning of the database (if no start could be found, the pointer is then outdated as it was deleted by TTL cleaning run).
    The isolation level is at READ_COMMITTED. Sometimes i use RMW locks for reading and updating status information.
    So i implemented a lock detection which retries at least the inserts if a lock fails. The reads can fail and report their exceptions to the outside world. This seems to work fine. Sometimes i get some LOCK_NOT_GRANTED_EXCEPTIONS and the recovery should work. But it works only sometimes.
    BDB-C complains with "transaction has active cursors" (see txn.c - line 1186). All subsequent calls to rollback or commit or any other method of BDB-C handles fails then with a RunRecoveryException. I am absolutely sure, i have no open cursors, as i close them implicitly before tx.commit() or tx.abort(). I wrote a reference counter to ensure that and it shows always a count of 0 open cursors as expected.
    The whole process is of course quite heavy stress for the DB - i am actually read, write, delete and checkpoint the db at the same position at the same time.
    I am tending to introduce MVCC to avoid those locking issues. As i understood concurrent updates of the same db page with multiversion transactions may also cause DeadLockExceptions. Can i use RMW locks to avoid concurrent updates on the same pages with MVCC?
    Some hints?
    TIA
    Dirk

    Hi,
    I have created a patch for Berkeley DB 4.6.21, which outputs information about all cursor open and close operations. This should help you see which cursor is not being closed.
    If you apply the patch below to your Berkeley DB tree, and rebuild.
    Then add setVerbose(VerboseConfig.CURSOR, true), to your EnvironmentConfig initialization.
    It will send log messages to the output stream configured via EnvironmentConfig.setMessageStream(). The default is System.out.
    There will likely be a lot of output.
    Please let me know if this helps you resolve the problem. Otherwise, ask more questions ;)
    Regards,
    Alex Gorrod
    diff -rc db-4.6.21.orig/build_windows/db.h db-4.6.21/build_windows/db.h
    *** db-4.6.21.orig/build_windows/db.h     Fri Sep 28 01:32:08 2007
    --- db-4.6.21/build_windows/db.h     Mon May 12 17:29:52 2008
    *** 60,66 ****
      #define     DB_VERSION_MAJOR     4
      #define     DB_VERSION_MINOR     6
      #define     DB_VERSION_PATCH     21
    ! #define     DB_VERSION_STRING     "Berkeley DB 4.6.21: (September 27, 2007)"
    --- 60,66 ----
      #define     DB_VERSION_MAJOR     4
      #define     DB_VERSION_MINOR     6
      #define     DB_VERSION_PATCH     21
    ! #define     DB_VERSION_STRING     "Berkeley DB 4.6.21: (May 12, 2008)"
    *** 2133,2138 ****
    --- 2133,2139 ----
      #define     DB_VERB_REGISTER     0x0010     /* Dump waits-for table. */
      #define     DB_VERB_REPLICATION     0x0020     /* Replication information. */
      #define     DB_VERB_WAITSFOR     0x0040     /* Dump waits-for table. */
    + #define     DB_VERB_CURSOR          0x0080     /* Cursor open/close information. */
           u_int32_t      verbose;     /* Verbose output. */
           void          app_private;     / Application-private handle. */
    diff -rc db-4.6.21.orig/db/db_am.c db-4.6.21/db/db_am.c
    *** db-4.6.21.orig/db/db_am.c     Thu Jun 14 04:21:30 2007
    --- db-4.6.21/db/db_am.c     Mon May 12 17:32:08 2008
    *** 283,288 ****
    --- 283,291 ----
           TAILQ_INSERT_TAIL(&dbp->active_queue, dbc, links);
           F_SET(dbc, DBC_ACTIVE);
           MUTEX_UNLOCK(dbenv, dbp->mutex);
    +      if (FLD_ISSET(dbenv->verbose, DB_VERB_CURSOR))
    +           __db_msg(dbenv,
    +               "Cursor (%p) opened, associated txn: %p", dbc, dbc->txn);
           *dbcp = dbc;
           return (0);
    diff -rc db-4.6.21.orig/db/db_cam.c db-4.6.21/db/db_cam.c
    *** db-4.6.21.orig/db/db_cam.c     Tue Jun  5 21:46:24 2007
    --- db-4.6.21/db/db_cam.c     Mon May 12 17:32:25 2008
    *** 71,76 ****
    --- 71,79 ----
           opd = cp->opd;
           ret = 0;
    +      if (FLD_ISSET(dbenv->verbose, DB_VERB_CURSOR))
    +           __db_msg(dbenv,
    +               "Cursor (%p) closed, associated txn: %p", dbc, dbc->txn);
            * Remove the cursor(s) from the active queue.  We may be closing two
            * cursors at once here, a top-level one and a lower-level, off-page
    diff -rc db-4.6.21.orig/dbinc/db.in db-4.6.21/dbinc/db.in
    *** db-4.6.21.orig/dbinc/db.in     Fri Jun 29 00:23:35 2007
    --- db-4.6.21/dbinc/db.in     Mon May 12 17:23:20 2008
    *** 2106,2111 ****
    --- 2106,2112 ----
      #define     DB_VERB_REGISTER     0x0010     /* Dump waits-for table. */
      #define     DB_VERB_REPLICATION     0x0020     /* Replication information. */
      #define     DB_VERB_WAITSFOR     0x0040     /* Dump waits-for table. */
    + #define     DB_VERB_CURSOR          0x0080     /* Cursor open/close information. */
           u_int32_t      verbose;     /* Verbose output. */
           void          app_private;     / Application-private handle. */
    diff -rc db-4.6.21.orig/dist/pubdef.in db-4.6.21/dist/pubdef.in
    *** db-4.6.21.orig/dist/pubdef.in     Fri Jul  6 10:22:52 2007
    --- db-4.6.21/dist/pubdef.in     Mon May 12 17:21:48 2008
    *** 432,437 ****
    --- 432,438 ----
      DB_USERCOPY_SETDATA          * I N
      DB_USE_ENVIRON               D I J
      DB_USE_ENVIRON_ROOT          D I J
    + DB_VERB_CURSOR               D I J
      DB_VERB_DEADLOCK          D I J
      DB_VERB_FILEOPS               D I J
      DB_VERB_FILEOPS_ALL          D I J
    diff -rc db-4.6.21.orig/env/env_method.c db-4.6.21/env/env_method.c
    *** db-4.6.21.orig/env/env_method.c     Fri May 18 01:15:11 2007
    --- db-4.6.21/env/env_method.c     Mon May 12 17:28:57 2008
    *** 1070,1075 ****
    --- 1070,1076 ----
           int *onoffp;
           switch (which) {
    +      case DB_VERB_CURSOR:
           case DB_VERB_DEADLOCK:
           case DB_VERB_FILEOPS:
           case DB_VERB_FILEOPS_ALL:
    *** 1098,1103 ****
    --- 1099,1105 ----
           int on;
           switch (which) {
    +      case DB_VERB_CURSOR:
           case DB_VERB_DEADLOCK:
           case DB_VERB_FILEOPS:
           case DB_VERB_FILEOPS_ALL:
    diff -rc db-4.6.21.orig/java/src/com/sleepycat/db/EnvironmentConfig.java db-4.6.21/java/src/com/sleepycat/db/EnvironmentConfig.java
    *** db-4.6.21.orig/java/src/com/sleepycat/db/EnvironmentConfig.java     Fri Jul  6 10:22:54 2007
    --- db-4.6.21/java/src/com/sleepycat/db/EnvironmentConfig.java     Mon May 12 17:25:22 2008
    *** 114,119 ****
    --- 114,120 ----
          private boolean yieldCPU = false;
          /* Verbose Flags */
    +     private boolean verboseCursor = false;
          private boolean verboseDeadlock = false;
          private boolean verboseFileops = false;
          private boolean verboseFileopsAll = false;
    *** 846,851 ****
    --- 847,855 ----
          public void setVerbose(final VerboseConfig flag, boolean enable) {
              int iflag = flag.getInternalFlag();
              switch (iflag) {
    +         case DbConstants.DB_VERB_CURSOR:
    +             verboseCursor = enable;
    +             break;
              case DbConstants.DB_VERB_DEADLOCK:
                  verboseDeadlock = enable;
                  break;
    *** 876,881 ****
    --- 880,887 ----
          public boolean getVerbose(final VerboseConfig flag) {
              int iflag = flag.getInternalFlag();
              switch (iflag) {
    +         case DbConstants.DB_VERB_CURSOR:
    +             return verboseCursor;
              case DbConstants.DB_VERB_DEADLOCK:
                  return verboseDeadlock;
              case DbConstants.DB_VERB_FILEOPS:
    *** 1151,1156 ****
    --- 1157,1167 ----
                  dbenv.set_flags(offFlags, false);
              /* Verbose flags */
    +         if (verboseCursor && !oldConfig.verboseCursor)
    +             dbenv.set_verbose(DbConstants.DB_VERB_CURSOR, true);
    +         if (!verboseCursor && oldConfig.verboseCursor)
    +             dbenv.set_verbose(DbConstants.DB_VERB_CURSOR, false);
    +
              if (verboseDeadlock && !oldConfig.verboseDeadlock)
                  dbenv.set_verbose(DbConstants.DB_VERB_DEADLOCK, true);
              if (!verboseDeadlock && oldConfig.verboseDeadlock)
    *** 1342,1347 ****
    --- 1353,1359 ----
              yieldCPU = ((envFlags & DbConstants.DB_YIELDCPU) != 0);
              /* Verbose flags */
    +         verboseCursor = dbenv.get_verbose(DbConstants.DB_VERB_CURSOR);
              verboseDeadlock = dbenv.get_verbose(DbConstants.DB_VERB_DEADLOCK);
              verboseFileops = dbenv.get_verbose(DbConstants.DB_VERB_FILEOPS);
              verboseFileopsAll = dbenv.get_verbose(DbConstants.DB_VERB_FILEOPS_ALL);
    diff -rc db-4.6.21.orig/java/src/com/sleepycat/db/VerboseConfig.java db-4.6.21/java/src/com/sleepycat/db/VerboseConfig.java
    *** db-4.6.21.orig/java/src/com/sleepycat/db/VerboseConfig.java     Fri Jul  6 10:22:54 2007
    --- db-4.6.21/java/src/com/sleepycat/db/VerboseConfig.java     Mon May 12 17:28:07 2008
    *** 12,17 ****
    --- 12,19 ----
      import com.sleepycat.db.internal.DbEnv;
      public final class VerboseConfig {
    +     public static final VerboseConfig CURSOR =
    +         new VerboseConfig("CURSOR", DbConstants.DB_VERB_CURSOR);
          public static final VerboseConfig DEADLOCK =
              new VerboseConfig("DEADLOCK", DbConstants.DB_VERB_DEADLOCK);
          public static final VerboseConfig FILEOPS =
    diff -rc db-4.6.21.orig/java/src/com/sleepycat/db/internal/DbConstants.java db-4.6.21/java/src/com/sleepycat/db/internal/DbConstants.java
    *** db-4.6.21.orig/java/src/com/sleepycat/db/internal/DbConstants.java     Fri Sep 28 01:32:07 2007
    --- db-4.6.21/java/src/com/sleepycat/db/internal/DbConstants.java     Mon May 12 17:29:17 2008
    *** 194,199 ****
    --- 194,200 ----
          int DB_UPGRADE = 0x0000001;
          int DB_USE_ENVIRON = 0x0004000;
          int DB_USE_ENVIRON_ROOT = 0x0008000;
    +     int DB_VERB_CURSOR = 0x0080;
          int DB_VERB_DEADLOCK = 0x0001;
          int DB_VERB_FILEOPS = 0x0002;
          int DB_VERB_FILEOPS_ALL = 0x0004;

  • Deadlocks and more on a large database.

    Hi,
    We are running a multithreaded server. The database is performing as cache, which is queried/updated about 500 times/sec by about 500 threads.
    Most queries are misses.
    We are experiencing a large number of deadlocks in the database, and sometime get/put operations fails to complete even after 20 retries.
    We would please like to know the following (not all questions related to the deadlock issue):
    1. How can we reduce the number of deadlocks ?
    2. Is using deferred mode favorable in this use case ?
    3. How do we find the optimal log file size ?
    4. Are there any more configuration properties we need to consider ?
    Thanks,
    Lior

    've tried changing the lock timeout to the default,
    as specified in the je.properties file. (50000).
    however, it seems the change has no effect, and the
    deadlocks occur in the same frequency as before.Ok.
    we've also tried without much success:
    envConfig.setConfigParam("je.env.sharedLatches",
    "true");
    envConfig.setConfigParam("je.lock.nLockTables",
    "401");The above params have no impact on deadlocks.
    Is there any other configuration properties I can
    change?No, except perhaps to increase the cache size so that operations complete more quickly. An operation that does a read from disk will take much longer than when the record is in cache.
    Here's the stack trace, after modifying the lock
    timeout property:
    2008-07-23 16:55:54,977 WARN
    [com.flash.http.postOptCache.service.PostOptCacheFacad
    eImpl] com.sleepycat.je.DeadlockException: (JE
    3.2.68) Lock expired. Locker
    -1_HPI-executor707_ThreadLocker: waited for lock on
    database=postoptcache node=4211902 type=WRITE
    grant=WAIT_NEW timeoutMillis=500
    startTime=1216821354475 endTime=1216821354977
    Owners: [<LockInfo
    locker="-1_HPI-executor825_ThreadLocker"
    type="WRITE"/>]
    Waiters: []
    com.sleepycat.je.DeadlockException: (JE 3.2.68) Lock
    expired. Locker -1_HPI-executor707_ThreadLocker:
    waited for lock on database=postoptcache node=4211902
    type=WRITE grant=WAIT_NEW timeoutMillis=500
    startTime=1216821354475 endTime=1216821354977This information tells us that thread HPI-executor707 is trying to perform a write operation and times out after 500 millis. The write lock on the record is not granted because thread HPI-executor825 owns the write lock on that record.
    Thread HPI-executor707 is performing a Database.delete. We don't know what thread HPI-executor825 is doing, but we can assume that is holding the record lock for more than 500 millis.
    You have a couple options:
    1) Find out why HPI-executor825 is holding the lock for so long, and try to correct that problem. For example, if it is keeping a cursor open, try to close the cursor earlier.
    2) If you're sure that no thread is keeping a cursor open for any longer than necessary, then it is possible that something else is slowing things down. Is I/O very slow on this system? Are full GCs taking a long time? If so, you can try to correct these problems, but you can also increase the lock timeout to a value larger than any delay caused by I/O or GC.
    Basically, you need to find out why the record lock is being held by thread HPI-executor825 for more than 500 millis. Then either correct that situation so that the lock is not held for so long, or increase the lock timeout if there is no way to reduce the time that the lock is held.
    --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   

  • Concurrently write and read caused program hang infinitely.

    Hi, everyone
    I used BDB XML 2.4.11
    os : windows xp service pack 2
    I have set the java option -Xms64M -Xmx256M
    I used a util class for retrieving the XmlManager and XmlContainer handle, the code is below
    import java.io.File;
    import java.io.FileNotFoundException;
    import java.util.HashMap;
    import java.util.Iterator;
    import java.util.Map;
    import com.sleepycat.db.CheckpointConfig;
    import com.sleepycat.db.DatabaseException;
    import com.sleepycat.db.Environment;
    import com.sleepycat.db.EnvironmentConfig;
    import com.sleepycat.db.LockDetectMode;
    import com.sleepycat.dbxml.XmlContainer;
    import com.sleepycat.dbxml.XmlContainerConfig;
    import com.sleepycat.dbxml.XmlException;
    import com.sleepycat.dbxml.XmlManager;
    import com.sleepycat.dbxml.XmlManagerConfig;
    public class BdbXmlUtil {
        private static Map<String, XmlContainer> openedContainer = new HashMap<String, XmlContainer>();
        private static XmlManager xmlManager = null;
        public static synchronized XmlManager getXmlManager() {
            if (xmlManager == null) {
                EnvironmentConfig envConf = new EnvironmentConfig();
                envConf.setAllowCreate(true);
                envConf.setInitializeCache(true);
                envConf.setInitializeLocking(true);
                envConf.setInitializeLogging(true);
                envConf.setRunRecovery(true);
                envConf.setTransactional(true);
                envConf.setLockDetectMode(LockDetectMode.MINWRITE);
                envConf.setCacheSize(256 * 1024 * 1024);
                envConf.setTxnMaxActive(5000);
                envConf.setMultiversion(true);
                envConf.setTemporaryDirectory(new File("E:/temp/bdb-xml/tmp"));
                envConf.setErrorStream(System.err);
                // for performance
                // The setTxnNoWait has to set ture, only the transaction set true will not take effect.
                envConf.setTxnNoSync(true);
                //envConf.setTxnNoWait(true);
                Environment environment = null;
                try {
                    environment = new Environment(new File("E:/temp/bdb-xml/bdb_xml_testenv"), envConf);
                    CheckpointConfig checkpointConf = new CheckpointConfig();
                    checkpointConf.setKBytes(1024);
                    environment.checkpoint(checkpointConf);
                } catch (FileNotFoundException e) {
                    e.printStackTrace();
                } catch (DatabaseException e) {
                    e.printStackTrace();
                XmlManagerConfig managerConfig = new XmlManagerConfig();
                managerConfig.setAdoptEnvironment(true);
                try {
                    xmlManager = new XmlManager(environment, managerConfig);
                } catch (XmlException e) {
                    e.printStackTrace();
            return xmlManager;
        public static synchronized XmlContainer getXmlContainer(String xmlContainerName) {
            XmlContainer xmlContainer = null;
            if (openedContainer.containsKey(xmlContainerName)) {
                xmlContainer =
                    (XmlContainer) openedContainer.get(xmlContainerName);
            } else {
                XmlContainerConfig containerConf = new XmlContainerConfig();
                containerConf.setTransactional(true);
                containerConf.setAllowCreate(true);
                containerConf.setNodeContainer(false);
                containerConf.setMultiversion(true);
                /*containerConf.setReadUncommitted(true);*/
                try {
                    xmlContainer =
                        xmlManager.openContainer(
                            xmlContainerName, containerConf);
                } catch (XmlException e) {
                    e.printStackTrace();
                openedContainer.put(xmlContainerName, xmlContainer);
            return xmlContainer;
        public static synchronized void closeEnv() {
            try {
                Iterator<String> containerNames = openedContainer.keySet().iterator();
                while (containerNames.hasNext()) {
                    XmlContainer container = openedContainer.get(containerNames.next());
                    container.close();
                if (xmlManager != null) {
                    xmlManager.close();
            } catch (Exception e) {
                e.printStackTrace();
    }I used two classes to test the concurrency, one is responsible for put document to database, and the other is used to query database while the put operation is being performed. The run method of the two classes is below:
    put document process code:
            boolean retry = true;
            int retry_count = 0;
            // while loop is used for deadlock retries
            while (retry) {
                XmlTransaction txn = null;
                try {
                    XmlUpdateContext context = xmlManager.createUpdateContext();
                    TransactionConfig tc = new TransactionConfig();
                    tc.setNoSync(true);
                    txn = xmlManager.createTransaction(null, tc);
                    String doc = generateId() + ".xml";
                    xmlContainer.putDocument(txn, doc, "<user><name>hello world</name></user>", context);
                    txn.commit();
                    txn = null;
                    System.out.println("created : " + doc);
                    retry = false;
                } catch (XmlException xe) {
                    retry = false;
                    if (xe.getDatabaseException() instanceof DeadlockException) {
                        System.out.println(getName()
                                + " got deadlock exception!");
                        if (retry_count < 20) {
                            System.err.println(getName() + " : Retrying operation.");
                            retry = true;
                            retry_count++;
                            try {
                                Thread.sleep(100);
                            } catch (InterruptedException e) {}
                        } else {
                            System.err.println(getName()
                                    + " : out of retries. Giving up.");
                    } else {
                        System.err.println("XmlException during concurrent-test: "
                                + xe.getMessage());
                } finally {
                    if (txn != null) {
                        try {
                            txn.abort();
                        } catch (Exception e) {
                            System.err.println("Error aborting txn in concurrent-test: "
                                    + e.toString());
                            e.printStackTrace();
            }the query process code :
            XmlManager xmlManager = BdbXmlUtil.getXmlManager();
            XmlResults res = null;
            XmlQueryExpression expr = null;
            XmlTransaction txn = null;
            try {
                BdbXmlUtil.getXmlContainer("entry.dbxml");
                XmlQueryContext qc = xmlManager.createQueryContext();
                expr = xmlManager.prepare("doc('dbxml:/entry.dbxml/1.xml')//user/name", qc);
                TransactionConfig tc = new TransactionConfig();
                tc.setSnapshot(true);
                txn = xmlManager.createTransaction(null, tc);
                res = expr.execute(txn, qc);
                System.out.println("The query, '" + expr.getQuery() +
                           "'\n\t returned " + res.size() + " result(s)");
                while (res.hasNext()) {
                    XmlValue value = res.next();
                    System.out.println(value.asString());
                txn.commit();
                txn = null;
                increaseInvokeCount();
            } catch (XmlException e) {
                e.printStackTrace();
            } finally {
                if (res != null) {
                    res.delete();
                if (expr != null) {
                    expr.delete();
                if (txn != null) {
                    try {
                        txn.abort();
                    } catch (Exception e) {
                        System.err.println("Error aborting txn in query test: "
                                + e.toString());
                        e.printStackTrace();
            }in the main class, I have generated 100 threads to run the put document process and 30 threads to run the query process. After insert several documents to database, the whole process hangs infinitely. I think it is caused by the query process. When i set the query thread number to 1 or disable the query process, the put document process ended with success.
    I think it is caused by the lock. I try to execute the query in Snapshot isolation, but it doesn't works. I also tried the No Wait On Blocks, it seem ok, but alway throws DeadLockException, which can not bear. I doubt about that the lock granularity is per page, and in my program I used the Wholedoc Containers. The Query Process just take action in one document, which is 1.xml, but the lock seem that the whole container is locked. Could it be the page is so large that it covers the whole container? I referenced the document Getting Started with Transaction Processing For Java, but find no hint.
    The BDB XML is excellent in performance, and I want use it in my program. I really hope that this problem can be solved. I don't know whether it is a bug or my reason.Any suggestion will be very helpful, thanks advance.
    Jhon Kao
    Message was edited to better format the Java code:
    gmfeinberg
    added some more info to describe the problem:
    Excalibur
    null

    Dear gmfeinberg,
    You are so kind. I appreciate very much for your approval. You are alway very warm-hearted to help others. Thanks very much.
    I followed your advice and changed the prepare() to add the transaction as a parameter. The complete code I have already posted above. But unfortunately the program hangs also. The complete output of the db_stat -CA when the hang happens is below:
    Default locking region information:
    395     Last allocated locker ID
    0x7fffffff     Current maximum unused locker ID
    9     Number of lock modes
    1000     Maximum number of locks possible
    1000     Maximum number of lockers possible
    1000     Maximum number of lock objects possible
    132     Number of current locks
    134     Maximum number of locks at any one time
    544     Number of current lockers
    582     Maximum number of lockers at any one time
    12     Number of current lock objects
    19     Maximum number of lock objects at any one time
    3935     Total number of locks requested
    3697     Total number of locks released
    0     Total number of locks upgraded
    58     Total number of locks downgraded
    257     Lock requests not available due to conflicts, for which we waited
    101     Lock requests not available due to conflicts, for which we did not wait
    0     Number of deadlocks
    0     Lock timeout value
    0     Number of locks that have timed out
    0     Transaction timeout value
    0     Number of transactions that have timed out
    472KB     The size of the lock region
    938     The number of region locks that required waiting (3%)
    =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
    Lock REGINFO information:
    Lock     Region type
    5     Region ID
    __db.005     Region name
    0x720000     Original region address
    0x720000     Region address
    0x720048     Region primary address
    0     Region maximum allocation
    0     Region allocated
    Region allocations: 3005 allocations, 0 failures, 0 frees, 1 longest
    REGION_JOIN_OK     Region flags
    =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
    Lock region parameters:
    65571     Lock region region mutex [938/28484 3% 4244/7164]
    1031     locker table size
    1031     object table size
    448     obj_off
    45880     locker_off
    0     need_dd
    =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
    Lock conflict matrix:
    0     0     0     0     0     0     0     0     0     
    0     0     1     0     1     0     1     0     1     
    0     1     1     1     1     1     1     1     1     
    0     0     0     0     0     0     0     0     0     
    0     1     1     0     0     0     0     1     1     
    0     0     1     0     0     0     0     0     1     
    0     1     1     0     0     0     0     1     1     
    0     0     1     0     1     0     1     0     0     
    0     1     1     0     1     1     1     0     1     
    =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
    Locks grouped by lockers:
    Locker   Mode      Count Status  ----------------- Object ---------------
           2 dd=464 locks held 2    write locks 0    pid/thread 1704/5836  
           2 READ          1 HELD    entry.dbxml               handle        2
           2 READ          1 HELD    entry.dbxml               handle        0
           5 dd=463 locks held 0    write locks 0    pid/thread 1704/5836  
           6 dd=462 locks held 1    write locks 0    pid/thread 1704/5836  
           6 READ          1 HELD    entry.dbxml               handle        4
           9 dd=461 locks held 0    write locks 0    pid/thread 1704/5836  
           a dd=460 locks held 1    write locks 0    pid/thread 1704/5836  
           a READ          1 HELD    entry.dbxml               handle        6
           d dd=459 locks held 0    write locks 0    pid/thread 1704/5836  
           e dd=458 locks held 1    write locks 0    pid/thread 1704/5836  
           e READ          1 HELD    entry.dbxml               handle        8
          11 dd=457 locks held 0    write locks 0    pid/thread 1704/5836  
          12 dd=456 locks held 1    write locks 0    pid/thread 1704/5836  
          12 READ          1 HELD    entry.dbxml               handle       10
          15 dd=455 locks held 0    write locks 0    pid/thread 1704/5836  
          16 dd=454 locks held 2    write locks 0    pid/thread 1704/5836  
          16 READ          1 HELD    entry.dbxml               handle       12
          16 READ          5 HELD    entry.dbxml               handle        0
          19 dd=453 locks held 0    write locks 0    pid/thread 1704/7764  
          1d dd=452 locks held 1    write locks 0    pid/thread 1704/5836  
          1d READ          1 HELD    entry.dbxml               handle       14
          20 dd=451 locks held 0    write locks 0    pid/thread 1704/6460  
          21 dd=450 locks held 2    write locks 0    pid/thread 1704/5836  
          21 READ          1 HELD    entry.dbxml               handle       16
          21 READ          2 HELD    entry.dbxml               handle        0
          24 dd=449 locks held 0    write locks 0    pid/thread 1704/5836  
          52 dd=448 locks held 2    write locks 0    pid/thread 1704/5836  
          52 READ          1 HELD    entry.dbxml               handle       18
          52 READ          1 HELD    entry.dbxml               handle        0
          55 dd=447 locks held 0    write locks 0    pid/thread 1704/5836  
          56 dd=446 locks held 1    write locks 0    pid/thread 1704/1168  
          56 READ          1 HELD    entry.dbxml               page         13
          57 dd=335 locks held 0    write locks 0    pid/thread 1704/4692  
          58 dd=334 locks held 0    write locks 0    pid/thread 1704/4692  
          59 dd=333 locks held 0    write locks 0    pid/thread 1704/4692  
          5a dd=332 locks held 0    write locks 0    pid/thread 1704/4692  
          5b dd=331 locks held 0    write locks 0    pid/thread 1704/4692  
          5c dd=330 locks held 0    write locks 0    pid/thread 1704/4692  
          5d dd=327 locks held 0    write locks 0    pid/thread 1704/7184  
          5e dd=324 locks held 0    write locks 0    pid/thread 1704/1168  
          5f dd=320 locks held 0    write locks 0    pid/thread 1704/6876  
          60 dd=319 locks held 0    write locks 0    pid/thread 1704/6380  
          61 dd=318 locks held 0    write locks 0    pid/thread 1704/6752  
          62 dd=317 locks held 0    write locks 0    pid/thread 1704/7348  
          63 dd=316 locks held 0    write locks 0    pid/thread 1704/1844  
          64 dd=315 locks held 0    write locks 0    pid/thread 1704/5224  
          65 dd=314 locks held 0    write locks 0    pid/thread 1704/5668  
          66 dd=313 locks held 0    write locks 0    pid/thread 1704/8164  
          67 dd=312 locks held 0    write locks 0    pid/thread 1704/6272  
          68 dd=310 locks held 0    write locks 0    pid/thread 1704/6268  
          69 dd=309 locks held 0    write locks 0    pid/thread 1704/7148  
          6a dd=308 locks held 0    write locks 0    pid/thread 1704/7184  
          6b dd=307 locks held 0    write locks 0    pid/thread 1704/6596  
          6c dd=306 locks held 0    write locks 0    pid/thread 1704/6044  
          6d dd=302 locks held 0    write locks 0    pid/thread 1704/6476  
          6e dd=301 locks held 0    write locks 0    pid/thread 1704/6916  
          6f dd=300 locks held 0    write locks 0    pid/thread 1704/4784  
          70 dd=299 locks held 0    write locks 0    pid/thread 1704/7444  
          71 dd=298 locks held 0    write locks 0    pid/thread 1704/5544  
          72 dd=297 locks held 0    write locks 0    pid/thread 1704/7140  
          73 dd=296 locks held 0    write locks 0    pid/thread 1704/5724  
          74 dd=295 locks held 0    write locks 0    pid/thread 1704/956  
          75 dd=294 locks held 0    write locks 0    pid/thread 1704/7916  
          76 dd=293 locks held 0    write locks 0    pid/thread 1704/6116  
          77 dd=292 locks held 0    write locks 0    pid/thread 1704/7928  
          78 dd=291 locks held 0    write locks 0    pid/thread 1704/6116  
          79 dd=290 locks held 0    write locks 0    pid/thread 1704/7160  
          7a dd=289 locks held 0    write locks 0    pid/thread 1704/6656  
          7b dd=288 locks held 0    write locks 0    pid/thread 1704/7212  
          7c dd=287 locks held 0    write locks 0    pid/thread 1704/6116  
          7d dd=286 locks held 0    write locks 0    pid/thread 1704/5820  
          7e dd=285 locks held 0    write locks 0    pid/thread 1704/5088  
          7f dd=284 locks held 0    write locks 0    pid/thread 1704/1108  
          80 dd=283 locks held 0    write locks 0    pid/thread 1704/7372  
          81 dd=282 locks held 0    write locks 0    pid/thread 1704/4692  
          82 dd=281 locks held 0    write locks 0    pid/thread 1704/7460  
          83 dd=280 locks held 0    write locks 0    pid/thread 1704/5528  
          84 dd=279 locks held 0    write locks 0    pid/thread 1704/7160  
          85 dd=278 locks held 0    write locks 0    pid/thread 1704/1168  
          86 dd=275 locks held 0    write locks 0    pid/thread 1704/5464  
          87 dd=274 locks held 0    write locks 0    pid/thread 1704/5652  
          88 dd=273 locks held 0    write locks 0    pid/thread 1704/6628  
          89 dd=272 locks held 0    write locks 0    pid/thread 1704/5164  
          8a dd=271 locks held 0    write locks 0    pid/thread 1704/7372  
          8b dd=270 locks held 0    write locks 0    pid/thread 1704/5236  
          8c dd=269 locks held 0    write locks 0    pid/thread 1704/6532  
          8d dd=268 locks held 0    write locks 0    pid/thread 1704/4656  
          8e dd=267 locks held 0    write locks 0    pid/thread 1704/7708  
          8f dd=266 locks held 0    write locks 0    pid/thread 1704/5112  
          90 dd=265 locks held 0    write locks 0    pid/thread 1704/7484  
          91 dd=259 locks held 0    write locks 0    pid/thread 1704/7352  
          92 dd=258 locks held 0    write locks 0    pid/thread 1704/7408  
          93 dd=257 locks held 0    write locks 0    pid/thread 1704/5472  
          94 dd=256 locks held 0    write locks 0    pid/thread 1704/7372  
          95 dd=255 locks held 0    write locks 0    pid/thread 1704/7552  
          96 dd=252 locks held 0    write locks 0    pid/thread 1704/5112  
          97 dd=249 locks held 0    write locks 0    pid/thread 1704/5472  
          98 dd=248 locks held 0    write locks 0    pid/thread 1704/5472  
          99 dd=247 locks held 0    write locks 0    pid/thread 1704/6460  
          9a dd=246 locks held 0    write locks 0    pid/thread 1704/5432  
          9b dd=245 locks held 0    write locks 0    pid/thread 1704/7220  
          9c dd=244 locks held 0    write locks 0    pid/thread 1704/7768  
          9d dd=243 locks held 0    write locks 0    pid/thread 1704/5088  
          9e dd=242 locks held 0    write locks 0    pid/thread 1704/5544  
          9f dd=241 locks held 0    write locks 0    pid/thread 1704/5272  
          a0 dd=240 locks held 0    write locks 0    pid/thread 1704/7284  
          a1 dd=239 locks held 0    write locks 0    pid/thread 1704/3884  
          a2 dd=238 locks held 0    write locks 0    pid/thread 1704/4552  
          a3 dd=237 locks held 0    write locks 0    pid/thread 1704/6460  
          a4 dd=236 locks held 0    write locks 0    pid/thread 1704/6324  
          a5 dd=235 locks held 0    write locks 0    pid/thread 1704/6272  
          a6 dd=234 locks held 0    write locks 0    pid/thread 1704/7276  
          a7 dd=233 locks held 0    write locks 0    pid/thread 1704/4552  
          a8 dd=232 locks held 0    write locks 0    pid/thread 1704/6876  
          a9 dd=231 locks held 0    write locks 0    pid/thread 1704/5296  
          aa dd=230 locks held 0    write locks 0    pid/thread 1704/5668  
          ab dd=229 locks held 0    write locks 0    pid/thread 1704/8164  
          ac dd=228 locks held 0    write locks 0    pid/thread 1704/5444  
          ad dd=227 locks held 0    write locks 0    pid/thread 1704/7916  
          ae dd=226 locks held 0    write locks 0    pid/thread 1704/1944  
          af dd=225 locks held 0    write locks 0    pid/thread 1704/7040  
          b0 dd=224 locks held 0    write locks 0    pid/thread 1704/5548  
          b1 dd=223 locks held 0    write locks 0    pid/thread 1704/3880  
          b2 dd=222 locks held 0    write locks 0    pid/thread 1704/5428  
          b3 dd=221 locks held 0    write locks 0    pid/thread 1704/5224  
          b4 dd=220 locks held 0    write locks 0    pid/thread 1704/6420  
          b5 dd=219 locks held 0    write locks 0    pid/thread 1704/7104  
          b6 dd=218 locks held 0    write locks 0    pid/thread 1704/5944  
          b7 dd=217 locks held 0    write locks 0    pid/thread 1704/7728  
          b8 dd=216 locks held 0    write locks 0    pid/thread 1704/7632  
          b9 dd=215 locks held 0    write locks 0    pid/thread 1704/7436  
          ba dd=214 locks held 0    write locks 0    pid/thread 1704/5472  
          bb dd=213 locks held 0    write locks 0    pid/thread 1704/7148  
          bc dd=212 locks held 0    write locks 0    pid/thread 1704/4200  
          bd dd=211 locks held 0    write locks 0    pid/thread 1704/4716  
          be dd=210 locks held 0    write locks 0    pid/thread 1704/5652  
          bf dd=209 locks held 0    write locks 0    pid/thread 1704/7012  
          c0 dd=208 locks held 0    write locks 0    pid/thread 1704/6532  
          c1 dd=207 locks held 0    write locks 0    pid/thread 1704/4692  
          c2 dd=206 locks held 0    write locks 0    pid/thread 1704/7936  
          c3 dd=205 locks held 0    write locks 0    pid/thread 1704/656  
          c4 dd=204 locks held 0    write locks 0    pid/thread 1704/6460  
          c5 dd=203 locks held 0    write locks 0    pid/thread 1704/7348  
          c6 dd=202 locks held 0    write locks 0    pid/thread 1704/4768  
          c7 dd=201 locks held 0    write locks 0    pid/thread 1704/6080  
          c8 dd=200 locks held 0    write locks 0    pid/thread 1704/6380  
          c9 dd=199 locks held 0    write locks 0    pid/thread 1704/376  
          ca dd=198 locks held 0    write locks 0    pid/thread 1704/5468  
          cb dd=197 locks held 0    write locks 0    pid/thread 1704/7128  
          cc dd=196 locks held 0    write locks 0    pid/thread 1704/6408  
          cd dd=195 locks held 0    write locks 0    pid/thread 1704/7396  
          ce dd=194 locks held 0    write locks 0    pid/thread 1704/7972  
          cf dd=193 locks held 0    write locks 0    pid/thread 1704/5088  
          d0 dd=192 locks held 0    write locks 0    pid/thread 1704/7848  
          d1 dd=191 locks held 0    write locks 0    pid/thread 1704/5912  
          d2 dd=190 locks held 0    write locks 0    pid/thread 1704/5820  
          d3 dd=189 locks held 0    write locks 0    pid/thread 1704/4552  
          d4 dd=188 locks held 0    write locks 0    pid/thread 1704/7140  
          d5 dd=187 locks held 0    write locks 0    pid/thread 1704/5116  
          d6 dd=186 locks held 0    write locks 0    pid/thread 1704/4784  
          d7 dd=185 locks held 0    write locks 0    pid/thread 1704/5472  
          d8 dd=184 locks held 0    write locks 0    pid/thread 1704/5756  
          d9 dd=183 locks held 0    write locks 0    pid/thread 1704/6584  
          da dd=182 locks held 0    write locks 0    pid/thread 1704/5236  
          db dd=181 locks held 0    write locks 0    pid/thread 1704/7444  
          dc dd=180 locks held 0    write locks 0    pid/thread 1704/7928  
          dd dd=179 locks held 0    write locks 0    pid/thread 1704/5724  
          de dd=178 locks held 0    write locks 0    pid/thread 1704/6476  
          df dd=177 locks held 0    write locks 0    pid/thread 1704/7768  
          e0 dd=176 locks held 0    write locks 0    pid/thread 1704/5444  
          e1 dd=175 locks held 0    write locks 0    pid/thread 1704/1944  
          e2 dd=174 locks held 0    write locks 0    pid/thread 1704/1448  
          e3 dd=173 locks held 0    write locks 0    pid/thread 1704/5088  
          e4 dd=172 locks held 0    write locks 0    pid/thread 1704/6420  
          e5 dd=171 locks held 0    write locks 0    pid/thread 1704/6532  
          e6 dd=170 locks held 0    write locks 0    pid/thread 1704/5088  
          e7 dd=169 locks held 0    write locks 0    pid/thread 1704/5272  
          e8 dd=168 locks held 0    write locks 0    pid/thread 1704/6896  
          e9 dd=167 locks held 0    write locks 0    pid/thread 1704/5652  
          ea dd=166 locks held 0    write locks 0    pid/thread 1704/7040  
          eb dd=165 locks held 0    write locks 0    pid/thread 1704/6272  
          ec dd=164 locks held 0    write locks 0    pid/thread 1704/3880  
          ed dd=163 locks held 0    write locks 0    pid/thread 1704/5236  
          ee dd=162 locks held 0    write locks 0    pid/thread 1704/6532  
          ef dd=161 locks held 0    write locks 0    pid/thread 1704/656  
          f0 dd=160 locks held 0    write locks 0    pid/thread 1704/6836  
          f1 dd=159 locks held 0    write locks 0    pid/thread 1704/7916  
          f2 dd=158 locks held 0    write locks 0    pid/thread 1704/6324  
          f3 dd=157 locks held 0    write locks 0    pid/thread 1704/3884  
          f4 dd=156 locks held 0    write locks 0    pid/thread 1704/7220  
          f5 dd=155 locks held 0    write locks 0    pid/thread 1704/6188  
          f6 dd=154 locks held 0    write locks 0    pid/thread 1704/7960  
          f7 dd=153 locks held 0    write locks 0    pid/thread 1704/7396  
          f8 dd=152 locks held 0    write locks 0    pid/thread 1704/7632  
          f9 dd=151 locks held 0    write locks 0    pid/thread 1704/5088  
          fa dd=150 locks held 0    write locks 0    pid/thread 1704/7728  
          fb dd=149 locks held 0    write locks 0    pid/thread 1704/5428  
          fc dd=148 locks held 0    write locks 0    pid/thread 1704/5468  
          fd dd=147 locks held 0    write locks 0    pid/thread 1704/7276  
          fe dd=146 locks held 0    write locks 0    pid/thread 1704/4200  
          ff dd=145 locks held 0    write locks 0    pid/thread 1704/6352  
         100 dd=144 locks held 0    write locks 0    pid/thread 1704/4896  
         101 dd=143 locks held 0    write locks 0    pid/thread 1704/5548  
         102 dd=142 locks held 0    write locks 0    pid/thread 1704/7128  
         103 dd=141 locks held 0    write locks 0    pid/thread 1704/5088  
         104 dd=140 locks held 0    write locks 0    pid/thread 1704/5088  
         105 dd=139 locks held 0    write locks 0    pid/thread 1704/6460  
         106 dd=138 locks held 0    write locks 0    pid/thread 1704/376  
         107 dd=137 locks held 0    write locks 0    pid/thread 1704/6956  
         108 dd=136 locks held 0    write locks 0    pid/thread 1704/7848  
         109 dd=135 locks held 0    write locks 0    pid/thread 1704/5432  
         10a dd=134 locks held 0    write locks 0    pid/thread 1704/5912  
         10b dd=133 locks held 0    write locks 0    pid/thread 1704/6876  
         10c dd=132 locks held 0    write locks 0    pid/thread 1704/5896  
         10d dd=131 locks held 0    write locks 0    pid/thread 1704/5156  
         10e dd=130 locks held 0    write locks 0    pid/thread 1704/6260  
         10f dd=129 locks held 0    write locks 0    pid/thread 1704/7232  
         110 dd=128 locks held 0    write locks 0    pid/thread 1704/4784  
         111 dd=127 locks held 0    write locks 0    pid/thread 1704/6896  
         112 dd=126 locks held 0    write locks 0    pid/thread 1704/7116  
         113 dd=125 locks held 0    write locks 0    pid/thread 1704/6476  
         114 dd=124 locks held 0    write locks 0    pid/thread 1704/5756  
         115 dd=123 locks held 0    write locks 0    pid/thread 1704/4768  
         116 dd=122 locks held 0    write locks 0    pid/thread 1704/5944  
         117 dd=121 locks held 0    write locks 0    pid/thread 1704/7444  
         118 dd=120 locks held 0    write locks 0    pid/thread 1704/7436  
         119 dd=119 locks held 0    write locks 0    pid/thread 1704/5236  
         11a dd=118 locks held 0    write locks 0    pid/thread 1704/5116  
         11b dd=117 locks held 0    write locks 0    pid/thread 1704/7104  
         11c dd=116 locks held 0    write locks 0    pid/thread 1704/8144  
         11d dd=115 locks held 0    write locks 0    pid/thread 1704/5224  
         11e dd=114 locks held 0    write locks 0    pid/thread 1704/8164  
         11f dd=113 locks held 0    write locks 0    pid/thread 1704/5544  
         120 dd=112 locks held 0    write locks 0    pid/thread 1704/5724  
         121 dd=111 locks held 0    write locks 0    pid/thread 1704/7972  
         122 dd=110 locks held 1    write locks 0    pid/thread 1704/1108  
         122 READ          1 HELD    entry.dbxml               page         13
         123 dd=109 locks held 0    write locks 0    pid/thread 1704/5296  
         124 dd=108 locks held 0    write locks 0    pid/thread 1704/7348  
         125 dd=107 locks held 0    write locks 0    pid/thread 1704/6080  
         126 dd=106 locks held 0    write locks 0    pid/thread 1704/7184  
         126 READ          1 WAIT    entry.dbxml               page         13
         127 dd=105 locks held 0    write locks 0    pid/thread 1704/6408  
         128 dd=104 locks held 0    write locks 0    pid/thread 1704/7284  
         129 dd=103 locks held 0    write locks 0    pid/thread 1704/6584  
         12a dd=102 locks held 0    write locks 0    pid/thread 1704/6216  
         12b dd=101 locks held 0    write locks 0    pid/thread 1704/6380  
         12c dd=100 locks held 0    write locks 0    pid/thread 1704/5668  
         12d dd=99 locks held 0    write locks 0    pid/thread 1704/4120  
         12e dd=98 locks held 0    write locks 0    pid/thread 1704/5804  
         12f dd=97 locks held 0    write locks 0    pid/thread 1704/7140  
         130 dd=96 locks held 0    write locks 0    pid/thread 1704/6712  
         131 dd=95 locks held 0    write locks 0    pid/thread 1704/6564  
         132 dd=94 locks held 0    write locks 0    pid/thread 1704/7808  
         133 dd=93 locks held 0    write locks 0    pid/thread 1704/1844  
         134 dd=92 locks held 0    write locks 0    pid/thread 1704/4024  
         135 dd=91 locks held 0    write locks 0    pid/thread 1704/6900  
         136 dd=90 locks held 0    write locks 0    pid/thread 1704/6928  
         137 dd=89 locks held 0    write locks 0    pid/thread 1704/7764  
         138 dd=88 locks held 0    write locks 0    pid/thread 1704/7320  
         139 dd=87 locks held 0    write locks 0    pid/thread 1704/6752  
         13a dd=86 locks held 0    write locks 0    pid/thread 1704/6728  
         13b dd=85 locks held 0    write locks 0    pid/thread 1704/1864  
         13c dd=84 locks held 0    write locks 0    pid/thread 1704/956  
         13d dd=83 locks held 0    write locks 0    pid/thread 1704/1168  
         13d READ          1 WAIT    entry.dbxml               page         11
         13e dd=82 locks held 0    write locks 0    pid/thread 1704/7328  
         13f dd=81 locks held 0    write locks 0    pid/thread 1704/7496  
         140 dd=80 locks held 0    write locks 0    pid/thread 1704/6776  
         141 dd=79 locks held 0    write locks 0    pid/thread 1704/6916  
         142 dd=78 locks held 0    write locks 0    pid/thread 1704/6948  
         143 dd=77 locks held 0    write locks 0    pid/thread 1704/4236  
         144 dd=76 locks held 0    write locks 0    pid/thread 1704/7056  
         145 dd=75 locks held 0    write locks 0    pid/thread 1704/4176  
         146 dd=74 locks held 0    write locks 0    pid/thread 1704/6676  
         147 dd=73 locks held 0    write locks 0    pid/thread 1704/7764  
         148 dd=72 locks held 0    write locks 0    pid/thread 1704/1108  
         148 READ          1 WAIT    entry.dbxml               page         11
         149 dd=71 locks held 0    write locks 0    pid/thread 1704/5100  
         14a dd=70 locks held 0    write locks 0    pid/thread 1704/4724  
         14b dd=69 locks held 0    write locks 0    pid/thread 1704/7236  
         14c dd=67 locks held 0    write locks 0    pid/thread 1704/3940  
         14d dd=66 locks held 0    write locks 0    pid/thread 1704/5688  
         14e dd=65 locks held 0    write locks 0    pid/thread 1704/6616  
         14f dd=64 locks held 0    write locks 0    pid/thread 1704/7764  
         150 dd=63 locks held 0    write locks 0    pid/thread 1704/7456  
         151 dd=62 locks held 0    write locks 0    pid/thread 1704/7428  
         152 dd=61 locks held 0    write locks 0    pid/thread 1704/6064  
         153 dd=60 locks held 0    write locks 0    pid/thread 1704/5464  
         154 dd=59 locks held 0    write locks 0    pid/thread 1704/7428  
         155 dd=58 locks held 0    write locks 0    pid/thread 1704/7148  
         155 READ          1 WAIT    entry.dbxml               page         13
         156 dd=57 locks held 0    write locks 0    pid/thread 1704/7148  
         157 dd=56 locks held 0    write locks 0    pid/thread 1704/5620  
         158 dd=55 locks held 0    write locks 0    pid/thread 1704/3716  
         159 dd=54 locks held 0    write locks 0    pid/thread 1704/7640  
         15a dd=53 locks held 0    write locks 0    pid/thread 1704/4692  
         15a READ          1 WAIT    entry.dbxml               page         13
         15b dd=52 locks held 0    write locks 0    pid/thread 1704/4692  
         15c dd=51 locks held 0    write locks 0    pid/thread 1704/5164  
         15c READ          1 WAIT    entry.dbxml               page         13
         15d dd=50 locks held 0    write locks 0    pid/thread 1704/5164  
         15e dd=49 locks held 0    write locks 0    pid/thread 1704/5820  
         15e READ          1 WAIT    entry.dbxml               page         13
         15f dd=48 locks held 0    write locks 0    pid/thread 1704/5820  
         160 dd=47 locks held 0    write locks 0    pid/thread 1704/6496  
         161 dd=46 locks held 0    write locks 0    pid/thread 1704/7624  
         162 dd=45 locks held 0    write locks 0    pid/thread 1704/4772  
         163 dd=44 locks held 0    write locks 0    pid/thread 1704/7552  
         163 READ          1 WAIT    entry.dbxml               page         13
         164 dd=43 locks held 0    write locks 0    pid/thread 1704/7552  
         165 dd=42 locks held 0    write locks 0    pid/thread 1704/7372  
         166 dd=41 locks held 0    write locks 0    pid/thread 1704/5536  
         167 dd=40 locks held 0    write locks 0    pid/thread 1704/6796  
         168 dd=39 locks held 0    write locks 0    pid/thread 1704/5268  
         169 dd=38 locks held 0    write locks 0    pid/thread 1704/8024  
         16a dd=37 locks held 0    write locks 0    pid/thread 1704/6460  
         16a READ          1 WAIT    entry.dbxml               page         13
         16b dd=36 locks held 0    write locks 0    pid/thread 1704/6460  
         16c dd=35 locks held 0    write locks 0    pid/thread 1704/4368  
         16d dd=34 locks held 0    write locks 0    pid/thread 1704/4368  
         16e dd=33 locks held 0    write locks 0    pid/thread 1704/6016  
         16f dd=32 locks held 0    write locks 0    pid/thread 1704/6908  
         170 dd=31 locks held 0    write locks 0    pid/thread 1704/7640  
         171 dd=30 locks held 0    write locks 0    pid/thread 1704/6016  
    80000018 dd=445 locks held 0    write locks 0    pid/thread 1704/1944  
         172 dd=29 locks held 0    write locks 0    pid/thread 1704/6532  
    80000019 dd=444 locks held 0    write locks 0    pid/thread 1704/6728  
         173 dd=28 locks held 0    write locks 0    pid/thread 1704/6588  
    8000001a dd=443 locks held 0    write locks 0    pid/thread 1704/7916  
         174 dd=27 locks held 0    write locks 0    pid/thread 1704/7420  
    8000001b dd=442 locks held 0    write locks 0    pid/thread 1704/4200  
         175 dd=26 locks held 0    write locks 0    pid/thread 1704/6336  
    8000001c dd=441 locks held 0    write locks 0    pid/thread 1704/5548  
         176 dd=25 locks held 0    write locks 0    pid/thread 1704/5088  
    8000001d dd=440 locks held 0    write locks 0    pid/thread 1704/1844  
         177 dd=24 locks held 0    write locks 0    pid/thread 1704/7212  
    8000001e dd=439 locks held 0    write locks 0    pid/thread 1704/6876  
         178 dd=23 locks held 0    write locks 0    pid/thread 1704/7212  
    8000001f dd=438 locks held 0    write locks 0    pid/thread 1704/5224  
         179 dd=22 locks held 0    write locks 0    pid/thread 1704/5088  
    80000020 dd=437 locks held 0    write locks 0    pid/thread 1704/5668  
         17a dd=20 locks held 0    write locks 0    pid/thread 1704/7660  
    80000021 dd=436 locks held 0    write locks 0    pid/thread 1704/7348  
         17b dd=19 locks held 0    write locks 0    pid/thread 1704/7160  
    80000022 dd=435 locks held 0    write locks 0    pid/thread 1704/8164  
         17c dd=18 locks held 0    write locks 0    pid/thread 1704/6656  
    80000023 dd=434 locks held 0    write locks 0    pid/thread 1704/6420  
         17d dd=17 locks held 0    write locks 0    pid/thread 1704/7740  
    80000024 dd=433 locks held 0    write locks 0    pid/thread 1704/4120  
         17e dd=16 locks held 0    write locks 0    pid/thread 1704/5652  
         17e READ          1 WAIT    entry.dbxml               page         13
    80000025 dd=432 locks held 0    write locks 0    pid/thread 1704/3884  
         17f dd=15 locks held 0    write locks 0    pid/thread 1704/5652  
    80000026 dd=431 locks held 0    write locks 0    pid/thread 1704/6272  
         180 dd=13 locks held 0    write locks 0    pid/thread 1704/7928  
         180 READ          1 WAIT    entry.dbxml               page         13
    80000027 dd=430 locks held 0    write locks 0    pid/thread 1704/5688  
         181 dd=12 locks held 0    write locks 0    pid/thread 1704/7928  
    80000028 dd=429 locks held 0    write locks 0    pid/thread 1704/6380  
         182 dd=11 locks held 0    write locks 0    pid/thread 1704/6596  
         182 READ          1 WAIT    entry.dbxml               page         13
    80000029 dd=428 locks held 0    write locks 0    pid/thread 1704/6752  
         183 dd=10 locks held 0    write locks 0    pid/thread 1704/6596  
    8000002a dd=427 locks held 0    write locks 0    pid/thread 1704/6584  
         184 dd= 8 locks held 0    write locks 0    pid/thread 1704/5088  
         184 READ          1 WAIT    entry.dbxml               page         13
    8000002b dd=426 locks held 0    write locks 0    pid/thread 1704/7328  
         185 dd= 7 locks held 0    write locks 0    pid/thread 1704/5088  
    8000002c dd=425 locks held 0    write locks 0    pid/thread 1704/7232  
         186 dd= 6 locks held 0    write locks 0    pid/thread 1704/5236  
         186 READ          1 WAIT    entry.dbxml               page         13
    8000002d dd=424 locks held 0    write locks 0    pid/thread 1704/5756  
         187 dd= 5 locks held 0    write locks 0    pid/thread 1704/5236  
    8000002e dd=423 locks held 0    write locks 0    pid/thread 1704/5896  
         188 dd= 4 locks held 0    write locks 0    pid/thread 1704/6532  
         188 READ          1 WAIT    entry.dbxml               page         13
    8000002f dd=422 locks held 0    write locks 0    pid/thread 1704/7276  
         189 dd= 3 locks held 0    write locks 0    pid/thread 1704/6532  
    80000030 dd=421 locks held 0    write locks 0    pid/thread 1704/4176  
         18a dd= 1 locks held 0    write locks 0    pid/thread 1704/5472  
         18a READ          1 WAIT    entry.dbxml               page         13
    80000031 dd=420 locks held 0    write locks 0    pid/thread 1704/1448  
         18b dd= 0 locks held 0    write locks 0    pid/thread 1704/5472  
    80000032 dd=419 locks held 0    write locks 0    pid/thread 1704/4024  
    80000033 dd=418 locks held 0    write locks 0    pid/thread 1704/376  
    80000034 dd=417 locks held 0    write locks 0    pid/thread 1704/7284  
    80000035 dd=416 locks held 0    write locks 0    pid/thread 1704/5432  
    80000036 dd=415 locks held 0    write locks 0    pid/thread 1704/6956  
    80000037 dd=414 locks held 0    write locks 0    pid/thread 1704/6080  
    80000038 dd=413 locks held 0    write locks 0    pid/thread 1704/6776  
    80000039 dd=412 locks held 0    write locks 0    pid/thread 1704/5912  
    8000003a dd=411 locks held 0    write locks 0    pid/thread 1704/5544  
    8000003b dd=410 locks held 0    write locks 0    pid/thread 1704/656  
    8000003c dd=409 locks held 0    write locks 0    pid/thread 1704/7220  
    8000003d dd=408 locks held 0    write locks 0    pid/thread 1704/4768  
    8000003f dd=407 locks held 0    write locks 0    pid/thread 1704/6092  
    80000040 dd=406 locks held 0    write locks 0    pid/thread 1704/5804  
    80000041 dd=405 locks held 0    write locks 0    pid/thread 1704/5272  
    80000042 dd=404 locks held 0    write locks 0    pid/thread 1704/6712  
    80000043 dd=403 locks held 0    write locks 0    pid/thread 1704/7056  
    80000044 dd=402 locks held 0    write locks 0    pid/thread 1704/7104  
    80000045 dd=401 locks held 0    write locks 0    pid/thread 1704/7040  
    80000046 dd=400 locks held 0    write locks 0    pid/thread 1704/5116  
    80000047 dd=399 locks held 0    write locks 0    pid/thread 1704/7728  
    80000048 dd=398 locks held 0    write locks 0    pid/thread 1704/6900  
    80000049 dd=397 locks held 0    write locks 0    pid/thread 1704/7936  
    8000004a dd=396 locks held 0    write locks 0    pid/thread 1704/7236  
    8000004b dd=395 locks held 0    write locks 0    pid/thread 1704/8144  
    8000004c dd=394 locks held 0    write locks 0    pid/thread 1704/7140  
    8000004d dd=393 locks held 0    write locks 0    pid/thread 1704/6324  
    8000004e dd=392 locks held 0    write locks 0    pid/thread 1704/4552  
    8000004f dd=391 locks held 0    write locks 0    pid/thread 1704/5944  
    80000050 dd=390 locks held 0    write locks 0    pid/thread 1704/956  
    80000051 dd=389 locks held 0    write locks 0    pid/thread 1704/5156  
    80000052 dd=388 locks held 0    write locks 0    pid/thread 1704/5444  
    80000053 dd=387 locks held 0    write locks 0    pid/thread 1704/4784  
    80000054 dd=386 locks held 0    write locks 0    pid/thread 1704/4724  
    80000055 dd=385 locks held 0    write locks 0    pid/thread 1704/5100  
    80000056 dd=384 locks held 0    write locks 0    pid/thread 1704/3940  
    80000057 dd=383 locks held 0    write locks 0    pid/thread 1704/6676  
    80000058 dd=382 locks held 0    write locks 0    pid/thread 1704/7436  
    80000059 dd=381 locks held 0    write locks 0    pid/thread 1704/7768  
    8000005a dd=380 locks held 0    write locks 0    pid/thread 1704/7496  
    8000005b dd=379 locks held 0    write locks 0    pid/thread 1704/6476  
    8000005c dd=378 locks held 0    write locks 0    pid/thread 1704/6352  
    8000005d dd=377 locks held 0    write locks 0    pid/thread 1704/6928  
    8000005e dd=376 locks held 0    write locks 0    pid/thread 1704/6916  
    8000005f dd=375 locks held 0    write locks 0    pid/thread 1704/7128  
    80000060 dd=374 locks held 0    write locks 0    pid/thread 1704/5468  
    80000061 dd=373 locks held 0    write locks 0    pid/thread 1704/6408  
    80000062 dd=372 locks held 0    write locks 0    pid/thread 1704/6260  
    80000063 dd=371 locks held 0    write locks 0    pid/thread 1704/6948  
    80000064 dd=370 locks held 0    write locks 0    pid/thread 1704/1864  
    80000065 dd=369 locks held 0    write locks 0    pid/thread 1704/5296  
    80000066 dd=368 locks held 0    write locks 0    pid/thread 1704/7848  
    80000067 dd=367 locks held 0    write locks 0    pid/thread 1704/7444  
    80000068 dd=366 locks held 0    write locks 0    pid/thread 1704/7320  
    80000069 dd=365 locks held 0    write locks 0    pid/thread 1704/7972  
    8000006a dd=364 locks held 0    write locks 0    pid/thread 1704/5724  
    8000006b dd=363 locks held 0    write locks 0    pid/thread 1704/7456  
    8000006c dd=362 locks held 0    write locks 0    pid/thread 1704/6896  
    8000006d dd=361 locks held 0    write locks 0    pid/thread 1704/6616  
    8000006e dd=360 locks held 0    write locks 0    pid/thread 1704/7632  
    8000006f dd=359 locks held 0    write locks 0    pid/thread 1704/6216  
    80000070 dd=358 locks held 0    write locks 0    pid/thread 1704/7116  
    80000071 dd=357 locks held 0    write locks 0    pid/thread 1704/7396  
    80000072 dd=356 locks held 0    write locks 0    pid/thread 1704/4896  
    80000073 dd=355 locks held 0    write locks 0    pid/thread 1704/5428  
    80000074 dd=354 locks held 0    write locks 0    pid/thread 1704/6564  
    80000075 dd=353 locks held 0    write locks 0    pid/thread 1704/7960  
    80000076 dd=352 locks held 0    write locks 0    pid/thread 1704/6188  
    80000077 dd=351 locks held 0    write locks 0    pid/thread 1704/3880  
    80000078 dd=350 locks held 0    write locks 0    pid/thread 1704/6836  
    80000079 dd=349 locks held 0    write locks 0    pid/thread 1704/4236  
    8000007a dd=348 locks held 0    write locks 0    pid/thread 1704/7808  
    8000007b dd=347 locks held 0    write locks 0    pid/thread 1704/1168  
    8000007c dd=346 locks held 0    write locks 0    pid/thread 1704/5236  
    8000007d dd=345 locks held 0    write locks 0    pid/thread 1704/5088  
    8000007e dd=344 locks held 0    write locks 0    pid/thread 1704/6460  
    8000007f dd=343 locks held 0    write locks 0    pid/thread 1704/4692  
    80000081 dd=341 locks held 0    write locks 0    pid/thread 1704/5472  
    80000082 dd=340 locks held 0    write locks 0    pid/thread 1704/1108  
    80000083 dd=339 locks held 0    write locks 0    pid/thread 1704/5164  
    80000084 dd=338 locks held 0    write locks 0    pid/thread 1704/7184  
    80000085 dd=337 locks held 0    write locks 0    pid/thread 1704/7552  
    80000086 dd=336 locks held 0    write locks 0    pid/thread 1704/7928  
    80000092 dd=329 locks held 0    write locks 0    pid/thread 1704/5652  
    80000094 dd=328 locks held 0    write locks 0    pid/thread 1704/6532  
    80000096 dd=326 locks held 0    write locks 0    pid/thread 1704/6596  
    80000098 dd=325 locks held 0    write locks 0    pid/thread 1704/7148  
    8000009b dd=323 locks held 0    write locks 0    pid/thread 1704/5820  
    800000a2 dd=434 locks held 0    write locks 0    pid/thread 1704/6420  
    800000a2 WRITE         1 WAIT    entry.dbxml               page         11
    800000a3 dd=433 locks held 0    write locks 0    pid/thread 1704/4120  
    800000a3 WRITE         1 WAIT    entry.dbxml               page         11
    800000a4 dd=439 locks held 0    write locks 0    pid/thread 1704/6876  
    800000a4 WRITE         1 WAIT    entry.dbxml               page         11
    800000a5 dd=429 locks held 0    write locks 0    pid/thread 1704/6380  
    800000a5 WRITE         1 WAIT    entry.dbxml               page         11
    800000a6 dd=428 locks held 0    write locks 0    pid/thread 1704/6752  
    800000a6 WRITE         1 WAIT    entry.dbxml               page         11
    800000a7 dd=436 locks held 0    write locks 0    pid/thread 1704/7348  
    800000a7 WRITE         1 WAIT    entry.dbxml               page         11
    800000a8 dd=435 locks held 0    write locks 0    pid/thread 1704/8164  
    800000a8 WRITE         1 WAIT    entry.dbxml               page         11
    800000a9 dd=440 locks held 0    write locks 0    pid/thread 1704/1844  
    800000a9 WRITE         1 WAIT    entry.dbxml               page         11
    800000aa dd=438 locks held 0    write locks 0    pid/thread 1704/5224  
    800000aa WRITE         1 WAIT    entry.dbxml               page         11
    800000ab dd=437 locks held 0    write locks 0    pid/thread 1704/5668  
    800000ab WRITE         1 WAIT    entry.dbxml               page         11
    800000ac dd=431 locks held 0    write locks 0    pid/thread 1704/6272  
    800000ac WRITE         1 WAIT    entry.dbxml               page         11
    800000b9 dd=412 locks held 0    write locks 0    pid/thread 1704/5912  
    800000b9 WRITE         1 WAIT    entry.dbxml               page         11
    800000bd dd=416 locks held 0    write locks 0    pid/thread 1704/5432  
    800000bd WRITE         1 WAIT    entry.dbxml               page         11
    800000be dd=409 locks held 0    write locks 0    pid/thread 1704/7220  
    800000be WRITE         1 WAIT    entry.dbxml               page         11
    800000bf dd=422 locks held 0    write locks 0    pid/thread 1704/7276  
    800000bf WRITE         1 WAIT    entry.dbxml               page         11
    800000c0 dd=424 locks held 0    write locks 0    pid/thread 1704/5756  
    800000c0 WRITE         1 WAIT    entry.dbxml               page         11
    800000c1 dd=410 locks held 0    write locks 0    pid/thread 1704/656  
    800000c1 WRITE         1 WAIT    entry.dbxml               page         11
    800000c5 dd=420 locks held 0    write locks 0    pid/thread 1704/1448  
    800000c5 WRITE         1 WAIT    entry.dbxml               page         11
    800000c6 dd=417 locks held 0    write locks 0    pid/thread 1704/7284  
    800000c6 WRITE         1 WAIT    entry.dbxml               page         11
    800000c7 dd=415 locks held 0    write locks 0    pid/thread 1704/6956  
    800000c7 WRITE         1 WAIT    entry.dbxml               page         11
    800000c8 dd=423 locks held 0    write locks 0    pid/thread 1704/5896  
    800000c8 WRITE         1 WAIT    entry.dbxml               page         11
    800000c9 dd=401 locks held 0    write locks 0    pid/thread 1704/7040  
    800000c9 WRITE         1 WAIT    entry.dbxml               page         11
    800000ca dd=418 locks held 0    write locks 0    pid/thread 1704/376  
    800000ca WRITE         1 WAIT    entry.dbxml               page         11
    800000cb dd=432 locks held 0    write locks 0    pid/thread 1704/3884  
    800000cb WRITE         1 WAIT    entry.dbxml               page         11
    800000cd dd=393 locks held 0    write locks 0    pid/thread 1704/6324  
    800000cd WRITE         1 WAIT    entry.dbxml               page         11
    800000d0 dd=427 locks held 0    write locks 0    pid/thread 1704/6584  
    800000d0 WRITE         1 WAIT    entry.dbxml               page         11
    800000d2 dd=408 locks held 0    write locks 0    pid/thread 1704/4768  
    800000d2 WRITE         1 WAIT    entry.dbxml               page         11
    800000d3 dd=406 locks held 0    write locks 0    pid/thread 1704/5804  
    800000d3 WRITE         1 WAIT    entry.dbxml               page         11
    800000d4 dd=405 locks held 0    write locks 0    pid/thread 1704/5272  
    800000d4 WRITE         1 WAIT    entry.dbxml               page         11
    800000d5 dd=373 locks held 0    write locks 0    pid/thread 1704/6408  
    800000d5 WRITE         1 WAIT    entry.dbxml               page         11
    800000d6 dd=369 locks held 0    write locks 0    pid/thread 1704/5296  
    800000d6 WRITE         1 WAIT    entry.dbxml               page         11
    800000d7 dd=389 locks held 0    write locks 0    pid/thread 1704/5156  
    800000d7 WRITE         1 WAIT    entry.dbxml               page         11
    800000d8 dd=370 locks held 0    write locks 0    pid/thread 1704/1864  
    800000d8 READ          1 WAIT    entry.dbxml               page         11
    800000d9 dd=411 locks held 0    write locks 0    pid/thread 1704/5544  
    800000d9 WRITE         1 WAIT    entry.dbxml               page         11
    800000da dd=365 locks held 0    write locks 0    pid/thread 1704/7972  
    800000da WRITE         1 WAIT    entry.dbxml               page         11
    800000de dd=381 locks held 0    write locks 0    pid/thread 1704/7768  
    800000de WRITE         1 WAIT    entry.dbxml               page         11
    800000e0 dd=395 locks held 0    write locks 0    pid/thread 1704/8144  
    800000e0 WRITE         1 WAIT    entry.dbxml               page         11
    800000e1 dd=384 locks held 0    write locks 0    pid/thread 1704/3940  
    800000e1 READ          1 WAIT    entry.dbxml               page         11
    800000e2 dd=385 locks held 0    write locks 0    pid/thread 1704/5100  
    800000e2 READ          1 WAIT    entry.dbxml               page         11
    800000e3 dd=396 locks held 0    write locks 0    pid/thread 1704/7236  
    800000e3 READ          1 WAIT    entry.dbxml               page         11
    800000e4 dd=383 locks held 0    write locks 0    pid/thread 1704/6676  
    800000e4 READ          1 WAIT    entry.dbxml               page         11
    800000e5 dd=407 locks held 1    write locks 1    pid/thread 1704/6092  
    800000e5 WRITE         1 WAIT    entry.dbxml               page         13
    800000e5 WRITE         1 HELD    entry.dbxml               page         11
    800000e6 dd=392 locks held 0    write locks 0    pid/thread 1704/4552  
    800000e6 WRITE         1 WAIT    entry.dbxml               page         11
    800000e7 dd=386 locks held 0    write locks 0    pid/thread 1704/4724  
    800000e7 READ          1 WAIT    entry.dbxml               page         11
    800000e8 dd=402 locks held 0    write locks 0    pid/thread 1704/7104  
    800000e8 WRITE         1 WAIT    entry.dbxml               page         11
    800000e9 dd=376 locks held 0    write locks 0    pid/thread 1704/6916  
    800000e9 READ          1 WAIT    entry.dbxml               page         11
    800000ea dd=379 locks held 0    write locks 0    pid/thread 1704/6476  
    800000ea WRITE         1 WAIT    entry.dbxml               page         11
    800000eb dd=387 locks held 0    write locks 0    pid/thread 1704/4784  
    800000eb WRITE         1 WAIT    entry.dbxml               page         11
    800000ec dd=367 locks held 0    write locks 0    pid/thread 1704/7444  
    800000ec WRITE         1 WAIT    entry.dbxml               page         11
    800000ed dd=394 locks held 0    write locks 0    pid/thread 1704/7140  
    800000ed WRITE         1 WAIT    entry.dbxml               page         11
    800000ef dd=390 locks held 0    write locks 0    pid/thread 1704/956  
    800000ef READ          1 WAIT    entry.dbxml               page         11
    800000f0 dd=364 locks held 0    write locks 0    pid/thread 1704/5724  
    800000f0 WRITE         1 WAIT    entry.dbxml               page         11
    800000f1 dd=443 locks held 0    write locks 0    pid/thread 1704/7916  
    800000f1 WRITE         1 WAIT    entry.dbxml               page         11
    800000f2 dd=377 locks held 0    write locks 0    pid/thread 1704/6928  
    800000f2 WRITE         1 WAIT    entry.dbxml               page         11
    800000f4 dd=398 locks held 0    write locks 0    pid/thread 1704/6900  
    800000f4 WRITE         1 WAIT    entry.dbxml               page         11
    800000f7 dd=374 locks held 0    write locks 0    pid/thread 1704/5468  
    800000f7 WRITE         1 WAIT    entry.dbxml               page         11
    800000f8 dd=348 locks held 0    write locks 0    pid/thread 1704/7808  
    800000f8 WRITE         1 WAIT    entry.dbxml               page         11
    800000f9 dd=378 locks held 0    write locks 0    pid/thread 1704/6352  
    800000f9 WRITE         1 WAIT    entry.dbxml               page         11
    800000fa dd=375 locks held 0    write locks 0    pid/thread 1704/7128  
    800000fa WRITE         1 WAIT    entry.dbxml               page         11
    800000fb dd=388 locks held 0    write locks 0    pid/thread 1704/5444  
    800000fb WRITE         1 WAIT    entry.dbxml               page         11
    800000fc dd=400 locks held 0    write locks 0    pid/thread 1704/5116  
    800000fc WRITE         1 WAIT    entry.dbxml               page         11
    800000fe dd=352 locks held 0    write locks 0    pid/thread 1704/6188  
    800000fe WRITE         1 WAIT    entry.dbxml               page         11
    80000100 dd=349 locks held 0    write locks 0    pid/thread 1704/4236  
    80000100 READ          1 WAIT    entry.dbxml               page         11
    80000101 dd=356 locks held 0    write locks 0    pid/thread 1704/4896  
    80000101 WRITE         1 WAIT    entry.dbxml               page         11
    80000102 dd=444 locks held 0    write locks 0    pid/thread 1704/6728  
    80000102 WRITE         1 WAIT    entry.dbxml               page         11
    80000103 dd=421 locks held 0    write locks 0    pid/thread 1704/4176  
    80000103 READ          1 WAIT    entry.dbxml               page         11
    80000104 dd=355 locks held 0    write locks 0    pid/thread 1704/5428  
    80000104 WRITE         1 WAIT    entry.dbxml               page         11
    80000105 dd=430 locks held 0    write locks 0    pid/thread 1704/5688  
    80000105 READ          1 WAIT    entry.dbxml               page         11
    80000108 dd=413 locks held 0    write locks 0    pid/thread 1704/6776  
    80000108 READ          1 WAIT    entry.dbxml               page         11
    8000010b dd=380 locks held 0    write locks 0    pid/thread 1704/7496  
    8000010b READ          1 WAIT    entry.dbxml               page         11
    8000010c dd=357 locks held 0    write locks 0    pid/thread 1704/7396  
    8000010c WRITE         1 WAIT    entry.dbxml               page         11
    80000119 dd=445 locks held 0    write locks 0    pid/thread 1704/1944  
    80000119 WRITE         1 WAIT    entry.dbxml               page         11
    8000011a dd=360 locks held 0    write locks 0    pid/thread 1704/7632  
    8000011a WRITE         1 WAIT    entry.dbxml               page         11
    8000011b dd=351 locks held 0    write locks 0    pid/thread 1704/3880  
    8000011b WRITE         1 WAIT    entry.dbxml               page         11
    8000011c dd=441 locks held 0    write locks 0    pid/thread 1704/5548  
    8000011c WRITE         1 WAIT    entry.dbxml               page         11
    8000011d dd=362 locks held 0    write locks 0    pid/thread 1704/6896  
    8000011d WRITE         1 WAIT    entry.dbxml               page         11
    8000011e dd=366 locks held 0    write locks 0    pid/thread 1704/7320  
    8000011e WRITE         1 WAIT    entry.dbxml               page         11
    8000011f dd=414 locks held 0    write locks 0    pid/thread 1704/6080  
    8000011f WRITE         1 WAIT    entry.dbxml               page         11
    80000120 dd=350 locks held 0    write locks 0    pid/thread 1704/6836  
    80000120 WRITE         1 WAIT    entry.dbxml               page         11
    80000121 dd=403 locks held 0    write locks 0    pid/thread 1704/7056  
    80000121 READ          1 WAIT    entry.dbxml               page         11
    80000122 dd=391 locks held 0    write locks 0    pid/thread 1704/5944  
    80000122 WRITE         1 WAIT    entry.dbxml               page         11
    80000125 dd=425 locks held 0    write locks 0    pid/thread 1704/7232  
    80000125 WRITE         1 WAIT    entry.dbxml               page         11
    80000126 dd=404 locks held 0    write locks 0    pid/thread 1704/6712  
    80000126 WRITE         1 WAIT    entry.dbxml               page         11
    80000127 dd=397 locks held 0    write locks 0    pid/thread 1704/7936  
    80000127 WRITE         1 WAIT    entry.dbxml               page         11
    80000129 dd=399 locks held 0    write locks 0    pid/thread 1704/7728  
    80000129 WRITE         1 WAIT    entry.dbxml               page         11
    8000012a dd=371 locks held 0    write locks 0    pid/thread 1704/6948  
    8000012a READ          1 WAIT    entry.dbxml               page         11
    8000012b dd=372 locks held 0    write locks 0    pid/thread 1704/6260  
    8000012b WRITE         1 WAIT    entry.dbxml               page         11
    8000012e dd=354 locks held 0    write locks 0    pid/thread 1704/6564  
    8000012e WRITE         1 WAIT    entry.dbxml               page         11
    80000131 dd=353 locks held 0    write locks 0    pid/thread 1704/7960  
    80000131 WRITE         1 WAIT    entry.dbxml               page         11
    80000132 dd=426 locks held 0    write locks 0    pid/thread 1704/7328  
    80000132 READ          1 WAIT    entry.dbxml               page         11
    80000134 dd=442 locks held 0    write locks 0    pid/thread 1704/4200  
    80000134 WRITE         1 WAIT    entry.dbxml               page         11
    80000135 dd=382 locks held 0    write locks 0    pid/thread 1704/7436  
    80000135 WRITE         1 WAIT    entry.dbxml               page         11
    80000136 dd=419 locks held 0    write locks 0    pid/thread 1704/4024  
    80000136 WRITE         1 WAIT    entry.dbxml               page         11
    80000137 dd=359 locks held 0    write locks 0    pid/thread 1704/6216  
    80000137 WRITE         1 WAIT    entry.dbxml               page         11
    8000013a dd=368 locks held 0    write locks 0    pid/thread 1704/7848  
    8000013a WRITE         1 WAIT    entry.dbxml               page         11
    8000013c dd=361 locks held 0    write locks 0    pid/thread 1704/6616  
    8000013c READ          1 WAIT    entry.dbxml               page         11
    8000013d dd=358 locks held 0    write locks 0    pid/thread 1704/7116  
    8000013d WRITE

  • Deadlock issues with one thread?

    I've been trying to nail down a deadlock issue for quite a while now and I'm just at a loss. My application basically has one main thread of execution which does get(), put(), and delete() on a database, and another background thread that goes through the database periodically (e.g. every 5 minutes) and prunes the db. The pruning thread works by using a cursor to iterate over the db and evaluates each record and, if it's expired, deletes it, stopping when it reaches the end of the DB or when it's deleted 100,000 records. I had been getting DeadlockExceptions and figured it was a conflict between the main thread and the pruning thread competing for locks on the same objects, so I modified my code so that when the pruning thread wakes up and starts pruning, it makes the main thread wait(), so theoretically the two threads should never be doing db operations at the same time. However, this doesn't seem to have resolved the issue, as I still get DeadlockExceptions. I've raised the lock timeout to 1000 ms and set it to retry up to 20x to no avail. I've even tried it with the pruning threads completely disabled and I still got it, so it seems the deadlock is occurring even within the single thread. Any ideas? I'm not using transactions, if that matters.
    Also, on this page - http://www.oracle.com/technology/documentation/berkeley-db/db/gsg_txn/JAVA/lockingsubsystem.html#deadlockresolve - there's reference to "EnvironmentConfig.setLockDetectMode()". I'm using JE 3.2.23 and there doesn't appear to be any such function. Is this out-of-date documentation or am I using and old version?

    The first step is to take a look at the information in the DeadlockException -- it will tell you which threads are conflicting. Please post the stack traces and I'll be happy to help interpret them.
    Also please take a look at this FAQ:
    http://www.oracle.com/technology/products/berkeley-db/faq/je_faq.html#23
    If you set je.txn.deadlockStackTrace and je.txn.dumpLocks as described, you'll get more information when the deadlock occurs.
    Also, on this page -
    http://www.oracle.com/technology/documentation/berkeley-db/db/gsg_txn/JAVA/lockingsubsystem.html#deadlockresolve
    - there's reference to "EnvironmentConfig.setLockDetectMode()". I'm using JE 3.2.23 and
    there doesn't appear to be any such function. Is this out-of-date documentation or am I
    using and old version?The link above is for documentation in Berkeley DB the C Edition, not Berkeley DB Java Edition (JE) -- you're reading the doc for the wrong product. The JE doc is here:
    http://www.oracle.com/technology/documentation/berkeley-db/je/TransactionGettingStarted/jelock.html
    There is no lock detect mode in JE, it always uses a timeout of the oldest locker.
    Mark

  • Reducing number of deadlock exceptions

    com.sleepycat.je.DeadlockException: (JE 3.2.13) Lock expired. Locker 15347458_pool-1-thread-10_Txn: waited for lock on database=tasks13587 node=73038 type=WRITE grant=WAIT_NEW timeoutMillis=500 startTime=1170088002362 endTime=1170088002866
    Owners: [<LockInfo locker="15347467_pool-1-thread-8_Txn" type="WRITE"/>]
    Waiters: []
    at com.sleepycat.je.txn.LockManager.lock(LockManager.java:266)
    at com.sleepycat.je.txn.Txn.lockInternal(Txn.java:296)
    at com.sleepycat.je.txn.Locker.lock(Locker.java:257)
    at com.sleepycat.je.dbi.CursorImpl.lockLNDeletedAllowed(CursorImpl.java:2349)
    at com.sleepycat.je.dbi.CursorImpl.lockLN(CursorImpl.java:2271)
    at com.sleepycat.je.dbi.CursorImpl.searchAndPosition(CursorImpl.java:1963)
    at com.sleepycat.je.Cursor.searchInternal(Cursor.java:1178)
    at com.sleepycat.je.Cursor.searchAllowPhantoms(Cursor.java:1148)
    at com.sleepycat.je.Cursor.search(Cursor.java:1014)
    at com.sleepycat.je.Cursor.getSearchKey(Cursor.java:556)
    at com.redwerk.webcrawler.storage.url.BDBJEStorage.update(BDBJEStorage.java:298)
    at com.redwerk.webcrawler.filter.record.UrlProcessedFilter._accept(UrlProcessedFilter.java:36)
    at com.redwerk.webcrawler.filter.record.ChainedFilter.accept(ChainedFilter.java:23)
    at com.redwerk.webcrawler.filter.record.ChainedFilter.accept(ChainedFilter.java:25)
    at com.redwerk.webcrawler.Crawler.addToQueue(Crawler.java:372)
    at com.redwerk.webcrawler.ConsumerThread.processNestedURLs(ConsumerThread.java:343)
    at com.redwerk.webcrawler.ConsumerThread.run(ConsumerThread.java:145)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
    at java.util.concurrent.FutureTask.run(FutureTask.java:123)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
    at java.lang.Thread.run(Thread.java:595)
    such exception occurs about 400 times on each 1500 database operations
    the method, which throws such exception, looks like this:
         * @see UrlStorageInterface#update(String, Record)
        public void update(String key, Record url) throws RetryRequiredException,
                RecoveryRequiredException, Exception {
            key = UrlNormalizer.toNormalcyForm(key);
            if (log.isDebugEnabled())
                log.debug("Updating " + url.getUrl());
            //put(key, url);
            Transaction txn = null;
            Record record = null;
            TupleBinding valueBinding = new RecordBinding();
            DatabaseEntry entry = new DatabaseEntry();
            Cursor cursor = null;
            try {
                txn = env.beginTransaction(null, null);
                cursor = tasksDB.openCursor(txn, null);
                if (cursor.getSearchKey(new DatabaseEntry(key.getBytes()), entry,
                        LockMode.RMW) == OperationStatus.SUCCESS) {
                    record = (Record) valueBinding.entryToObject(entry);
                    url.cloneInfo(record);
                    valueBinding.objectToEntry(record, entry);
                    cursor.putCurrent(entry);
                } else {
                    valueBinding.objectToEntry(url, entry);
                    tasksDB.put(txn, new DatabaseEntry(key.getBytes()), entry);
                cursor.close();
                cursor = null;
                txn.commit();
                txn = null;
            } catch (DeadlockException e) {
                log.error(e,e);
                throw new RetryRequiredException();
            } catch (RunRecoveryException e) {
                log.error(e,e);
                throw new RecoveryRequiredException();
            } catch (DatabaseException e) {
                log.error(e, e);
            } finally {
                if (cursor != null)
                    try {
                        cursor.close();
                    } catch (DatabaseException e) {
                        log.error(e, e);
                if (txn != null)
                    try {
                        txn.abort();
                    } catch (DatabaseException e) {
                        log.error(e, e);
        }

    Probably because many threads are trying to access
    the same record, as you say. This will cause lots of
    contention (blocking) in any system. Blocking will
    cause DeadlockException if the lock timeout is
    exceeded.That's true, as far as I can see in logs, it is possible several threads are accessing same record at a time.
    May be due to large amount of data records I needto
    increase deadlock detection timeout or somethinglike
    this?If you increase the lock timeout, you will probably
    see less deadlocks. But there will be roughly the
    same amount of contention. But you should try
    increasing the lock timeout and measure the
    performance impact. The number of deadlocks is only
    one measurement. The measurement that matters most
    is throughput (ops/sec).Looks like I can increase a timeout for all databases only, thus with another database (queue) I will see some issues because there are 2 secondary databases attacted to a queue database.
    In a crawler application do many crawling threads
    update the same URLs in a real world data set? Yes, imagine you have standard header and footer for a page, and you have several hundreds of thousands pages (online store). Our application is fetching the URLs from the page (including urls from header, which are common for entire website), then, before adding the url to queue, it checks - if the url is downloaded and parsed yet, if so - it updates the record in processed urls database (adds a new parent and probably some another information about the page). So several threads can update same record at a time.
    Or does this only occur in your test data set? In fact, it occurs only with live data (customer's website) :(
    Whenever many threads are competing for exclusive
    access to the same resource (a database record, or
    any other shared resource), the only way to reduce
    contention is to change the application so that this
    is not true.
    For example, you can partition the key space so that
    each thread that writes to the database works on a
    separate key range (where keys are URLs in your
    case). This means operations would have to be queued
    by crawler threads and dequeued by database threads,
    partitioning the keys by thread during the queuing
    process. That's one idea. The best approach depends
    on trade-offs in your application.Thank you, I was thinking about a bit different idea (which I was implementing earlier, and rejected to place in production) - utilize java.util.concurrent package with it's nice queues. So the storage implementation will be wrapped with decorator, which will intercept put, update and push operations and instead of accessing database - will just add records to queue. Separate thread will populate records from a queue and update database. To avoid case when such improvement is nothing than synchronized operations, it might be useful to add some kind of in-memory updating, so when a record is added to a queue, if record with similar key is already exists in a queue - it will be updated within queue.
    Separate thread will fetch records from queue and update respective databases.

Maybe you are looking for

  • Safari 6.0.5 pdf still won't print with cache emptied and relaunched on my MAC 10.8.5

    Emptied cache and relaunched Safari. The pdf I click on that shows a document goes blank when I click on print. I did learn that other documents in Safari will print after all, but not this one. Enabled cookies to no avail. Firefox works. Must be som

  • Need help in IS Oil and Gas

    Hi All, I have 3 years of Experience in SAP ABAP. Now i got an oppertunity to work on SAP IS oil and Gas. I don't have any Idea on IS oil and gas. how to start learn and is i am the suitable person to learn with ABAP knowledge. I have only one advant

  • 1099 WTH tax reporting

    Hi, We are looking at generating 1099 forms for previous years i.e. 2011 and 2012. The problem is every year the 1099 form is updated by a SAP Note as per changes made by IRS and whenever I print 2011 data, I get it in 2014 format. Has someone faced

  • CD drive makes a lot of noise when I insert a CD

    Whenever I insert a CD into my slot it makes a noise that sounds like its eating my CD -- Not sure if there is always a noise or if i have a problem with my computer.

  • What has happened to the graphics in my emails??

    What has happened to all the graphics in my emails?