Deadlocks and more on a large database.

Question

Hi,
We are running a multithreaded server. The database is performing as cache, which is queried/updated about 500 times/sec by about 500 threads.
Most queries are misses.
We are experiencing a large number of deadlocks in the database, and sometime get/put operations fails to complete even after 20 retries.
We would please like to know the following (not all questions related to the deadlock issue):
1. How can we reduce the number of deadlocks ?
2. Is using deferred mode favorable in this use case ?
3. How do we find the optimal log file size ?
4. Are there any more configuration properties we need to consider ?
Thanks,
Lior

Answer 1

've tried changing the lock timeout to the default,
as specified in the je.properties file. (50000).
however, it seems the change has no effect, and the
deadlocks occur in the same frequency as before.Ok.
we've also tried without much success:
envConfig.setConfigParam("je.env.sharedLatches",
"true");
envConfig.setConfigParam("je.lock.nLockTables",
"401");The above params have no impact on deadlocks.
Is there any other configuration properties I can
change?No, except perhaps to increase the cache size so that operations complete more quickly. An operation that does a read from disk will take much longer than when the record is in cache.
Here's the stack trace, after modifying the lock
timeout property:
2008-07-23 16:55:54,977 WARN
[com.flash.http.postOptCache.service.PostOptCacheFacad
eImpl] com.sleepycat.je.DeadlockException: (JE
3.2.68) Lock expired. Locker
-1_HPI-executor707_ThreadLocker: waited for lock on
database=postoptcache node=4211902 type=WRITE
grant=WAIT_NEW timeoutMillis=500
startTime=1216821354475 endTime=1216821354977
Owners: [<LockInfo
locker="-1_HPI-executor825_ThreadLocker"
type="WRITE"/>]
Waiters: []
com.sleepycat.je.DeadlockException: (JE 3.2.68) Lock
expired. Locker -1_HPI-executor707_ThreadLocker:
waited for lock on database=postoptcache node=4211902
type=WRITE grant=WAIT_NEW timeoutMillis=500
startTime=1216821354475 endTime=1216821354977This information tells us that thread HPI-executor707 is trying to perform a write operation and times out after 500 millis. The write lock on the record is not granted because thread HPI-executor825 owns the write lock on that record.
Thread HPI-executor707 is performing a Database.delete. We don't know what thread HPI-executor825 is doing, but we can assume that is holding the record lock for more than 500 millis.
You have a couple options:
1) Find out why HPI-executor825 is holding the lock for so long, and try to correct that problem. For example, if it is keeping a cursor open, try to close the cursor earlier.
2) If you're sure that no thread is keeping a cursor open for any longer than necessary, then it is possible that something else is slowing things down. Is I/O very slow on this system? Are full GCs taking a long time? If so, you can try to correct these problems, but you can also increase the lock timeout to a value larger than any delay caused by I/O or GC.
Basically, you need to find out why the record lock is being held by thread HPI-executor825 for more than 500 millis. Then either correct that situation so that the lock is not held for so long, or increase the lock timeout if there is no way to reduce the time that the lock is held.
--mark