Thread deadlocked on itself?

Question

Hi,
we have a production server, where threads become occassionally unresponsive. At that time, we take a thread dump. A few of those thread dumps show a thread is waiting to lock an object. Strangely, a few lines down in the stack trace, the same thread has the object locked. No other thread has the particular object locked or waits on it.
How is it possible that a thread can be waitin for an object that, it has locked itself?
FYI, the first time, I saw this, I thought it was a fluke. Perhaps the thread wasn't really waiting, but in the progress of getting the lock. Though, this issues is showing up a few times a week.
Because the entire thread dump is very huge (long stack traces, lots of threads), here are only the last lines of the relevant stack trace:
"/atg/dynamo/server/DrpServer-11:ipaddr=68.212.254.42;path=/store/web/login/login.jsp;sessionid=54ACNL0PIMC5UCQ2M5SSAOQKDRT4IIV0" prio=1 tid=0x85252f08 nid=0x3b15 in Object.wait() [86c88000..86c8a8c4]
at java.lang.Object.wait(Native Method)
- waiting on <0x71c4bfc8> (a atg.service.lockmanager.ClientLockEntry)
at atg.service.lockmanager.ClientLockEntry.acquireReadLock(ClientLockEntry.java:597)
- locked <0x71c4bfc8> (a atg.service.lockmanager.ClientLockEntry)
at atg.service.lockmanager.ClientLockManager.acquireReadLock(ClientLockManager.java:977)
at atg.adapter.gsa.ItemTransactionState.acquireReadLock(ItemTransactionState.java:719)
at atg.adapter.gsa.GSAItem.getPersistentPropertyValue(GSAItem.java:583)
at atg.adapter.gsa.GSAItem.getPropertyValue(GSAItem.java:478)
at atg.adapter.gsa.GSAItem.getItemTransactionState(GSAItem.java:1471)
at atg.adapter.gsa.GSAItem.getItemTransactionState(GSAItem.java:1351)
at atg.adapter.gsa.GSAItem.getItemTransactionStateUnchecked(GSAItem.java:1543)
at atg.adapter.gsa.GSAItem.getPropertyValue(GSAItem.java:735)
at atg.repository.RepositoryItemImpl.getPropertyValue(RepositoryItemImpl.java:127)

Answer 1

we have a production server, where threads become
occassionally unresponsive. At that time, we take a
thread dump. A few of those thread dumps show a
thread is waiting to lock an object. Strangely, a few
lines down in the stack trace, the same thread has
the object locked. No other thread has the particular
object locked or waits on it.You said occasionally...you mean not frequently, don�t you? Just a wild guess: perhaps the notify() method is being used instead of notifyAll(). When you use notify() you wake up only one thread. Therefore, if some problem occur in this only one awaken thread, so that it is not able to wake up the other threads, then you will have a big deadlock like the problem you are describing.
The problem that might happen with that only one awaken thread I mentioned can be everything unpredictable you can imagine. The JVM environment is subject to any kind of unpredictable problems. Not only JVM, but also any other software is subject to these same kinds of problem. Although these problems are rare, they can happen.
If notifyAll() is used instead of notify(), at least all the threads are supposed to be awaken, and then, even if some problem occur with the awaken thread that is holding the monitor, other threads are permitted to hold the monitor, since all of them are awaken, and therefore the probability of occurring that big unpredictable deadlock decreases significantly.
Maybe it sounds like notifyAll() is always better than notify(). I am not saying that. I am just saying something that is possibly happening in your case. I suggest you reviewing all the code where the handling of threads is being done. Try to find some suspicious piece of code that is causing the problem, related to those basic issues about threads. And after you are sure that everything related to the basic rules of thumb about thread handling is ok, then think about what I said (about those unpredictable and rare problems), and check if using notifyAll() instead of notify() makes your application works better.