JMS Deadlock

Question

Hi,
Recently we experienced a Deadlock which seemed to be related to JMS processing. I did a Thread Dump whilst the server was in this state which showed a number of "Blocked Lock Chains", which boiled down to 2 "Open Lock Chains".
The first was weblogic.socket.Muxer thread that seemed to be waiting in an "epollWait" loop, and held a fat lock on a java/lang/String object (that a number of other threads were waiting on).
The second was an ExecuteThread that had been initiated as a part of Tx completion processing, for which we have a Tx afterCompletion() synchronisation listener registered. The Tx Synchronisation listener sends one or more messages to JMS Topics - or at least in this case it attempts to. It gets stuck with the following message:
-- Waiting for notification on: weblogic/jms/backend/BEProducerSendRequest@0x2ac20bb500[fat lock]
and inlcudes a number of the following messages in the stack trace:
-- Lock released while waiting: weblogic/jms/backend/BEProducerSendRequest@0x2ac20bb500[fat lock]
These problems would seemed to be a consequence of a number of the following errors that had occurred in the hours prior to the deadlock:
<BEA-000503> <Incoming message header or abbreviation processing failed - Bad abbreviation value: '244' <- a number of these occurred, some with different values
<BEA-000503> <Incoming message header or abbreviation processing failed - ClassCastException: weblogic.rjvm.ClassTableEntry
Could the deadlock be the result of erroneous failure recovery from the above message processing failures?
Or could it be related to my other suspicion of some kind of race condition related to fact that we are trying to send a JMS Topic message as a part of the Tx afterCompletion processing potentially from the receipt of a JMS message itself, whilst a number of other JMS messages are also being processed?
We are using Weblogic 9.2 MP2 under JRockit 3.0.3 64 bit on Redhat Linux AS4 Update 4.
We use a number of JMS Queues, some of which are controlled via the UnitOfOrder feature to ensure in order message processing. The JMS Topics mentioned above will have a number of Durable Subscribers attached.
Any thoughts/suggestions would be much appreciated.

Answer 1

Greg, did you get to a solution on this?
I am seeing the same type of messages:
Open lock chains
================
Chain 1:
"ExecuteThread: '33' for queue: 'weblogic.kernel.Default'" id=62 idx=0x80 tid=19223 waiting for java/lang/String@0xbd00b10 held by:
"ExecuteThread: '3' for queue: 'weblogic.kernel.Default'" id=32 idx=0x44 tid=19192 (waiting on notification)
Chain 2:
"ExecuteThread: '45' for queue: 'weblogic.kernel.Default'" id=74 idx=0x98 tid=19235 waiting for java/lang/String@0x1d1083a8 held by:
"ExecuteThread: '11' for queue: 'weblogic.kernel.Default'" id=40 idx=0x54 tid=19200 (waiting on notification)
Chain 3:
"ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" id=100 idx=0xc8 tid=19297 waiting for java/lang/String@0x9c1fb10 held by:
"ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" id=102 idx=0xcc tid=19299 (active)
Can you let me know if you have had any success with this issue?
Thanks for all your help.
Regards
RV