Packet acknowledge failed after failover

Question

Hello,
We're running MQ 3.5 SP1 EE and we are wondering how to deal with the following problem. We have a client that consists of 2 producers and 1 consumer. There is a single connection for the client. Each producer creates its own session with:createSession(TRANSACTED_SESSION,Session.AUTO_ACKNOWLEDGE) The consumer creates its session with:
createSession(TRANSACTED_SESSION,Session.DUPS_OK_ACKNOWLEDGE)                       We have the following reconnect settings:imqReconnectEnabled = true
imqReconnectAttempts = 1
imqAddressListIterations = -1
imqReconnectInterval = 30000
imqAddressListBehavior = RANDOMSometimes when we kill one of our cluster servers, the client threads begin throwing exceptions as follows. From the send() method from one of the producers:
com.sun.messaging.jms.JMSException: [C4000]: Packet acknowledge failed.From the send() method of the other producer (a few seconds later):com.sun.messaging.jms.JMSException: [C4001]: Write packet failed. - cause: java.net.SocketException: Socket is closedFinally, an exception is caught by the consumer's exception listener:
JMSException caught: [C4002]: Read packet failed. - cause: java.net.SocketException: Socket is closedWhat is the best way to recover from this? We were thinking of closing the connection, reopening, and starting all sessions again. Is there anything smarter we can do?
Kernel: 2.4.21-9.0.3.ELsmp
Dist: RedHat ES 3.0
Thanks,
Aaron

Answer 1

I know this is a long shot, but what was the outcome of this? How did you fix this issue? I am facing the same issue with MQ 4.3 (flaky network, occasional failure to recover, even with infinite recovery configured).
Once the 'Packet acknowledge failed' error occurs (see below), the consumer recovery thread gets stuck in a loop of retrying, but never recovers, even though it says it has (it thinks it's connected, but no messages are ever delivered to it). Since the network is flaky, the consuming client is often disconnected, but usually recovers okay (thanks to using the imq reconnect attributes), but if the error below occurs, that client never reconnects to the broker:
2009-10-28 13:58:52,812 INFO [com.xxx.MessageListenerContainer] Setup of JMS message listener invoker failed - trying to recover: com.sun.messaging.jms.JMSException: [C4000]: Packet acknowledge failed. user=user, broker=xxx:1093(8095)
Here's the logs of the recovery that results in a consumer that never gets anything delivered to it...
2009-10-28 13:58:52,875 ERROR [STDERR] 28-Oct-2009 13:58:52 com.sun.messaging.jmq.jmsclient.ConnectionRecover logRecoverState
INFO: [I107]: Connection recover state: RECOVER_TRANSPORT_CONNECTED, broker: xxx1:1093(8095)
2009-10-28 13:58:52,875 ERROR [STDERR] 28-Oct-2009 13:58:52 com.sun.messaging.jmq.jmsclient.ConnectionRecover logRecoverState
INFO: [I107]: Connection recover state: RECOVER_STARTED, broker: xxx1:1093(8095)
2009-10-28 13:58:52,875 ERROR [STDERR] 28-Oct-2009 13:58:52 com.sun.messaging.jmq.jmsclient.ConnectionRecover logRecoverState
INFO: [I107]: Connection recover state: RECOVER_IN_PROCESS, broker: xxx1:1093(8095)
2009-10-28 13:58:52,875 ERROR [STDERR] 28-Oct-2009 13:58:52 com.sun.messaging.jmq.jmsclient.ConnectionRecover logRecoverState
INFO: [I107]: Connection recover state: RECOVER_SUCCEEDED, broker: xxx1:1093(8095)
2009-10-28 13:58:52,875 ERROR [STDERR] 28-Oct-2009 13:58:52 com.sun.messaging.jmq.jmsclient.ConnectionRecover logRecoverState
INFO: [I107]: Connection recover state: RECOVER_INACTIVE, broker: xxx1:1093(8095)
2009-10-28 13:58:53,000 INFO [com.xxx.MessageListenerContainer] Successfully refreshed JMS Connection
But ultimately, the client goes into a state where it doesn't recover. Here you can see 4 clients trying to recover from previous failures:
2009-10-28 14:02:39,796 ERROR [STDERR] 28-Oct-2009 14:02:39 com.sun.messaging.jmq.jmsclient.ExceptionHandler logCaughtException
WARNING: [I500]: Caught JVM Exception: java.net.SocketException: Socket closed
2009-10-28 14:02:42,500 ERROR [STDERR] 28-Oct-2009 14:02:42 com.sun.messaging.jmq.jmsclient.ExceptionHandler logCaughtException
WARNING: [I500]: Caught JVM Exception: java.net.SocketException: Socket closed
2009-10-28 14:02:42,500 ERROR [STDERR] 28-Oct-2009 14:02:42 com.sun.messaging.jmq.jmsclient.ExceptionHandler logCaughtException
WARNING: [I500]: Caught JVM Exception: java.net.SocketException: Socket closed
2009-10-28 14:02:42,500 ERROR [STDERR] 28-Oct-2009 14:02:42 com.sun.messaging.jmq.jmsclient.ExceptionHandler logCaughtException
WARNING: [I500]: Caught JVM Exception: java.net.SocketException: Socket closed
Each time the failure occurs, we get an additional exception. Rebooting the MQ broker allows a new client to be created and the messages are delivered, but the existing (previously failed) clients, are still left trying to connect.
Sounds like we're using some stale connection perhaps? We're using Spring's DefaultMessageListenerContainer. Could it be that the container is caching an old connection and trying to re-use it rather than creating a new one? It's not clear to me (yet) whether the mq client hides the failure from Spring's DefaultMessageListenerContainer and so doesn't give the Spring class the chance to recover itself.
Many thanks