Client reconnection to conventional cluster taking about 9 minutes

Question

Hi,
I've set up a 3 node OpenMq (4.4) conventional cluster and am having trouble with client reconnections when I simulate a halt on a node.
I have two threads within the same process, sending to a pre-configured destination and replying using a temporary queue. Each have a JMS connection to the same home broker. If I bring the home broker down with either the imqcmd command or with a ctrl+c, the goodbye messages are sent and the client connections are immediately reestablished by the connection factory with the next broker. But... if I remove the broker from the cluster by simulating a BSOD, power failure, etc, no goodbye messages are sent (obviously) and the connected clients are left 'connected' to the dead broker and only reconnect to another broker after about 9 minutes.
I read in the docs that the 'imqPingInterval' property can be used to test the client connections but it doesn't seem to be doing the trick. After I've cut the power to the broker's virtual machine, I'm seeing the ping messages in the logs, e.g.:
FINEST: Outbound Packet:PING(54):296-127.0.1.1(bd:ea:13:9b:ae:a)-51155-1267024410861;BrokerAddress=10.59.148.9:7676(1082), ConnectionID=4090718717872600064, ReconnectEnabled: true, IsConnectedToHABroker: falseAt this point, telnet cannot connect to 10.59.148.9:7676 so it's definitely gone.
Finally after about 9 minutes, the following shows up in the logs:
24-Feb-2010 15:26:26 com.sun.messaging.jmq.jmsclient.ExceptionHandler throwJMSException
FINER: I501
com.sun.messaging.jms.JMSException: [C4002]: Read packet failed. - cause: java.net.SocketException: No route to host
        at com.sun.messaging.jmq.jmsclient.ExceptionHandler.getJMSException(ExceptionHandler.java:380)
        at com.sun.messaging.jmq.jmsclient.ExceptionHandler.handleException(ExceptionHandler.java:331)
        at com.sun.messaging.jmq.jmsclient.ProtocolHandler.readPacket(ProtocolHandler.java:1796)
        at com.sun.messaging.jmq.jmsclient.ReadChannel.run(ReadChannel.java:1197)
        at java.lang.Thread.run(Thread.java:619)
Caused by: java.net.SocketException: No route to host
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
        at com.sun.messaging.jmq.io.ReadOnlyPacket.readFully(ReadOnlyPacket.java:271)
        at com.sun.messaging.jmq.io.ReadOnlyPacket.readFixedHeader(ReadOnlyPacket.java:191)
        at com.sun.messaging.jmq.io.ReadOnlyPacket.readPacket(ReadOnlyPacket.java:151)
        at com.sun.messaging.jmq.io.ReadWritePacket.readPacket(ReadWritePacket.java:82)
        at com.sun.messaging.jmq.jmsclient.ProtocolHandler.readPacket(ProtocolHandler.java:1758)
        ... 2 more
24-Feb-2010 15:26:26 com.sun.messaging.jmq.jmsclient.ConnectionImpl logLifeCycle
FINE: Connection closed.  The connection is closed due to a network problem, broker crashed, or internal error: BrokerAddress=10.59.148.9:7676(1082), ConnectionID=4090718717872604160, ReconnectEnabled: true, IsConnectedToHABroker: falseAnd we can then send/receive again.
The connection factory details are as follows:
Class:                  com.sun.messaging.ConnectionFactory
getVERSION():           3.0
isReadonly():           false
getProperties():        {imqOverrideJMSPriority=false, imqConsumerFlowLimit=1000, imqOverrideJMSExpiration=false, imqAddressListIterations=1, imqLoadMaxToServerSession=true, imqConnectionType=TCP, imqPingInterval=30, imqSetJMSXUserID=false, imqConfiguredClientID=, imqSSLProviderClassname=com.sun.net.ssl.internal.ssl.Provider, imqJMSDeliveryMode=PERSISTENT, imqConnectionFlowLimit=1000, imqConnectionURL=http://localhost/imq/tunnel, imqBrokerServiceName=, imqJMSPriority=4, imqBrokerHostName=localhost, imqJMSExpiration=0, imqAckOnProduce=, imqEnableSharedClientID=false, imqAckTimeout=10000, imqAckOnAcknowledge=, imqConsumerFlowThreshold=50, imqDefaultPassword=guest, imqQueueBrowserMaxMessagesPerRetrieve=1000, imqDefaultUsername=guest, imqReconnectEnabled=true, imqConnectionFlowCount=100, imqAddressListBehavior=PRIORITY, imqReconnectAttempts=1, imqSetJMSXAppID=false, imqConnectionHandler=com.sun.messaging.jmq.jmsclient.protocol.tcp.TCPStreamHandler, imqSetJMSXRcvTimestamp=false, imqBrokerServicePort=0, imqDisableSetClientID=false, imqSetJMSXConsumerTXID=false, imqOverrideJMSDeliveryMode=false, imqBrokerHostPort=7676, imqQueueBrowserRetrieveTimeout=60000, imqSetJMSXProducerTXID=false, imqSSLIsHostTrusted=false, imqConnectionFlowLimitEnabled=false, imqReconnectInterval=3000, imqAddressList=mq://10.59.148.9,mq://10.59.148.17,mq://10.59.148.11, imqOverrideJMSHeadersToTemporaryDestinations=false}The brokers are virtualbox guests running 'WinXP 64' and the producer and consumer are both running on Ubuntu 9.10.
Handling a machine failure seems like a pretty standard scenario for a cluster so I'm sure there's something I've mis-configured.
Is there something I can do to fix this?
If not, the only other alternatives I see are to either:
1) Try the HA custer. But I'm not sure this will fix the problem if the client is waiting on a dead socket.
2) Lower the message ack to something like 10 seconds and reconnect to the cluster for a retry every time we get a JMSException. But this is obviously not ideal either.
Any suggestions would be great.
Thanks,
Nick

Answer 1

Hi Nigel,
The producer is configured to send messages every 5 seconds and the consumer is sitting with a message listener on the queue.
The producer send is failing each time with the following trace...
5-Feb-2010 12:13:19 com.sun.messaging.jmq.jmsclient.ProtocolHandler writePacketNoAck
FINEST: Outbound Packet:OBJECT_MESSAGE(5):30-127.0.1.1(a2:3f:b1:e2:fd:33)-45573-1267099999075;BrokerAddress=10.59.148.9:7676(1184), ConnectionID=882185479630361088, ReconnectEnabled: true, IsConnectedToHABroker: false
25-Feb-2010 12:13:19 com.sun.messaging.jmq.jmsclient.ProtocolHandler writePacketNoAck
FINEST: sent packet ... OBJECT_MESSAGE(5):30-127.0.1.1(a2:3f:b1:e2:fd:33)-45573-1267099999075
25-Feb-2010 12:13:19 com.sun.messaging.jmq.jmsclient.ProtocolHandler writePacketNoAck
FINEST: Outbound Packet:PING(54):31-127.0.1.1(eb:66:36:db:8:79)-45574-1267099999156;BrokerAddress=10.59.148.9:7676(1184), ConnectionID=882185479630365184, ReconnectEnabled: true, IsConnectedToHABroker: false
25-Feb-2010 12:13:19 com.sun.messaging.jmq.jmsclient.ProtocolHandler writePacketNoAck
FINEST: sent packet ... PING(54):31-127.0.1.1(eb:66:36:db:8:79)-45574-1267099999156
25-Feb-2010 12:13:24 com.sun.messaging.jmq.jmsclient.ProtocolHandler writePacketNoAck
FINEST: Outbound Packet:PING(54):32-127.0.1.1(a2:3f:b1:e2:fd:33)-45573-1267100004156;BrokerAddress=10.59.148.9:7676(1184), ConnectionID=882185479630361088, ReconnectEnabled: true, IsConnectedToHABroker: false
25-Feb-2010 12:13:24 com.sun.messaging.jmq.jmsclient.ProtocolHandler writePacketNoAck
FINEST: sent packet ... PING(54):32-127.0.1.1(a2:3f:b1:e2:fd:33)-45573-1267100004156
25-Feb-2010 12:13:24 com.sun.messaging.jmq.jmsclient.ProtocolHandler writePacketNoAck
FINEST: Outbound Packet:PING(54):33-127.0.1.1(eb:66:36:db:8:79)-45574-1267100004156;BrokerAddress=10.59.148.9:7676(1184), ConnectionID=882185479630365184, ReconnectEnabled: true, IsConnectedToHABroker: false
25-Feb-2010 12:13:24 com.sun.messaging.jmq.jmsclient.ProtocolHandler writePacketNoAck
FINEST: sent packet ... PING(54):33-127.0.1.1(eb:66:36:db:8:79)-45574-1267100004156
25-Feb-2010 12:13:29 com.sun.messaging.jmq.jmsclient.AckQueue printInfo
WARNING: [W2003]: Broker not responding [OBJECT_MESSAGE(5)] for 10 seconds. Still trying..., broker addr=10.59.148.9:7676(1184), connectionID=882185479630361088, clientID=null, consumerID=14
25-Feb-2010 12:13:29 com.sun.messaging.jmq.jmsclient.ExceptionHandler throwJMSException
FINER: I501
com.sun.messaging.jms.JMSException: [C4000]: Packet acknowledge failed. user=guest, broker=10.59.148.9:7676(1184)
     at com.sun.messaging.jmq.jmsclient.ProtocolHandler.writePacketWithAck(ProtocolHandler.java:712)
     at com.sun.messaging.jmq.jmsclient.ProtocolHandler.writePacketWithAck(ProtocolHandler.java:575)
     at com.sun.messaging.jmq.jmsclient.ProtocolHandler.writePacketWithReply(ProtocolHandler.java:430)
     at com.sun.messaging.jmq.jmsclient.ProtocolHandler.writeJMSMessage(ProtocolHandler.java:1919)
     at com.sun.messaging.jmq.jmsclient.WriteChannel.sendWithFlowControl(WriteChannel.java:154)
     at com.sun.messaging.jmq.jmsclient.WriteChannel.writeJMSMessage(WriteChannel.java:107)
     at com.sun.messaging.jmq.jmsclient.SessionImpl.writeJMSMessage(SessionImpl.java:770)
     at com.sun.messaging.jmq.jmsclient.MessageProducerImpl.writeJMSMessage(MessageProducerImpl.java:203)
     at com.sun.messaging.jmq.jmsclient.MessageProducerImpl.writeJMSMessage(MessageProducerImpl.java:192)
     at com.sun.messaging.jmq.jmsclient.MessageProducerImpl.send(MessageProducerImpl.java:624)
     at com.sun.messaging.jmq.jmsclient.QueueSenderImpl.send(QueueSenderImpl.java:97)
     at uk.co.mydomain.jms.ClientJmsConnection.sendMessage(ClientJmsConnection.java:278)
     at uk.co.mydomain.jms.JmsClientMessageBroker.sendMessage(JmsClientMessageBroker.java:147)
     at test.uk.co.mydomain.messaging.quicktest.QuickTest$2.run(QuickTest.java:67)Is there any trace I could get that may be useful?
Thanks,
Nick