Sun's MQ Clustering issue.

Question

Cluster issue with Sun Message Queue 3.5
We are developing a basic ESB (Enterprise Service Bus) system with MQ3 as the JMS
provider. Naturally we want to cluster multiple brokers on different boxes incase one
fails. For development and testing we�ve setup two MQ3 brokers on two different
Windows 2000 machines and both are configured the same and have created the same
destinations on both. We have one producer client running on a RS6000 AIX machine
sending messages to a message queue, and one consumer client running on a iSeries
reading the messages. (Note: There are no problems when we�re not clustering and using
just one broker). The problem is after the consumer processes about 1k-3k messages it
like drops its active connection to the message queue. Looking at the Administration
Console it appears the active consumer rolls down to backup consumer. Here is what the
console shows.
When working ok:
Current Number of Messages: 65231
Current Total Message Bytes: 501033112 bytes
     Current Number of Producers: 1
Current Number of Active Consumers: 1
Current Number of Backup Consumers: 0
When it stops receiving messages:
Current Number of Messages: 72652
Current Total Message Bytes: 515987678 bytes
     Current Number of Producers: 1
Current Number of Active Consumers: 0
Current Number of Backup Consumers: 1
At this point the consumer does not pick up anymore messages, it waits on onMessage().
However, the producer keeps on running just fine. Also we�re using persistent queues,
not topics.
Following are the configuration settings.
Here is the startup screen shot, both brokers are the same except for the name, this one is
CFERRIS the other is CFERRIS2K so I just show one here.
C:\Program Files\Sun\MessageQueue3\bin>imqbrokerd -license try
[12/Apr/2005:11:35:40 CDT]
================================================================================
Sun Java(tm) System Message Queue
Sun Microsystems, Inc.
Version:  3.5 SP2  (Build 48-G)
Compile:  Fri 08/27/2004
Copyright ⌐ 2004 Sun Microsystems, Inc. All rights reserved.
Use is subject to license terms.
This product includes code licensed from RSA Data Security.
================================================================================
Java Runtime: 1.4.2_05 Sun Microsystems Inc. c:\j2sdk1.4.2_05\jre
*****  Trial license enabled.  This license will expire on     *****
*****  Jun 28, 2005.  This license is for evaluation purposes  *****
*****  only. You are in violation of the license agreement if  *****
*****  you use this product with this license enabled in a     *****
*****  production environment.                                 *****
[12/Apr/2005:11:35:40 CDT]    IMQ_HOME=C:\Program Files\Sun\MessageQueue3
[12/Apr/2005:11:35:40 CDT] IMQ_VARHOME=C:\Program Files\Sun\MessageQueue3\var
[12/Apr/2005:11:35:41 CDT] Windows 2000 5.0 x86 CFERRIS (1 cpu) cferris
[12/Apr/2005:11:35:41 CDT] Java Heap Size: max=194432k, current=16256k
[12/Apr/2005:11:35:41 CDT] Arguments: -license try
[12/Apr/2005:11:35:41 CDT] [B1060]: Loading persistent data...
[12/Apr/2005:11:35:41 CDT] Using built-in file-based persistent store: C:\Progra
m Files\Sun\MessageQueue3\var\instances\imqbroker\
[12/Apr/2005:11:35:45 CDT] [B1039]: Broker "imqbroker@CFERRIS:7676" ready.From the config.properties for both brokers, they are the same:
#Last Update:
#Fri Apr 08 15:19:45 CDT 2005
imq.jms.min_threads=10
imq.instanceconfig.version=300
imq.autocreate.queue=false
imq.jms.tcp.port=0
imq.cluster.url=file\:\\H\:\\MIS\\928DataBrokerInterfaces\\MOM\\cluster.properties
imq.autocreate.topic=false
imq.jms.max_threads=1000
From the cluster.properties file:
#Last Update:
#Fri Mar 25 15:47:49 CST 2005
imq.cluster.brokerlist=cferris2k:7676,cferris:7676
imq.cluster.masterbroker=cferris2k
From imqcmd query bkr which are the same on both boxes.
Querying the broker specified by:
Host         Primary Port
localhost    7676
Version                                              3.5 SP2
Instance Name                                        imqbroker
Primary Port                                         7676
Current Number of Messages in System                 0
Current Total Message Bytes in System                0
Max Number of Messages in System                     unlimited (-1)
Max Total Message Bytes in System                    unlimited (-1)
Max Message Size                                     70m
Auto Create Queues                                   false
Auto Create Topics                                   false
Auto Created Queue Max Number of Active Consumers    1
Auto Created Queue Max Number of Backup Consumers    0
Cluster Broker List (active)                         CFERRIS/10.3.xxx.xx:7676 (imqbroker), CFERRIS2K/10.3.xxx.xx:7676(imqbroker)
Cluster Broker List (configured)                     cferris2k:7676,cferris:7676
Cluster Master Broker                                cferris2k
Cluster URL                                          file:\H:\MIS\Project Office
\928DataBrokerInterfaces\A-Request-Requirements\MOM\cluster.properties
Log Level                                            INFO
Log Rollover Interval (seconds)                      604800
Log Rollover Size (bytes)                            unlimited (-1)
Successfully queried the broker.From imqcmd query dst
C:\>imqcmd query dst -t q -n resultsFromBOS
Querying the destination where:
Destination Name    Destination Type
resultsFromBOS      Queue
On the broker specified by:
Host         Primary Port
localhost    7676
Destination Name                      resultsFromBOS
Destination Type                      Queue
Destination State                     RUNNING
Created Administratively              true
Current Number of Messages            0
Current Total Message Bytes           0
Current Number of Producers           0
Current Number of Active Consumers    0
Current Number of Backup Consumers    0
Max Number of Messages                unlimited (-1)
Max Total Message Bytes               unlimited (-1)
Max Bytes per Message                 unlimited (-1)
Max Number of Producers               unlimited (-1)
Max Number of Active Consumers        unlimited (-1)
Max Number of Backup Consumers        3
Limit Behavior                        REJECT_NEWEST
Consumer Flow Limit                   1000
Is Local Destination                  false
Local Delivery is Preferred           false
Successfully queried the destination.For development and testing I am not using JDNI. The following shows
how I configured the connection factory.
    public ConsumerClient(int acknowledgementMode) {
      // Initiate the factory and connection.
      // Also directly instantiate and configure the object.
      connectFactory = new com.sun.messaging.ConnectionFactory();
      try {
          connectFactory.setProperty("imqAddressList",
                                                      �mq\://cferris2k\:7676/jms,mq\://cferris\:7676/jms�);
          connectFactory.setProperty("imqReconnectEnabled", �true�);
          // Set client ID/user ID
          connectFactory.setProperty("imqDefaultUsername", "admin");
          connectFactory.setProperty("imqDefaultPassword", "admin");
          connectFactory.setProperty("imqAddressListIterations", "-1");
          connectFactory.setProperty("imqReconnectAttempts", "5");
          connectFactory.setProperty("imqAddressListBehavior", "RANDOM");
          // Next create the connection and session
          connection = connectFactory.createConnection();
          session = connection.createSession(false, acknowledgementMode);
          logger.info("A connection and session have been established by ConsumerClient");
      catch(JMSException jmsError){
          reportError("Error creating the connection factory. Following is the message: ",
                       jmsError.getMessage());
    }  // of constructor I added the three properties imqAddressListIterations, imqReconnectAttempts
and imqAddressListBehavior as an attempt to fix it, but this did not work, so
previously they were the defaults. I am wondering if I needed to create destination
queues on all the brokers or just the master?
Any help with figuring this will greatly be appreciated!!
Thanks, Chris

Answer 1

Chris,
This is fishy.
Since the queue is configured to have unlimited active consumers, it should never attach a consumer as a backup. It seems like when the client reconnects, it somehow is establishing the consumers as a backup.
The first thing I'd try is to change maxNumBackupConsumers on the queue to 0. Hopefully this will either keep the consumer a primary (active) consumer, or trigger another error that would shed more light.
The other curious thing is why is the client loosing it's connection in the first place? To narrow this down I would turn off reconnect (i.e. set imqReconnectEnabled to false), and register a connection exception listener (Connection.setExceptionListener) and see what exception is being generated. Also check the broker's log file and see if there are any interesting messages.
When you have brokers clustered, you can create a destination on any broker in the cluster, and the destination will be propogated across the cluster. The following SunSolve document contains more details:
79051 Sun Java[TM] System Message Queue: Understanding the Role of a Master Broker in an MQ Cluster
And one other thing about clusters. Messages are persisted on the broker the sender is connected to. Messages are then forwarded to consumers anywhere on the cluster. Acknowledgements flow back to the "producing" broker and are processed by it. If a broker goes down, the messages it holds in its persistent store are not available until that broker comes back up. We're looking at improving data availability in a future MQ release and you can get it today in conjuction with the SunCluster product.
Joe