Cannot retrieve data in all members of my distributed queue

Question

I have a distributed queue with 3 members, queue member 1, queue member 2, and queue member 3.
I triggered 2 events separately to go to the distributed queue, and it was successful as the 1st event triggered went to queue member 1 (physical queue1 – located on server 1) and the next event sent on the next trigger went to queue member 2 (physical queue 2 – located on server 2).
The problem now is, after triggering a job that gets the event from distributed queue, only the event in queue member 1 was picked up. The job got completed without picking up the event in queue member 2 (different VM). The data in queue member 2 was left. The config file for the JMS Queue Connectivity may be seen below:
jms.url=t3://<MS1_IP>:<MS1_PORT>,<MS2_IP>:<MS2_PORT>,<MS3_IP>:<MS3_PORT>
jms.connfactory=ConnFactoryName
jms.timeout=30000
jms.queue=DistributedQueueName
Note also that the connection factory is deployed to all managed servers as I could see the connection factory in the jndi tree of each managed server.
Our job was able to get data from each of the managed server, one by one. An example I tried is I placed an event in queue member 1 then triggered the job, then the data was picked up. Then next, I placed the data in queue member 2 (with no other data in queue member 1 and 3), then the data in queue member 2 was picked up after the job run, and same when placed only in queue member 3.
The main problem here is when there are data in all of the physical queues (queue member 1 to 3). As if the application is focused only to one physical queue, which shouldn’t be. Researching now how should it read all the members in the distributed queue.

Answer 1

Hi,
A comma separated URL list helps a client setup its initial JNDI connection into the cluster - it doesn't directly control load balancing of future JEE EJB, RMI, or JMS calls. By default, for remote clients, JEE calls will automatically load balance a new direct connection randomly among all servers in the cluster, even if the initial JNDI URL only references a single server's address. It's actually not truly random, as there may be internal heuristics that effect the load balance decision, plus the behavior is actually tunable, but, for almost all use cases, its best to stay with the default behavior and think of it as effectively random.
WebLogic JMS load-balancing actually occurs at three layers: URL/DNS, "smart stub" connection creation (RMI, EJB, and JMS connection factory), and finally the distributed destination layers. For a full end-to-end explanation of load balancing at the different layers of the WL stack, I highly recommend the JMS chapter of the book "Professional Oracle WebLogic".
What's happening in your case is that JMS individual consumer always load balances to one particular queue member, and then sticks to that queue member for their lifetime. This is expected behavior for consumer load balancing, and is fully documented in the distributed destination documention of the JMS programmer's guide.
If you want to ensure full coverage of a distributed queue, in order of highest to lowest preference:
* Most highly recommend: Use a WL MDB or a 11gR1PS3 SOA Adapter (if you happen to be using Oracle's SOA layered product). These automatically ensure that all distributed queue members are serviced by at least one dedicated consumer.
* Ensure you have more sessions+consumers than queue members. Periodically refresh connections/sessions/consumers in order to ensure that newly started members are serviced even when all consumers are already connected (eg, code your clients to reconnect every few minutes).
* If your application is not performance sensitive, configure the "queue forward delay" setting on a distributed queue. Messages that are trapped on a queue member with no consumers for longer than this delay will be automatically forwarded a member that has consumers.
* Least recommended, very advanced usage only: Code clients to use the new 10.3.4 public "destination availability" extensions. These extension APIs provide a notification service for distributed destination member up/down events.
Regards,
Tom