Queue: one producer vs multiple consumers

Hi,
This may be a simple question. In my application, I have a master/multiple slaves queue architecture. So the master enqueues messages. Each message is adressed to only one slave. Since all the slaves are waiting after queue elements, they all should receive the message and all dismiss it but one, the destinator. Apparently, it is not working like that. Once the message is dequeued by a slave, it is no more available in the queue and the others are not notified. Is that right?
I could use a seperate queue for each slave, but when it comes to be 100 slaves, I find it a but heavy to manage hundred queues. I tried instead to do a router-like architecture. Everyone is listening and for each message they all verify who is the destinator. Only the destinator processes the message and the other ones simply dismiss it. Is there a way for me to implement that?
Joe

It actually sounds like a fairly dicey problem, depending on your need for response times and total messaging bandwidth. Many of the issues seem to revolve around multiple producers and consumers wanting simultaneous read and write access to a continuously changing entity. At least that's how it sounded to me, that all the consumers are independent and should be allowed to consume their own messages just as soon as they're able to notice them.
For example:
Consumer #3 performs a preview and discovers that there's a message waiting at position #9. In order to pull it out of the queue, it must first dequeue the first 8 messages in line. Then after dequeueing its own message, it must re-enqueue those 1st 8 messages, this time using "Enqueue at Opposite End" to restore the original sequence.
However, before it can finish re-enqueue'ing those 8, Consumer #11 performs a preview and discovers that there's a message at (apparent) position 4. This really *should* be position 12, but how can Consumer #11 know what #3 is in the middle of?
Meanwhile, one of Consumer #3's 8 messages is addressed to Consumer #11, and it's actually quite important that Consumer #11 gets this one first instead of the one that it just found for itself
And so on...
Now, some (most? all?) of this kind of conflict can be resolved through the use of semaphores, but the penalty is that whenever any Consumer is accessing the shared queue, 100's of other Consumers are stuck waiting their turn. And so all the independence and parallelism you were after in the first place with the producer / consumer pattern starts slipping away...
I haven't personally built an app around a massively parallel messaging architecture. I hope someone who has gets involved here because I'd like to learn how it's done. I'd wager there are a few different elegant solutions, but all I can manage to think of are the pitfalls.
-Kevin P.

Similar Messages

Queue - 1 Producer and 2 Consumers

I look for structure Producer-Consumer but i like that my aquisition data posted in queue would be dequeue for 2 consumers (like copies of data).
In first consumer, the data would be analized e show in graphs. In second consumer, the data would be accumulated until 10 positions (staus queue = 10) and after this, the data would go to file in disk.
This would be the better solution?
I would can do it with 2 queues or have other way?
Thx for any clue...
Leonardo de S. Cavadas
Maintenance Engineer and Inspection - Bureau Veritas do Brasil
Engineer Metallurgist with emphasis in Advanced Materials
Technologist in Computer Science

Hello Jaime Rodriguez and Laemmermann,
Thx for 2 proposes, but i want that all data would be stored in disk and my consumer for show analysis can't limit the process. I think in use discard rotine for put the show loop very fast for dont full queue (example: IF queue full, discard next analisys of data(that include several rotines) for show...until queue status = 50%...is one simple analysis for show only). I will have other aplication for complete analysis the all data). Limit of queue is computer's memory.
My propose is in VI in attached (Labview 7.1).
Wait for your good clues. Thx for all.
Leonardo de S. Cavadas
Maintenance Engineer and Inspection - Bureau Veritas do Brasil
Engineer Metallurgist with emphasis in Advanced Materials
Technologist in Computer Science
Attachments:
queue_exemplo.vi ‏68 KB

IP_IN_QUEUE : Multiple Consumers

Hi,
This might be more of a AQ question but just wanted to check if anyone has had the requirement to use the IP_IN_QUEUE as a topic rather than a queue. i.e. use multiple consumers with the same consumer name receiving the same message. Is it advisable to create a custom topic as the point of entry as opposed to using the built in queue to forward to a topic.
regards,
Narayanan

Hi Narayanan,
Actually, IP_IN_QUEUE and IP_OUT_QUEUE are topics instead of queues because they can have multiple consumer names. The Document Routing ID is used to set the consumer name when B2B enqueues to IP_IN_QUEUE.
This sql statement is used to create the queue table.
execute dbms_aqadm.create_queue_table (queue_table => 'B2B.IP_QTAB', queue_payload_type => 'B2B.IP_MESSAGE_TYPE', multiple_consumers => TRUE);
Hope this helps,
Eng

Single queue: concurrent processing of messages in multiple consumers

Hi,
I am new to jms . The goal is to process messages concurrently from a queue in an asynchronous listener's onMessage method by attaching a listener instance to multiple consumer's with each consumer using its own session and running in a separate thread, that way the messages are passed on to the different consumers for concurrent processing.
1) Is it ossible to process messsages concurrently from a single queue by creating multiple consumers ?
2) I came up with the below code, but would like to get your thoughts on whether the below code looks correct for what I want to accomplish.
import java.io.IOException;
import java.io.InputStreamReader;
import java.util.Properties;
import javax.jms.Connection;
import javax.jms.ConnectionFactory;
import javax.jms.Destination;
import javax.jms.JMSException;
import javax.jms.Message;
import javax.jms.MessageConsumer;
import javax.jms.MessageListener;
import javax.jms.Session;
import javax.jms.TextMessage;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import com.walmart.platform.jms.client.JMSConnectionFactory;
public class QueueConsumer implements Runnable, MessageListener {
public static void main(String[] args) {
   // Create an instance of the client
    QueueConsumer consumer1 = new QueueConsumer();
    QueueConsumer consumer2 = new QueueConsumer();
    try {
    consumer1.init("oms","US.Q.CHECKOUT-ORDER.1.0.JSON");   //US.Q.CHECKOUT-ORDER.1.0.JSON   is the queue name
    consumer2.init("oms","US.Q.CHECKOUT-ORDER.1.0.JSON");
    }catch( JMSException ex ){
    ex.printStackTrace();
    System.exit(-1);
    // Start the client running
    Thread newThread1 = new Thread(consumer1);
    Thread newThread2 = new Thread(consumer1);
    newThread1.start();newThread2.start();
    InputStreamReader aISR = new InputStreamReader(System.in);
          char aAnswer = ' ';
          do {
              try {
aAnswer = (char) aISR.read();
catch (IOException e)
// TODO Auto-generated catch block
e.printStackTrace();
} while ((aAnswer != 'q') && (aAnswer != 'Q'));
          newThread1.interrupt();
          newThread2.interrupt();
          try {
newThread1.join();newThread2.join();
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
          System.out
.println("--------------------exiting main thread------------------------"+Thread.currentThread().getId());
        System.exit(0);
// values will be read from a resource properties file
private static String connectionFactoryName = null;
private static String queueName = null;
// thread safe object ref
private static ConnectionFactory qcf = null;
private static Connection queueConnection = null;
// not thread safe
private Session ses = null;
private Destination queue = null;
private MessageConsumer msgConsumer = null;
public static final Logger logger = LoggerFactory
.getLogger(QueueConsumer.class);
public QueueConsumer() {
super();
public void onMessage(Message msg) {
if (msg instanceof TextMessage) {
try {
System.out
.println("listener is "+Thread.currentThread().getId()+"--------------------Message recieved from queue is ------------------------"
+ ((TextMessage) msg).getJMSMessageID());
} catch (JMSException ex) {
ex.printStackTrace();
public void run() {
// Start listening
try {
queueConnection.start();
} catch (JMSException e) {
e.printStackTrace();
System.exit(-1);
while (!Thread.currentThread().isInterrupted()) {
synchronized (this) {
try {
wait();
} catch (InterruptedException ex) {
break;
* This method is called to set up and initialize the necessary Session,
* destination and message listener
* @param queue2
* @param factoryName
public void init(String factoryName, String queue2) throws JMSException {
try {
qcf = new JMSConnectionFactory(factoryName);
/* create the connection */
queueConnection = qcf.createConnection();
* Create a session that is non-transacted and is client
* acknowledged
ses = queueConnection.createSession(false,
Session.CLIENT_ACKNOWLEDGE);
queue = ses.createQueue(queue2);
logger.info("Subscribing to destination: " + queue2);
msgConsumer = ses.createConsumer(queue);
/* set the listener */
msgConsumer.setMessageListener(this);
System.out.println("Listening on queue " +queue2);
} catch (Exception e) {
e.printStackTrace();
System.exit(-1);
private static void setConnectionFactoryName(String name) {
connectionFactoryName = name;
private static String getQueueName() {
return queueName;
private static void setQueueName(String name) {
queueName = name;

Hi Mark,
if the messages are sent with quality of service EO (=exactly once) you can add queues.
From documentation:
Netweaver XI -> Runtime -> Processing xml messages -> Queues for asynchronous message processing
...For incoming messages you set the parameter EO_INBOUND_PARALLEL from the Tuning category.
If more than one message is sent to the same receiver, use the configuration parameter EO_OUTBOUND_PARALLEL of the Tuning category to process messages simultaneously in different queues.
Transaction: SXMB_ADM
Regards
Holger

Could multiple consumers attach to a JMS queue simultaneously?

Could a JMS message queue have multiple consumers simultaneously? If can, how to implement that? My experiences showed that at any time there is only one consumer allowed for a JMS message queue. If a subsequent consumer tried to attach to the same queue, the following exception would be thrown out:
=====================================================
javax.jms.ResourceAllocationException: [C4073]: A JMS destination limit was reached. Too many Subscribers/Receivers for Queue : PhysicalQueue user=guest, broker=CTPLIT015PC:7676(1462)======================================================
On the contrary, definitely multiple consumers can be attached to the same Topic.

Well I need tor receive a message only once and this is why I would still have to use a queue. I have heard selector are slow so I do not want to use them.
Now the problem is since I am using onMessage event; the message push is like 150 message/second. While the message dequeue process always stuck up at 14 message/second.
I really need to increase the speed some good number may be 50 messages/second. I need to know whether I need to setup a threadmanager (with threadpool) in onMessage event, or I need to create multiple queue session.
If I create multiple queue session what would be the behavior of queue?
My main concern is to get this number up.
regards
shantanu

Conditionally sending messages to multiple consumers

How would I best accomplish this requirement?
I have one system which produces messages
Multiple systems may want to consume those messages
I want the producer to be able to send messages conditionally
In one instance perhaps I send same message to each consumer
However in another case maybe I only want to send a message to consumers A and B
It would be OK if I could flag certain messages to be for certain consumers, and then those consumers only retrieve those messages designated to them. However I dont think this is possible as any such flag the consumer would have to read once it actually consumes the message and reads it, I dont think there is any way for the queue to make that decision for them.
So it sounds like what I need is a hybrid of point-to-point and publish-subscribe
I want a universal way to conditionally send messages to one or more consumers, and the conditions are dynamic.
Any suggestions?
Must it be a PTP with separate queue for every application?

lozina wrote:
Interesting... message filters. I like it.
So with this approach, I can send some messages to everyone, and other messages I can have tailored to specific subscribers?pretty much.
But the subscriber does not actually have to "fetch" each message to determine if the message is meant for him or not, right? When using message filters this act at the queue, not at the client? Basically I want to avoid a client downloading 1,000 messages only to find out none of them are for him. unfortunately, that's probably up to the implementation. the implementation may choose to work that way (download every message and check it), or it may have a way of applying the filter at the server. however, the filters work only on the message properties, so the impl may be able to just get message headers and filter w/out getting the entire message. you'll have to test with the jms impl you are planning on using and see how that works.
Thanks!
[Edit] these are called Message Selectors right? yes.
And looks like they are specified at the time the client looks up the queue in JNDI?yes, these are provided at subscription time.
Now is this limited to publish-subscribe?no you can use selectors when subscribing to a queue as well. but, queues won't scale very well to lots of consumers (and are hard to manage with dynamic numbers of consumers).

Configuring queue for Producer - first-available Consumer delivery

Hello,
Setting
consumerFlowLimit to 1
consumerFlowThreshold to 0
doesn't appear to be enough to ensure that messages from a queue are delivered to the first available consumer.
We are trying to implement a classic Producer(s) -> Consumer(s) job distribution scheme where our jobs are lengthy and vary in duration.
It's important that msgs are sent to the next available Consumer and not waiting/buffered at a busy Consumer.
What we see with the settings above is that more than one message appears to be buffered at each consumer when it starts. After the startup buffering, the settings appear to be working as intended.
We are using openmq 4.3, but happy to move to 4.4 if required.
This question was posted earlier to the users list, but I've posted here with elaboration. Thanks!
Steven Marcus

This is a critical bug for us. Can we raise the priority of fixing this?
With the current implementation, openMQ guarantees to deliver messages out-of-order when using multiple consumers.

Producer to N Consumers notification

Hi,
I have the following situation (see also attached vi, made with LV2009SP1):
1) one producer that is used as a "commands generator" for the consumer threads
2) three consumers threads, each of them makes a particular task (let's say they checks something, each with different execution time)
How can the producer have the replies (in term of error cluster) from the three parallel threads?
Once I enqueue the command in the queues, each thread can work in parallel, but how can it report back?
Is the producer-consumer a pattern with intrinsic "mono-directional" communication from producer to consumer(s)?
Maybe one solution would be to avoid "producer consumer" and simply put three subVI in parallel, and then collecting their error outputs (with "find first error"). But this solution cannot update the LED controls that I want in the "main" VI.
Any advice would be appreciated.
thanks
Attachments:
Producer-MultiConsumer-Test.vi ‏27 KB

Here is a rough idea. Use just one queue for producer to consumers direction and one queue for consumers to producer direction. The queue element can be a cluster with one element being the consumer number (1, 2, or 3), and another element being the command. The producer queues up the command with the proper consumer number (use a different number to broadcast to all consumers, like 255). Each consumer previews the queue. When it sees its ID, it dequeues that element. Extra coding is need to handle the broadcast case. Some type of notifier to let the last consumer know to dequeue the broadcasted command.
In the other direction, same principle. Each consumer puts its ID and response in a cluster to be queued. The producer dequeues and acts appropriately.
Upon stop, the loops will have to be written so that they can be interupted at any time so that you don't wait on the slowest loop to complete before shutting down. While loops with a check for the shutdown command, or For Loops with the break condition enabled. For this, maybe a separate notifier could be used.
A wild idea with much coding, but I believe it could be made to work for your purpose. Is it better than having three separate queues (or 6 queues for both directions)? I don't know. It may be worth a try.
- tbob
Inventor of the WORM Global

How can I clean up queue one-time on Sun Java system messaging 6.3

Hi,
<address>Our Email server have a problem, When I run _./imsimta qm directory tcp_local_, There are about 5 Millions of messages in the queue.</address>
Now our Email server send messages very slowly, how can I clean up queue one-time?
the command _./imsimta qclean_ is very slowly.
What can I do to prevent this problem?
our Email server version is :
Sun Java(tm) System Messaging Server 6.3-6.03 (built Mar 14 2008; 32bit)
libimta.so 6.3-6.03 (built 17:12:37, Mar 14 2008; 32bit)
SunOS email-1 5.10 Generic_120011-14 sun4u sparc SUNW,Sun-Fire-V890
Thank you !

If you have more than 100,000 messages in the queue, then look at the MAX_MESSAGES parameter in [the job_controller.cnf file|http://wikis.sun.com/display/CommSuite/Job+Controller+Configuration+File]. If the parameter is not specified, it defaults to 100000. If you have more than that number of messages in the channel queues, it will take a long time for new/legitimate messages to be sent because job_controller is only considering the first 100,000 messages in the queue.
If you get into the "imsimta qm" command do do "sum -database", it will show a summary of what job_controller has in its internal cache and therefore what job_controller is working on. If you compare that to "sum -directory" you will probably see a difference.
If these are all legitimate messages, you need to increase MAX_MESSAGES, cnbuild, and restart job_controller.
If they are not, then preventing this in the future will require determining how they got into your queue and correcting that.
As for removing them, the "imsimta qm" commands allow you to select messages by various criteria and then you can "return" or "delete" them. But yes, that will take a long time if there are many messages because it is a single threaded process and it needs to open and examine each message. You may be able to accomplish the task more quickly with a shell script that works on individual channel queue subdirectories and then run multiple copies of that script in parallel. After you have cleaned out the queue, restart job_controller.
Also, you might want to consider [the subdirs channel keyword|http://msg.wikidoc.info/index.php/Subdirs_and_nosubdirs_Channel_Options].

Could Buffer replace the Queue in Producer/Consumer Design Pattern

Hello,
I have a question that the task of Buffer is to store the data and the queue is also of the same so could we use the Buffer inplace of queue in a Producer/Consumer Design Pattern.
Solved!
Go to Solution.

No, those buffer examples are not nearly equal to a queue and will never ever "replace" queues in producer/consumer.
The most important advantage of queues for producer/consumer (which none of the other buffer mechanics share) is that it works eventbased to notify the reader that data is available. So if you would simply replace the queue by overly elaborate buffer mechanics as you attached to your last post, you will lose a great deal of the the purpose using producer/consumer.
So, to compare both mechanics:
- Queue works eventbased, whereas the buffer example does not.
- Queue has to allocate memory during runtime if more elements are written to the queue than dequeued. This is also true for the buffer (it has to be resized).
- Since the buffer is effectively simply an array with overhead, memory management is getting slow and messy with increasing memory fragmentation. Queues perform way better here (but have their limits there too).
- The overhead for the buffer (array handling) has to be implemented manually. Queue functions encapsulate all necessary functionality you will ever need. So queues do have a simple API, whereas the buffer has not.
- Since the buffer is simply an array, you will have a hard time sharing the content in two parallel running loops. You will need to either implement additional overhead using data value references to manage the buffer or waste lots of memory by using mechanics like variables. In addition to wasting memory, you will presumably run into race conditions so do not even think about this.
So this leads to four '+' for the queue and only one point where "buffer" equals the queue.
i hope, this clears things up a bit.
Norbert
CEO: What exactly is stopping us from doing this?
Expert: Geometry
Marketing Manager: Just ignore it.

How to get one value from multiple duplication for a key figure field

Hi expert,
      I have a infoprovider, with following format:
        employee     hourly_rate   action_type count of action
     there are multiple rows for each employee, I want to create query as follows:
        employee     hourly_rate   action_type count of action
     in which hourly_rate is constant , only retriving one value from multiple rows for one employee, count of action should be summarized.
how to get this hourly_rate.
Many Thanks,

Hi,
put the employee in rows panel -> reaming object put it in free char panel.
suppress the all result rows for the all object except employee .
select the object -> go to query properties -> select display tab -> select result rows -> select suppress.
select the employee -> go to query properties -> select display tab -> select result rows -> select always display. - now it will give employee wise hourly rate summarize data.
Thanks,
Phani.

How do i take one layer with multiple selections...

how do i take one layer with multiple selections and make those selections their own layer?

Yes sir. It w was really hard to explain. It is discontiguous segments that are separated by transparent areas. And I want those non transparent areas to be on their own layer. I have a photo shop file that has a bunch of buttons on them and they are on a transparent background. It is one layer. I want to select all the buttons and put them on their own layers. I can do this manually by selecting a button and cmd Just to a new layer, but when I have 200 buttons it is a daunting task.
Sent by MailWise<http://www.mail-wise.com/installation/4> – Your emails, with style.

HT1206 Lots of info about one user using multiple computers. What about multiple users with separate Apple IDs using same computer? Having problems getting my wifes new iPhone talking to her apple account on the computer we share (2 users)

Lots of info about one user using multiple computers. What about multiple users with separate Apple IDs using same computer? Having problems getting my wifes new iPhone talking to her apple account on the computer we share (2 users)

You need to create a user account for your wife (or yourself depending on who has the current user account). When syncing, each of you should sign in as a separate user, login to iTunes and then sync. I had this problem when my sister got an iPhone. When we did her initial sync, everything on my iPhone showed up on hers. Apple gave me this solution.

Can i install itunes multiple times on one computer for multiple users?

Can i install itunes multiple times on one computer for multiple users?

Like most applications you only need to install it once for it to be available to all users. (That said it is possible to install an application and only make it available to the current user, but that's rarely needed.)
tt2

IDVD burning one project on multiple macs? where are project contents?

How do I burn one project on multiple macs? I have a idvd project I created in keynote, added music to it in Garageband, exported to quicktime, and now is in iDVD burning just fine. However I have tons of these dvd's to make, I would like to put all my mac's to work.
last night i tried to share the file, and it is just an alias when i drag it over from the macbook it was created on to my imac. the file says it is in use and locked, since the macbook that created it is burning it. If I wanted to brun on the imac, I had to wait until the macbook was done.
is there a way around this? What if I copied all files I used to make the idvd project to my imac or any computer i wanted to burn on? woudl it hen allow two macs to be burning the same project?
I figured I could do this just like on my PC, drag the file over and voila...no such luck, its like if i had dragged a shortcut over to another pc...if the first one is using the file, its locked. Basically how do i get to the actual contents of the project, so I can also bring them over and hopefully it will allow me to burn on multiple macs at the same time?
my next solution if that didn't work was to start copying dvd's using popcorn on the other macs while the macbook continues burning through idvd.
Any opinions on how i can duplicate a dvd fast...need alot of copies!

Also as to where the files are.
I-DVD project bundle: This is the file created when you save the project.
• [user]/Library/iDVD/Favorites: This folder is located in your home folder.
• [root]/Library/iDVD/Favorites: This folder is located at the root of your disk.
• /Network/Library/iDVD/Favorites: This folder is on a network system specifically
configured by your system administrator for this purpose. (This is not the same as
having a server networked into your computer.)

Queue: one producer vs multiple consumers

Similar Messages

Maybe you are looking for