Queue efficiency

Question

So I'm working on improving the efficiency/speed of my producer-consumer system.
I have two producers and one consumer (which among other things, writes files to the disk).
I currently have both producers feeding the one queue which is then processed by my currently single consumer.
Each queue item is a cluster of multiple data types and contains everything needed to "consume" it.
There are certain cases where more complex queue items get placed which take longer to process and slow down the consumer process.
I was thinking of adding a second consumer loop to run in parallel with the first to take the weight off the single loop so to speak.
My question is, would it be more efficient to have both consumers dequeueing from the same queue, or would it be more efficient to have each producer feeding its own queue. For the sake of the exercise assume that I can guarantee that the complex queue elements will only come from a specific producer.
Solved!
Go to Solution.

Answer 1

Hornless.Rhino wrote:
So I'm working on improving the efficiency/speed of my producer-consumer system. I have two producers and one consumer (which among other things, writes files to the disk). I currently have both producers feeding the one queue which is then processed by my currently single consumer. Each queue item is a cluster of multiple data types and contains everything needed to "consume" it. There are certain cases where more complex queue items get placed which take longer to process and slow down the consumer process. I was thinking of adding a second consumer loop to run in parallel with the first to take the weight off the single loop so to speak. My question is, would it be more efficient to have both consumers dequeueing from the same queue, or would it be more efficient to have each producer feeding its own queue. For the sake of the exercise assume that I can guarantee that the complex queue elements will only come from a specific producer.
From a performance perspective, the Queue primitives won't care if you create two queues or use one queue in two consumers. To have multiple slaves (consumers) processing from one queue is perfectly acceptable, so long as the order of consumption is not important.
If you create management code to hand off the 'complex' jobs to a dedicated secondary consumer then you are not taking full advantage of the multiple-slave framework. It all depends on the ratio of simple-to-complex jobs, and also the time it takes to complete the jobs. For example, if the complex jobs come in once every 10,000 simple jobs, then a dedicated second consumer for the complex jobs will be largely idle and therefore under-utilised. If however, the ratio is more like 1 complex job for every two or three simple jobs, then you could find the primary consumer is largely under-utilised.
The best balance is to allow both consumers to dequeue all job types and therefore both be working at maximum capacity.
Thoric (CLA, CLED, CTD and LabVIEW Champion)