Broker not delivering messages

Question

Hi,
I have a system which has been working fine for a year and is now causing some issues. It is running OpenMQ 4.2 with Java 1.6 (1.6.0_11-b03) on Solaris (SunOS 5.10 Generic_138888-07 sun4v sparc SUNW,Sun-Blade-T6340). No changes have been made to the configuration or the system recently.
The issue is:
1- A producer sends a bunch of messages (around 100) in a queue
2- A consumer listening to the same queue picks up say 96 of the messages and leaves 4 of them in the queue (4 random messages, not in sequence or first or last)
When I try to read these messages left on the queue using a separate script (different JVM than main software) which just connects to the queue and dumps its content on the screen, the call to consumer.receive() does not return any message. Yet listing the queue content (with "imqcmd list") actually shows the 4 messages in the queue and the consumer attached to it :/
When I restart the broker then the 4 messages are delivered correctly and processed by the application.
The broker log (set to default INFO) doesn't show any error or unusual trace, my application logs either.
This problem happened the first time after the broker had been running for 3 months in a row so we though it would probably not happen again if we restart it every month. Yet it now happened again after only 20 days since the last restart of the broker.
Another side problem: when I tried to dump the JVM content using "kill -3" I could not find it in the broker log (set using -Dimq.log.file.filename=broker.log on the command line starting the broker), any idea of how I should proceed?
Thank you very much for any help you could provide
Mickael

Answer 1

I agree with TravelEntity.
The issue is:
1- The producer (only one) sends a bunch of messages (around 100) in a queue
2- The consumer (only one) listening to the same queue picks up say 96 of the messages and leaves 4 of them in the queue (4 random messages, not in sequence or first or last)The basic behaviour here is that the moment this consumer is created (and the connection started) the broker will send all 100 messages to the consumer (assuming you're not using a message selector), where they will be cached in the client until they are delivered to the client application. Whilst they remain in the client they will be unavailable to other consumers. When the first consumer closes, any unconsumed messages will be returned to the broker and become available for delivery to other consumers. The number of messages that are "pre-fetched" to a consumer is defined by the consumerFlowLimit of the connection or destination (see MQ admin guide for details). The default is 1000.
So this appears to explain why when you created a second consumer, it didn't receive any messages despite there being four messages left on the queue.
However I don't understand what you mean by "leave 4 random messages in the queue". How were you doing that? Messages are consumed from the queue in order - unless you use a message selector.
3- The producer sends another bunch of messages in the same queue
4- The consumer picks up say 98 of the messages and leaves 2 of them in the queue
5- mqlist then shows 6 messages in the queue, consumer.receive() returns null if tried from the main client or from a separate queue dump tool What is mqlist? It's not a MQ tool.
You can use the MQ command imqcmd to report on how many messages are being cached on the client in this way: in the following sample of its usage, "Count" is the total number of messages on the queue, and "UnAck" tells you how many of those messages are being cached on the client.
C:\mq\bin>imqcmd list dst 
   Name     Type    State      Producers        Consumers                  Msgs
                            Total  Wildcard  Total  Wildcard  Count  Remote  UnAck  Avg Size
INPUT.Q     Queue  RUNNING  0      -         1      -         9268   0       4709   1237.2805
mq.sys.dmq  Queue  RUNNING  0      -         0      -         0      0       0      0.0