"Stuck" remote messages

Question

We have our monitoring software setup to query and chart the "Actual" number of messages (as reported by "imqcmd query bkr" and recently noticed that this number is remaining at a higher minimum value. Meaning, it never drops to 0 as though there is a pile of messages that cannot be processed and they are sitting in the queue
Our setup is a cluster of two brokers (not HA) and the cluster.properties file looks like this:
cluster.properties
imq.cluster.brokerlist=hobgoblin,smithwicks
imq.cluster.masterbroker=hobgoblinWe have 10 queue destinations varying in volume from dozens to hundreds of thousands of messages a day. Three of these queues (all relatively high volume) exhibit this issue. I will use a particular queue named "networkActivity" in the following commands.
When I run imqcmd query dst -t q -n networkActivity on hobgoblin (the master broker) the relevant piece of output is:
Current Number of Messages           
    Actual                            75
    Remote                            75
    Held in Transaction               0When I run the same on smithwicks it is:
Current Number of Messages           
    Actual                            284
    Remote                            284
    Held in Transaction               0The "Actual" number will occasionally increase when we are under heavy load but as we catch up on the load that number will only decrease to match the "Remote" number.
The other 7 queues do not have this issue and some of those do equally heavy volume. New messages continue to be processed fast on all queues.
What does this Remote number mean? I have struggled to find information in the documentation or online.
We recently switched to OpenMQ and it has been utterly stable and performant compared to our previous provider (where queue lockups and shutdown due to slow consumers was commonplace). Very happy with it in general. We are planning to move to an HA setup in the near future - would this issue still occur when using HA?
These "Remote" message counts are relatively small but they do seem to drift upward a small amount each day visually establishing a new baseline in our monitoring charts.
Any help is greatly appreciate. I can easily provide any other information that would be helpful - logs, imqcmd output, server information, etc.

Answer 1

I am surprised I found something not obvious in the docs because frankly the OpenMQ documentation is some of the best I have ever run into.
Fortunately/Unfortunately we don't have this issue any longer so I cannot collect more information. The remote counts dropped to zero when I bounced OpenMQ one day. If that means we lost these particular messages it isn't a big deal for us.
Our previous provider (ActiveMQ) had literally millions of backed up messages when we migrated to OpenMQ. A colleague here wrote a tool that pulled the messages off the AMQ queues and fed them to OpenMQ (OpenMQ proceeded to shred through those things) all at once and the remote number happened after that huge crush of old messages. I do not know if that is related at all but I am willing to chalk it up to a possibility as some of those AMQ messages were really old. There could have even been messages so old that our consumer logic didn't know how to handle them anymore.
Have had no issues since and we switched to an HA configuration this morning and everything is running perfectly.