Best way to handle pessimistic locking after dequeueing

Question

Dear all,
we have a simple single-consumer scenario where multiple worker
processes listen to one task queue. Every task is specific to exactly one business
object that is pessimistically locked (SELECT FOR UPDATE) as soon as
the AQ message has been fully parsed by a worker.
Obviously, this may lead to unnecessary waiting. For example, when
there are 5 successive messages belonging to the same business object,
having them dequeued by 5 worker processes in parallel will result
in 1 worker processing its task and 4 workers waiting.
To me, this seems to be a common use case. How is it "usually"
handled?
Best regards,
Michael

Answer 1

Hi Paul!
Thanks a lot for your answers that would completely solve
my problem if I had another setting :-) . I try to clarify
things by answering your questions (slightly reordered).
I have to apologize in advance that I take my description to the
next level of detail and hope not to deter you too much.
3. Why is this a single consumer Q when you have multiple business
areas hanging off it? It would seem an ideal candidate for having
a subscriber per business area and then subsequently a worker per subscriber.Instead of a multi-subscriber setting, we employ multiple
queues for different business areas and do some "queue routing"
in our middle-ware. We wanted the different areas to become as
detangled as possible (maybe that was unnecessary)
2. Why lock the database object before processing the message?
Cant you just rely on oracle's internal row locking mechanisms?
Do you lock the whole table? If so why? First of all, we only lock one row that designates the "coarse-grained lock".
I think you will understand this with the description below.
We have one component that orchestrates our workflows and updates the
states of our business objects. Let's assume, our basic "business object container"
was called 'stream'. Different streams are guaranteed to be independent (and hence may be
concurrently processed). So, a SELECT FOR UPDATE on a stream is enough
to pessimistically lock the whole business object.
Such a stream contains a complex graph of sub-objects. When the response
of, e.g. a 'youtube syncer' component, comes in, the state of many
objects in this graph may change (which we obviously want to do within a transaction).
E.g. when all children of a node become READY,
this node becomes READY which, in turn, may yield its
parent READY.
The problem with ordinary row-locking is now: responses for one stream tend to cluster,
i.e. sometimes there are 100 responses in the queue for one stream, and as state
updates often have large consequences for the whole state graph, blocking
(or NOWAIT exceptions) are the rule not the exception, resulting in performance that
is even worse than coarse-grained locking.
1. Why have multiple processes DQ'ing from this Q?
Is the message volume too great for one worker?Some tasks are long-running like transferring large files to remote hosts.
With workers running on different hosts and networks we have nice
load balancing.
Even the short-running jobs take often >1s CPU time and batch
jobs may generate several millions of AQ messages that we process
in a cluster.
Ok, if you reach this line, you certainly deserve my thankfulness :-)
Best regards,
Michael