Is there a bug in 10.1.2.0.2?

Question

We are experiencing a strange issue which we have a service request raised for, but I thought I would open this up to the BPEL community to see if any other customers using the 10.1.2.0.2 BPEL Engine on Windows 2003 with a 10g dehydration store are experiencing the same problem.
We are getting messages placed on the manual recovery queue with no explaination. Generally we see items end up on the manual recovery queue for a number of reasons :
1. JVM crash
2. JTA timeout
3. Database connection issues
4. Lack of available threads
etc
Everytime we get an item end up on the MRQ we get an entry in the
domain.log explaining why BPEL had a problem and so it is explainable.
However, in certain circumstance we get messages ending up on the MRQ which
has no corresponding entry in the domain.log (or any log). In these cases we
have no explaination as to why this occured.
We are running logging levels of INFO, but even setting the logging level to
DEBUG on the default.collaxa.cube.engine.dispatch log has failed to show any
evidence of why this is occuring. Neither has setting all logging to DEBUG shown up anything.
Investigations fail to show any reason why this would be the case. On the most recent occasion this occured we had less than 5 processes running at the time one invoke ended up on the MRQ. We have an active thread depth of 100
which according the BPEL Admin console is never exceeding 30), so this is not occuring when there is any sort of significant load, in fact it seems to be the opposite, as it seems to occur when there is little or no load.
When it placed the message on the MRQ I would have expected an entry in the domain.log, but there was NO entry in the logs. I also search the entire file system
for any other file that changed at that time and thus may give some clue, but
nothing...
So, the question is what can be causing an item to end up on the MRQ that does
not have a corresponding entry in the domain.log? In fact, what can be causing
the JMS message that processes the message to be lost with no evidence in the
logs?
It looks clear to me that BPEL is having a problem with something and failing to write a log entry for it. The issue is non-deterministic and hence I would like to know if anyone has any ideas.
Regards
Pete

Answer 1

Clemens
We have an SR raised which is now with development, but so far we have not been able to pin down what is causing this.
Basically we see entries end up on the recovery queue with nothing written in any logs, even if we increase the level of logging to debug.
This does not happen under load, the exact opposite. If nothing is going on and then we get 5-10 processes start, we see about 1% end on the queue. I pushed 500,000 messages through it last weekend and nothing ended up on the queue, yet a run on Monday of just 100 caused 2 entries on the queue with no explaination.
I have tried everything I can think of, but so far have not been able to re-create the condition at will, it is completely non-deterministic. It also happens on all 5 of our rigs, so it is not machine specific.
I have tuned the BPEL engine and it easily copes with our throughput. We are running with a 10g dehydration store, either single instance or active-passive depending on environment, but I do not think this is going to be causing an issue.
We have about 20 async processes and sub-processes and the problem can occur at any point, although mostly on Invoke and Callback. Each process contains a mixture of async (to BPEL), sync (to External) and a reasonably number of database adapters in each process.
It is not the fact it is placing the entry on the queue that is my issue, it is the fact we have no idea WHY it has ended there as there are no log entries.
Any ideas what else I can try?
Pete