Factory Finder failure and _tmmsgrcv() SEGV

Question

Hello,
we have a rather strange problem with WLE 5.1. One type of process, QD,
occasionally experiences a SEGV inside _tmmsgrcv(). The SEGV is usually
preceeded by a failure from the factory finder.
QD is a process that converts /Q messages into Croba requests. TMQFORWARD is
used to get the messages out of the /Q, and to invoke the Tuxedo service in
QD. QD then finds a Corba object that can handle the operation and invokes
the operation. There are multiple instances of TMQFORWARD, QD and CorbaApp
for the same service/object.
The functionality of QD is roughly this:
try {
f = find_on_factory_by_id( name);
if (CORBA::is_nil( f)) {
log( "no factory found");
return RETRY;
else {
h = Obj::_narrow( f);
if (CORBA::is_nil( h)) {
log( "narrow failed");
return RETRY;
catch( ...) {
log( "find factory failed");
if (tooManyFactoryFailures)
assert();
return( RETRY);
try {
h->ping(); // no-op to check that object is available
catch( INVALID_TRANSACTION) {
log( "object is busy");
CORBA::release( h);
return( RETRY);
catch ( ...) {
log( "ping failed");
CORBA::release( h);
return( RETRY);
try {
h->doOp( msg);
catch( ...) {
log( "operation failed");
CORBA::release( h);
return( RETRY);
CORBA::release( h);
return SUCCESS;
The outer function will call tpreturn indicating success, retry or abort.
Now, most of the time all is ok and this works for days. But once in a while
there is a problem.
The outcome is either an assert, because most attempts to find the factory
fail (there still are a few successes in between), or a SEGV on getting the
next message to be handled by the QD (it's a different message, not a
retry).
#0 0xfbaa4274 in _tmmsgrcv () from /opt/wle/lib/libtux.so.65
#1 0xfbad4c1c in _tmrcvrq () from /opt/wle/lib/libtux.so.65
#2 0xfbad6b58 in _tmrunserver () from /opt/wle/lib/libtux.so.65
#3 0xfbaba140 in _tmstartserver () from /opt/wle/lib/libtux.so.65
#4 0x4346c in main (argc=19, argv=<incomplete type>, N=<error type>) at
BS-6950.c:74
The failure scenario starts with a factory failure (log "find factory
failed", exception TRANSACTION_ROLLEDBACK). QD returns retry, and the
message remains on the /Q (with a delayed retry).
After that the behaviour is not consistent.
Sometimes QD crashes (SEGV) on receiving the next message.
Sometimes QD seems to succeed processing the next message (no errors logged,
no exceptions thrown), but the Corba Object never gets the message.
Sometimes QD fails again to get the factory, but the exception is either
INTERNAL or BAD_PARAM.
The above may be repeated in any combination until the final outcome (assert
or SEGV).
In one case (but only one) there was a LIBTUX_CAT:518: ERROR.
In some cases there is a LIBTUX_CAT:489: ERROR.
My suspicion is that there is a memory corruption somewhere. I have reviewed
our code many times and have not been able to find anything. I have also
searched BEA and the web, but have not found anything relevant.
Any suggestions?
Roger

Answer 1

Hi Todd,
thanks for your reply.
I started to answer your individual questions, but then we had a different
crash (in tmfmsgfree). I submitted that to BEA, and they found a known fix
in 8.1. Hopefully they will port it to WLE 5.1.
Have you tried running Purify or a similar tool to see if that helps?Yes, some time ago to fix a memory leak. At that time I didn't look
specifically for this problem, but anything unusual I would have noticed.
Also, have you been able to create a consistent reproducer?We can not reproduce it. On our performance test bed it happens about once
every other day (with 132 QD processes handling 3.5M messages). At the
customer site it happens two or 3 times a day, even though they have only
about 2M messages per day.
BEA Support can help youWe created a case and submitted a stack trace and core dump. They did not
find anything unusual.
but no releases were evident for your factories.That was my mistake in presenting the simplified code.
I'm also puzzled by your tooManyFactoryFailures assertion.Before I started working on the project, it was noticed that once in a blue
moon a QD process would enter a state where it would experience many factory
finder failures. No cause was found and the Assert was added to recover from
this. It was an early indication of the current problem. Since then the
factory failures have become more frequent and more severe (SEGV). That is
why I am looking at it again.
is there a reason you are hitting the factory finder each time?Again it is historical. I believe it was done to better do load-sharing.
Roger