Apache2 + mod_perl2 + dbxml issue

Question

Hello,
we are experiencing an annoying issue with the setup mentioned in the subject, and since we haven't been able to find a solution for a while now, we're seeking for help in this forum.
We have a web application built as a couple of perl modules, based on apache2 and mod_perl2 as our "framework". The application is spread across multiple virtual hosts, the virtual hosts load different perl modules. Upon apache's startup, each of the virtual hosts loads our app modules, and our perl modules in turn load Sleepycat::DbXml. The result is that within each virtual host, the dbxml env is opened once.
For those less familiar with modperl, mod_perl2 allows for a quite detailed configuration, regarding how the (potentially multiple) perl interpreters should be used per virtual host. The purpose of the different available conf. directives is to have an automatic and dynamic utilization of system resources, depending on the server load.
If configured appropriately, apache/mod_perl will spawn (clone) new perl interpreters
when there are more concurrent requests, and also automatically unload (destroy) perl interpreters when too many become idle.
Now, everything is fine if the mod_perl virtual host settings are set to a constant number of perl interpreters during the whole apache lifetime.
But the issue begins as soon as we allow for dynamic spawning/unloading of perl interpreters. We noticed that as soon as one perl interpreter is destroyed, we get a crash of the httpd child process with the message "illegal instruction", caused by a berkeley db / dbxml module.
Moreover, after such a crash the rest of the virtual hosts (perl interpreters) cannot perform any read or write operation on the dbxml. :(
(Below the backtrace from gdb.)
Does this sound familiar to anybody ?
Does anybody understand what happens here ?
(In case the above problem description was not clear enough, I'll be glad to provide more details)
Regards,
martin.
----------->
Program received signal SIGILL, Illegal instruction.
[Switching to Thread 7 (LWP 5)]
0x052da038 in ?? ()
(gdb) bt
#0 0x052da038 in ?? ()
#1 0xfe77c3d4 in DbXml::Manager::~Manager (this=0x4f0d778) at Manager.cpp:128
#2 0xfe74b288 in DbXml::ReferenceCountedProtected::release (this=0x4f0d77c) at ReferenceCounted.cpp:58
#3 0xfe91dd14 in XS_XmlManager_DESTROY ()
from /xxxxx/perl/lib/site_perl/5.8.7/sun4-solaris-thread-multi/auto/Sleepycat/DbXml/DbXml.so
#4 0xfe0f5f0c in Perl_pp_entersub () from /xxxxx/apache2/modules/mod_perl.so
#5 0xfe091480 in S_call_body () from /xxxxx/apache2/modules/mod_perl.so
#6 0xfe0911e4 in Perl_call_sv () from /xxxxx/apache2/modules/mod_perl.so
#7 0xfe0ff390 in Perl_sv_clear () from /xxxxx/apache2/modules/mod_perl.so
#8 0xfe0ffbe8 in Perl_sv_free () from /xxxxx/apache2/modules/mod_perl.so
#9 0xfe0f7270 in do_clean_objs () from /xxxxx/apache2/modules/mod_perl.so
#10 0xfe0f71bc in S_visit () from /xxxxx/apache2/modules/mod_perl.so
#11 0xfe0f73c8 in Perl_sv_clean_objs () from /xxxxx/apache2/modules/mod_perl.so
#12 0xfe08e158 in perl_destruct () from /xxxxx/apache2/modules/mod_perl.so
#13 0xfe081fb0 in modperl_perl_destruct () from /xxxxx/apache2/modules/mod_perl.so
#14 0xfe073fa4 in modperl_interp_destroy () from /xxxxx/apache2/modules/mod_perl.so
#15 0xfe0740a8 in interp_pool_shrink () from /xxxxx/apache2/modules/mod_perl.so
#16 0xfe0750ec in modperl_tipool_putback_base () from /xxxxx/apache2/modules/mod_perl.so
#17 0xfe0751a4 in modperl_tipool_putback_data () from /xxxxx/apache2/modules/mod_perl.so
#18 0xfe074254 in modperl_interp_unselect () from /xxxxx/apache2/modules/mod_perl.so
#19 0xff2c327c in run_cleanups (cref=0xa067c48) at memory/unix/apr_pools.c:2034
#20 0xff2c29f8 in apr_pool_destroy (pool=0xa067c38) at memory/unix/apr_pools.c:727
#21 0xff2c29c0 in apr_pool_clear (pool=0xa030fc8) at memory/unix/apr_pools.c:686
#22 0x000631e0 in worker_thread (thd=0x889870, dummy=0x0) at worker.c:897
#23 0xff2cb3a4 in dummy_worker (opaque=0x889870) at threadproc/unix/thread.c:138
#24 0xfeebb024 in threadstart () from /usr/lib/libthread.so.1
#25 0xfeebb024 in threadstart () from /usr/lib/libthread.so.1
<-----------

Answer 1

Hello,
okay - we have finally understood the problem, and so I want to share the results here... maybe it helps someone who wants to use Apache/mod_perl together with dbxml.
The key to understand the issue is to understand how mod_perl works with the perl interpreters. Important to know: there is a "parent" perl interpreter, that is spawned first.
(either one per httpd process, or - like in our case - one parent per virtual host).
Then additional interpreters are "cloned" from the "parent" perl, until the configured total number of perl interpreters is reached. As mentioned in the first posting below, the total number of perl interpreters can be either configured constant, or can be configured dynamic. The "parent" perl interpreter is created already during Apache's startup phase;
in case of the "dynamic configuration", the additional perl interpreters can be "cloned" or destroyed during http request time.
Now, the big mistake that we made, was to have the perl code handling the opening of dbxml env/manager/container already on the "top" level of our perl module. Thereby the code was of course executed as soon as the first perl interpreter was loaded, i.e. when the parent interpreter was loaded. Then this "parent" perl was "cloned" and so the dbxml object handles were cloned too. Later, when mod_perl decided to destroy one of the "cloned" interpreters, as it was not needed anymore, all the object destructors were called implicitly, this way also the dbxml objects. We had not expected a problem here. We assumed the "cloning" process would have properly taken care of duplicating the dbxml "object handles" (whatever that means :) but obviously that's not the case. As soon as one of the "cloned" perl interpreters is destroyed, the dbxml env/mgr/container variables are destroyed and that takes to death the other perl interpreters and the whole httpd process.
So... once understood, the solution for our problem was quite simple: To move the dbxml env/mgr/container opening away from the module top, and move it into a subroutine, that is called only if demanded. I.e. the dbxml read/write code first checks if the dbxml env. is already opened, and if not it is opened. That's a little bit of overhead, but it needs to happen only once per perl interpreter lifetime. Then the opened env. is reused all the time.
So in other words, the dbxml env/mgr/container handles are no more created in the "parent" interpreter, but only within the "cloned" interpreter. Whenever the "cloned" interpreter needs to be killed, all its objects can be happily killed too, including dbxml, but that's no problem anymore, because it does not affect the "parent" perl or the sister "clones" at all.
That's the whole story...
Regards,
martin.