Perl API: growing memory problem in loops over large sets of data

Question

Hi,
When going through all XmlResults like this:
while ($results->next($val)) {
print $val->asString, "\n";
The process size keeps growing. It does not when I comment $val->asString method out, but then I have no way of getting the results.
This becomes a significant problem when the number of results is huge. I am doing this on a database of over a million short XML documents (400-800 bytes each).
The more complete code is here:
eval {
$env = new DbEnv();
$env->open($dbDir, Db::DB_JOINENV | Db::DB_INIT_LOCK
| Db::DB_INIT_MPOOL | Db::DB_CREATE, 0);
my $mgr = new XmlManager($env, DbXml::DBXML_ADOPT_DBENV);
my $db = $mgr->openContainer(undef, $dbName, Db::DB_RDONLY);
my $context = $mgr->createQueryContext(XmlQueryContext::LiveValues,
XmlQueryContext::Lazy);
my $lookup = $mgr->createIndexLookup($db, "", $nodeName,
"node-$nodeType-equality-$syntax",
new XmlValue($types{$nodeType}, $value), XmlIndexLookup::GTE);
my $results = $lookup->execute(undef, $context);
my $val = new XmlValue();
while ($results->next($val)) {
print $val->asString, "\n";
if (my $e = catch std::exception) {
die $e->what() . "\n";
The process size just grows until the system limit is reached, then the process quits saying 'Out of memory'.
I suspect the problem is with the std::string result returned by C++ XmlValue::asString() const.
The (left-hand-side) result string is likely allocated by new std::string and receives the value by calling the string copy operator. Then the Perl scalar result is prepared, but when it gets returned to my code, the C++ string is not deleted.
Moving the Sleepycat::XmlValue Perl object inside the loop does not help either:
while ($results->hasNext()) {
my $val = new XmlValue();
$results->next($val);
print $val->asString, "\n";
In fact, the process seems to grow faster, possibly because the old $val instances do not get destroyed by Perl at the end of the loop. Where is Perl's garbage collection?
I am using DB XML version: 2.2.13; BDB version: 4.4.20.2; OS: FreeBSD 6-STABLE. However the problem seems to be common for any OS or BDB XML version as it involves Perl-to-C++ interface.
Has anyone experienced similar problems?
Thanks,
Konstantin.
Konstantin @ Chuguev.com

Answer 1

Good catch - you found a memory leak. Luckily the fix is very straightforward. Edit the file
dbxml/src/perl/common.h
and find this line
#define newSVfromString(str) newSVpvn(str.c_str(), str.length())
Change it to this
#define newSVfromString(str) sv_2mortal(newSVpvn(str.c_str(), str.length()))
and recompile the module.
Paul