Unable to create temporary backing file

Question

My program is throwing out the error:
temporary open: /var/tmp/BDB20825: Too many open files
unable to create temporary backing file
This happens after my program has been running for about 4 hours. The program appears to run out of file descriptors. The listing of /proc/<PID>/fd shows hundreds of lines naming the same file (/var/tmp/BDB20825), like this:
lrwx------ 1 zobell users 64 Jul 28 14:41 622 -> /var/tmp/BDB20825 (deleted)
That file does not exist. I suspect that someone is deleting but failing to close the file. A few open file descriptors with this deleted file occur slowly in the early hours, but not at a fast enough rate to bring the program down when it does. It looks like there is a sudden flood at the end.
The program repeats this error slowly, and is stuck inside the db code. Here is the gdb backtrace I am seeing every time I interrupt:
(gdb) where
#0 0x00e74410 in __kernel_vsyscall ()
#1 0x003d51dd in ___newselect_nocancel () from /lib/libc.so.6
#2 0x002e5d39 in __os_sleep () from /devel/tfm/external/RH4_32/dbxml/latest/install/lib/libdb-4.6.so
#3 0x002e43f1 in __os_openhandle () from /devel/tfm/external/RH4_32/dbxml/latest/install/lib/libdb-4.6.so
#4 0x002e52ad in __os_open () from /devel/tfm/external/RH4_32/dbxml/latest/install/lib/libdb-4.6.so
#5 0x002c0357 in __db_tmp_open () from /devel/tfm/external/RH4_32/dbxml/latest/install/lib/libdb-4.6.so
#6 0x002c006c in __db_appname () from /devel/tfm/external/RH4_32/dbxml/latest/install/lib/libdb-4.6.so
#7 0x002d50c9 in __memp_bhwrite () from /devel/tfm/external/RH4_32/dbxml/latest/install/lib/libdb-4.6.so
#8 0x002d4a27 in __memp_alloc () from /devel/tfm/external/RH4_32/dbxml/latest/install/lib/libdb-4.6.so
#9 0x002d62ec in __memp_fget () from /devel/tfm/external/RH4_32/dbxml/latest/install/lib/libdb-4.6.so
#10 0x00232003 in __bam_new_file () from /devel/tfm/external/RH4_32/dbxml/latest/install/lib/libdb-4.6.so
#11 0x002abc79 in __db_new_file () from /devel/tfm/external/RH4_32/dbxml/latest/install/lib/libdb-4.6.so
#12 0x002abaff in __db_open () from /devel/tfm/external/RH4_32/dbxml/latest/install/lib/libdb-4.6.so
#13 0x002a5d15 in __db_open_pp () from /devel/tfm/external/RH4_32/dbxml/latest/install/lib/libdb-4.6.so
#14 0x0080644c in Db::open () from /devel/tfm/external/RH4_32/dbxml/latest/install/lib/libdb_cxx-4.6.so
#15 0x0053ebbe in DbXml::DbWrapper::open (this=0x405bbee0, txn=0x0, type=DB_BTREE, flags=65537, mode=0) at Transaction.hpp:62
#16 0x00549247 in CacheDatabase (this=0x405bbed8, env=0x9c5a848, type=DB_BTREE, compare=0) at CacheDatabase.cpp:28
#17 0x005497e2 in DocDatabase (this=0x405bbed8, env=0x9c5a848, compare=0) at CacheDatabase.cpp:46
#18 0x00584ced in DbXml::Manager::createDocDatabase (this=0x9c5a830, compare=0) at Manager.cpp:694
#19 0x00519467 in DbXml::CacheDatabaseMinderImpl::verify (this=0xfffffdfe, entry=@0x6ac2f790, cid=-514) at CacheDatabaseMinder.hpp:31
#20 0x00519398 in DbXml::CacheDatabaseMinderImpl::findOrAllocate (this=0x6ac5b4f0, cid=1, allocate=true) at CacheDatabaseMinder.cpp:71
#21 0x00518e59 in DbXml::CacheDatabaseMinder::findOrAllocate (this=0x6acd9fa0, mgr=@0x9c5a830, cid=1, allocate=true) at CacheDatabaseMinder.cpp:21
#22 0x0055b12d in LazyDIResults (this=0x6acd9f98, context=0x6acd9fa0, contextItem=0x4979abc0, expr=@0x9ce5de8, txn=0x0, flags=0) at /devel/tfm/developers/szobell/berkeley-db-xml/dbxml-2.4.16/dbxml/include/dbxml/XmlManager.hpp:80
#23 0x0052f9cd in DbXml::QueryExpression::execute (this=0x9ce5de8, txn=0x0, contextItem=0x4979abc0, context=@0xbfcb0e04, flags=0) at /devel/tfm/developers/szobell/berkeley-db-xml/dbxml-2.4.16/dbxml/include/dbxml/XmlQueryContext.hpp:86
#24 0x00565a3a in DbXml::XmlQueryExpression::execute (this=0x0, contextItem=@0xfffffdfe, context=@0xbfcb0e04, flags=4294966782) at /devel/tfm/developers/szobell/berkeley-db-xml/dbxml-2.4.16/dbxml/include/dbxml/XmlValue.hpp:186
#25 0x0805af4b in XmlRecordWithCounts::ExtractQueryString (this=0xbfcb0dd0, select=@0xbfcb0e2c, result=@0xbfcac1d0, qDocument=0xbfcb0e14) at src/XmlRecordWithCounts.cpp:401
#26 0x080585f6 in OdRecord::GetRequiredContent (this=0xbfcb0dd0, name=@0xbfcac1e0, origin=@0xbfcac1d0, dest=@0xbfcac1c0, type=@0xbfcac1b0, dist=@0xbfcac198, totalCount=@0xbfcac194, route=@0xbfcac1a0, qDocument=0x0) at src/OdRecord.cpp:143
#27 0x0804d9c3 in ProcessOd (fpPrivate=0x8e11098, odDb=@0xbfcb0dd0, fdDb=@0xbfcb0d20) at src/BuildFdDb.cpp:263
#28 0x0804f4c8 in ProcessOneDest (fpPrivate=0x8e11098, odDb=@0xbfcb0dd0, fdDb=@0xbfcb0d20, dest={static npos = 4294967295, _M_dataplus = {<std::allocator<char>> = {<No data fields>}, Mp = 0x9da758c "LAX"}, static Sempty_rep_storage = {0, 0, 69041, 0}}) at src/BuildFdDb.cpp:464
#29 0x080501cd in main (argc=6, argv=0xbfcb10c4) at src/BuildFdDb.cpp:589
At this time, the program is trying to extract fields out of an XmlDocument that is the result of a query. The program uses a pre-calculated XmlQueryExpression to extract a specific field. This may not be the best way to extract a field from a document, given the apparent overhead of needing a temporary file!
My program queries two database files (OD.dbxml and FD.dbxml.new), and uses the information to add records to the second file. The databases are rather big:
bash-3.2$ ls -l /tmp/*dbxml*
-rw-rw---- 1 zobell users 5233745920 Jul 28 14:42 /tmp/FD.dbxml.new
-rw-rw---- 1 zobell users 6539427840 Jul 13 10:16 /tmp/OD.dbxml
(I keep them in /tmp because everywhere else uses NFS and performance is awful with NFS.)
My environment:
Red Hat Enterprise Linux Client release 5.3 (Tikanga) 32 bit.
Berkeley DB XML 2.4.16, C++ interface.
Disk space should not be an issue:
bash-3.2$ df /tmp /var
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/VolGroup00-tmp
28948572 13044384 14409968 48% /tmp
/dev/mapper/VolGroup00-root
144742920 5333544 131938272 4% /
I ran this program about a year ago and did not see this issue. I have made a few changes and linked to the newer version of the XML database since then. Any help is appreciated.

Answer 1

I have more information about this problem.
If I "disable" the routine __os_zerofill() in os_fzero.c, by having it always return (0), I no longer get an exception and my application appears to run smoothly. However, I have not determined what in this routine leads to the exception. And, I have no idea what the short and long term consequences of "disabling" this routine will be....
-Coralie