File Handler Leak?

Question

After we upgrade to 4.0.71 from 3.0, we see multiple file handlers are opened for the last jdb file. We have 100+ clients and each clients have their own environment, thus caused us top the file handler limit on linux.
We are using
OS: Linux 2.6.18-92.1.6.el5 #1 SMP Fri Jun 20 02:36:06 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux
The environment is writable.
I would like to confirm if this is related to the issue fixed in 4.0.92:
Fix a problem where cleaned and deleted log files could accumulate in the log cleaner's backlog, or list of files to be cleaned. This occurs when multiple cleaner threads are configured. The impacts of this problem are:
1. The EnvironmentStats.getCleanerBacklog stat is incorrect, which could lead the application to unnecessarily increase the number of cleaner threads.
2.
3. If EnvironmentConfig.CLEANER_MAX_BATCH_FILES is set to a non-zero value, log cleaning is disabled when the number of deleted files in the backlog reaches this limit.
[#18179]
or is another bug or expected behavior.
To be specific:
I ran the lsof for the berkeley DB directory:
java 13665 mybuys 147r REG 253,0 141869 25927939 /berkeley-db/00000000.jdb
java 13665 mybuys 148r REG 253,0 141869 25927939 /berkeley-db/00000000.jdb
java 13665 mybuys 279rw REG 253,0 0 25927936 /berkeley-db/je.info.0.lck
java 13665 mybuys 280w REG 253,0 0 25927937 /berkeley-db/je.info.0
java 13665 mybuys 281uw REG 253,0 0 25927938 /berkeley-db/je.lck
java 13665 mybuys 282r REG 253,0 141869 25927939 /berkeley-db//00000000.jdb
and note that there are 3 instances of 00000000.jdb. I only see one file hander to this file when server just restart. We have a periodical process to refresh the Berkeley DB from data feed, and we see two handler increased in each refresh. The environment handler, entityStore, and primaryIndex are all singleton per client.
Can someone explain why there is a need to have multiple hanlders opened for one jdb file (and seems always the last one). Is there a workaround with it.
Edited by: JoshWo on Mar 2, 2010 3:14 PM

Answer 1

Everything you asked about is per environment, not per environment handle. In general, we never allocate expensive resources like file handles for each environment handle.
For us, it is not critical to have many file handlers but we just need to be able to estimate before deployment so OS limit can be set correctly (I believe the max is 40K per process on Linux). Any formula to publish?For any environment:
<li> Read-only handles: You can set the maximum to 3, but that is the smallest maximum you can specify. If you only have one log file then only one handle will be open. But if you have 3 log files, then 3 file handles may be opened.
<li> je.lck: 1 handle is always open.
For a read-write environment, there are the following 3 additional handles (added to the above):
<li> Write handle: 1 handle for writing.
<li> Fsync handle: 1 handle for fsync.
<li> je.lck: 1 additional handle for exclusive locking.
Also, you mentioned "maximum number of open-for-read handles is reached". Can you point a documentation link on this parameter or elaborate more here? Is it per environment handler or per environment?EnvironmentConfig.LOG_FILE_CACHE_SIZE is in the javadoc, in the je package.
So overall (if you set LOG_FILE_CACHE_SIZE to 3), you should assume:
-- 4 handles for a read-only environment
-- 7 handles for a read-write environment
I suggest you test this. I'm giving you these numbers by reading the code, not by testing.
--mark