Mutex region size - further analysis

Question

Hi,
I seem to be having issues with mutex region size. Specific error message that I get is "Cannot allocate memory: unable to allocate memory for mutex; resize mutex region".
I have gone through all the posts and have increased set_max_mutex to a higher number and that seems to have resolved but I am trying to figure out the reasons for this problem surfacing now. As suggested in releasing mutexes, related to "unable to allocate memory for mutex" it could be a bug in my code but so far I havent found one.
Notable changes that I have made is:
1) Split multiple dbs into their own physical files.
2) Also added a second index db for the main db. Now I have roughly ~300 main dbs per environment and 2 index dbs for each of those totalling around 900 - 1000 primary and index dbs in an environment
API doc says that "Berkeley DB allocates a default number of mutexes based on the initial configuration of the database environment. That default calculation may be too small if the application has an unusual need for mutexes (for example, if the application opens an unexpectedly large number of databases)"
My questions are:
1) Could an increase from 600 - 700 to 1000 dbs cause this ? I dont see this happening on all environments. I have 15 - 20 of those.
2) This one is more about the way BDB uses these mutexes. I noticed that I get this error only on bdb writes and not on reads, in contrast to another post complaining about errors on read only. Can you confirm if we should see this only on writes and not on reads ? I am a bit puzzled by the fact that I dont see these on reads which I perform on snapshot isolation.
One more observation. I have a big env where this problem is pronounced. Env has multiple dbs with total size ~8 - 9 gigs. I have a process which scans all these dbs at night. At that time I see that this mutex count starts jumping and doesnt go down. I am performing these reads at snapshot level. Have already checked the code that is performing read op. I am closing cursors and transactions with proper java exception handling so on first thought I can not see this leak happening through my code, but this mutex count doesnt seem to drop at all.
3) I noticed that default value is 1 million which seems a bit too high. I increased it to 2 or 3 million on different environments which works but also seems a really high number; with my limited knowledge, I must add. Does this number seems normal by BDB standards?
Here ares some details on my bdb env:
1) uses JNI version with java API from tomcat
2) Highly transactional environment
3) Writes as default tralsaction type
4) All reads with snapshot isolation
Thanks in advance for all the help. I will be more than happy to provide more info about the bdb env and the way I use it.
Edited by: newbdb on Sep 29, 2010 9:29 AM

Answer 1

Hi Shishir,
Well, you seem to know how to resolve this already, that is, by increasing the number of mutexes via:
DB_ENV->mutex_set_max()
DB_ENV->mutex_set_increment()
Estimating the number of mutexes needed at run-time is complicated. The default value is based on a worst case scenario for small databases' page sizes. Each buffer (in cache) requires a mutex, there's a mutex for each region (environment shared memory region), a mutex for each logical database and open database handle etc.
1) Could an increase from 600 - 700 to 1000 dbs cause this ? Definitely yes.
With snapshot isolation, the cache size requirements are larger, hence the larger number of mutexes needed for the buffers in cache; also, there will be more mutexes for the buckets in cache (which will protect the chain of page versions).
The optimal way to approach this mutex sizing problem is to see how many mutexes are allocated during a normal initialization, add more for emergencies (say 1000), and go with that value.
You can used db_stat -x to see the mutex statistics. Or programmatically, using the DB_ENV->mutex_stat() method.
Also, if you are on a 5.0 or newer release, there will be additional mutex sizing requirements, because starting with 5.0 we switched to having a mutex per buffer in the cache, in order to increase concurrency. If so, look into the DB_ENV->set_mp_pagesize() method. This method sets the assumed page size which is used to provision the number of mutexes allocated to each memory pool cache. It defaults to 4KB if not explicitly set. If the page size for the databases in the application is smaller than 4KB then you should set to that page size.
Regards,
Andrei