Help with OutOfMemoryError

Question

hi,
our initial scenario is as follows. we were running a jboss server with a single bdb environment with the default memory setup (60% of vm space). the vm is allocated about 1GB of memory on a 4GB machine. during setup, we run an operation which pushes about 1 million values into the bdb database (values average ~2k, some much bigger, some smaller). we are using a transacted environment with setTxnWriteNoSync enabled (we run fsync at the end of the entire operation). the only other param tweaked is that we set je.log.faultReadSize=4096. now, we ran this operation quite happily initially.
however, for other performance reasons, we decided to split the data into 3 environments on the box. this led to scenario number two, which was 3 separate envs in the same jboss server, with the maxMemoryPercent for each set to 20%. We were actually able to run this load slightly faster, but hit an OOME about 1/5 of the way through. unfortunately we weren't able to get any mem dumps from the vm, but seeing as the only thing that changed is the bdb setup, it didn't seem like any of our code would be causing the issue. we ended up making two changes: we backed the mem percent of each env down to 15%, and upped the je.evictor.evictBytes to 5242880. this setup then ran happily to completion.
my initial guess as to the cause is that since the evictor was evicting such a relatively small amount of memory each time, we were out-running the garbage collector. if that is truly the case, we should be able to push the maxMemoryPercent back up to 20% and still have everything run happily (haven't had time to test this yet). are there any other settings that could be relevant here? if the cause really is the evictBytes setting, is it possible that this setting should be scaled based on the amount of cache size instead of a fixed size (5k may make sense if your cache size is 36MB but maybe not if it is 600MB)? how exact is the cache size ceiling anyway--do we need to allow for some amount of spillover per environment (e.g. 20% + 20% + 20% != 60%)? is there anything in the stats that we could look at to get some better guidelines? any other suggestions?
thanks,
-james

Answer 1

Hi James,
It is definitely true that there is some amount of memory overhead per environment that is not included in what we budget for the JE cache. We don't account for this memory in the implementation, and have not done an analysis to determine how much memory it is.
So I'm sorry to put this back on you, but I'm afraid you'll have to account for some fixed amount (1 or 2 MB perhaps) per environment, when dividing up memory among multiple environments. I'm afraid you'll have to experiment to find a value that works well. Our stats are not useful, since as I say this overhead per environment is not something we measure.
Please be aware that I'm only guessing that the per-environment overhead is the cause, based on your experiment. If you can reproduce the problem reliably and/or get stack traces, I may be able to be more certain.
From another perspective, if you're that tight on memory you may see high GC overhead and poor performance. For best performance and to avoid OOME I suggest giving the GC more working room.
On a somewhat related note, we recently encountered other issues with GC and multiple environments:
http://blogs.oracle.com/charleslamb/2007/02/22#a50
Mark