Massive accumulation of byte[]s

Question

Here's hoping someone has some insight on the following problem:
We're running a series of environments that are smallish -- they'll be a few gigabytes in size when fully populated -- but we're having problems getting past the (say) 200Mb barrier. We can currently run our system for about half an hour, adding documents at the rate of about ten to fifteen a second (no queries), per thread, using ten threads. The keys for this database come in two forms: one is just a long that is almost always monotonically increasing (not guaranteed); the other is that same long plus a String. There's 10 of the long/String keys for every long key. Values for the long key is a serialized object with about 10 instance fields; for the latter it's a series of String values.
Here's what we see when the machine starts dying:
* It's associated with garbage collection, specifically with full gc. Typically, it happens when a full GC is triggered. -verbose:gc gives us a time of around 1.0 - 1.5s for the collection but it's obvious, looking at a profiler, that the actual time taken is closer to 30s. I've tried tuning the young/tenured and eden/survivor ratios, but nothing seems to work (details available if you want them.) This behavior doesn't -always- happen with a full gc, but a full gc is going on every time we see the problem.
* The runnable process count goes through the roof. It's usually 1-2; it spikes to around 165-200. It stays in this condition for anywhere between 20 seconds and a minute and, when it's done immediately drops to 1-2 within three seconds.
* The number of context switches and interrupts drops and stays low. CS drops by an order of magnitude; interrupts by 10-20%.
* System time hits ~95% and stays there.
* The machine is unresponsive, even to ls, less, etc.
* There's an ever-increasing number of byte[]s all through the run. We do eventually get out of memory exceptions, but the GC problems and the spike in runnables also occur well before we run out of memory.
* If we remove the code that adds the long/String keys (i.e., if we reduce the amount of data written to 1/11th its normal amount) the problem still happens, but it takes much longer. Also, the number of byte[]s written is dramatically lower -- but still ever-increasing.
I've already tried waiting until all cursors and transactions are closed/committed, then calling sync() and evictMemory(), as per these threads:
OutOfMemoryError: Java heap space [Lucene + BDB JE transactions]
OutOfMemoryException when i do  data insertion continuously
We do not have any large transactions.
Environment:
* java -version: Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_09-b01, mixed mode)
* uname -a: Linux bubba 2.6.15-1-amd64-k8-smp #2 SMP Tue Mar 7 21:00:29 UTC 2006 x86_64 GNU/Linux
* JE 3.2.13
* Machines are dual CPU, dual-core Opterons @2.0GHz, 8Gb RAM, 10k disks on SATA-II, no RAID
* Environment is transactional, allowCreate, txnWriteNoSync = true, 10Mb cache size (too low, but increasing it had no effect)
* Databases are transactional, allowCreate
* All writes are performed with transactions and all transactions are explicitly committed or aborted
Help is much appreciated. This is a blocking issue for us (because the machine is unresponsive, it crashes everything in our app and affects other machines as a result.) I've been beating my head against it -- to no avail.
Thanks in advance for whatever help you can offer. I'm stumped (again).
-j

Answer 1

Mark, Linda and all,
I spoke too soon when I said I found and solved the problem with the number of byte arrays and the massive number of runnable threads. It turns out that there were two problems -- I had a memory leak in my own code that didn't release the bytes, and there's another problem that seems to involve JE and is related to the number of threads in the runnable queue. ALL of the symptoms I described above -- the number of threads in the runnable queue, the number of context switches and interrupts, the system time stats, etc. -- with the exception of the number of byte arrays, are accurate descriptions of the second problem. In other words, they're not related to the problem with my memory leak, and I'm hoping you can lend a hand.
I have a test I can run outside our application that reproduces the problem with the latest 3.2 build, though it occurs at random and is worse on some machines compared to others. The test allocates enough environments, each with 10Mb cache, to fill a certain percentage of the heap; that means hundreds environments on our largest machine. Each environment has a single database and the test tries to add 50 million, 2-5k records in parallel. Obviously you wouldn't do this in a "real" application but we see the same problem this test produces in our "real" code, which uses only ~25 environments and ten threads adding data concurrently.
To answer some of your comments above:
* All my tests are using the latest 3.2 build. Each test is started cleanly, so no data from previous runs exists beforehand (avoiding the problem with the log file formats.)
* I can reproduce the problem on Xeon as well as Opteron machines. I don't have access to a single-core machine, so all the tests have been run on machines that are dual-core, dual-CPU or both. One of our Xeon machines really, really likes producing this problem. It has no problems running anything else, however. I'm in the process of validating the hardware now.
* I tried using the je.evictor.lruOnly=false setting and had the same results
* The problem becomes worse when more of the heap is used. If I allocate enough Environments to use 85% of the heap as cache, I see lots and lots of blocked threads, but none of the high runnable-count behavior. If I bump that up to 95% I start seeing problems
* I can't get a stack dump of any sort -- the machine is so locked up that the kill -3 executes AFTER the machine has relaxed. Using nice, writing to ramdisks, writing to other machines via netcat -- none of them helped us get a stack dump. Running vmstat before the machine locks works, but it seems like there's no way to fire up a new process.
Obviously we're not using JE in the way suggested by the docs -- in particular, we're running with 24 environments and that doesn't use resources efficiently. We're willing to live with the inefficiency. The real problem here is that the machine occasionally locks so hard that I can't even use 'ls' or 'kill'. I'd be hard-pressed to purposely write something in Java that does that.
I have full logs with -verbose:gc and -XX:+CITime set on the vm; the logs include stats dumps from the databases and environments, and the output of vmstat at five second intervals. I can also provide the test that reproduces the problem.
As I've said, any help is much appreciated. If we're doing something wrong, I would like to know about it. Thanks for your help!
-j