Off-heap backing maps seem to generate lots of garbage at insert..!?

Question

I have been doing a lot of benchmarks of distributed caches with different backing maps. The results where partly positive (I hoped that partitioned (splitting) off-heap backing map would be almost as fast as a non-splitting on-heap backing map). For read and various types of queries this turned out to be mostly true (some queries were slightly slower - probably because they where performed per partition).
For inserts it does however sadly enough seem to be another story - already when using a non-splitting NIO backing map inserts seemed to generate a lot of garbage slowing the benchmark down significantly and when switching to a splitting NIO backing map this effect became so extreme that full GC occured more or less constantly on the cache nodes slowing execution down to almost a standstill :-(
Has anybody else tried this and seen the same results or do any of the Coherence developers have some theory?
To me it would seem like network-io to off-heap (using storage buffers allocated using nio just as the communication buffers!) should be at least as easy to perform without generating excessive garbage as to heap objects but since I dont know the internals of Coherence I cant say for sure if there are something that breaks this theory?
For me the main expected advantage with using off-heap rather than on-heap would have been REDUCED GC activity and shorter pauses but instead it seems like the result is the oposite - at least when doing inserts...
My example do not use (or need!) and secondary indexes (only performs get/put/lock/unlock) but each entry is locked before it is inserted and unlocked after (this is needed for the algorithm I am using as a benchmark) - as I have pointed out in another thread it is a pitty that no "lockAll" / unlockAll method calls exists (my benchmark is suffering a lot from all the lock/unlock remote calls) - the overhead for this is however nothing compared to the performance hit that comes from the all the GC...
I have tried to tune the GC in several ways but this has only to a very limited extent reduced the GC pauses length or the frequency of full-GC - it just seems like a LOT of garbage is generated for some reason...
The setings that so far was resulted in the least GC-overhead (still awfully bad though!) are -XX:+UseParallelGC -XX:+UseAdaptiveSizePolicy. I am using Coherence 3.5 GE and Sun JRE 1.6.0_14.
/Magnus
Edited by: MagnusE on Aug 10, 2009 3:01 PM

Answer 1

Thanks for ther info - I was indeed using different initial and max size in this experiment and seting them the same eased the problem (now I mostly get incremental rather than full GC messages). Insert do however still generate more GC activity than read (that seem to be more or less totally free from Java heap allocation / deallocation which is VERY good since read is so common!). Perhaps there is some more tweaking of the heap allocation/deallocation that can be done att the same time as you work on that bug you mentioned - it would really be nice with a NIO backing-map with close to zero Java heap usage for all primitive operations (read, insert, delete)!
/Magnus
Edited by: MagnusE on Aug 11, 2009 7:33 AM