Cache query results in too much garbage collection activity

Question

Oracle Coherence Version 3.6.1.3 Enterprise Edition: Production mode
JRE 6 Update 21
Linux OS 64 bit
The application is using a object Customer having following structure
Customer(CustID, FirstName, LastName, CCNumber, OCNumber)
Each property of Customer is a inner classes having getValue as one of the methods retuning a value. The getValue method of CCNumber and OCNumber return a Long value. There are 150m instances of Customer in cache. To hold this much data in cache we are running several nodes on 2 machines.
The following code is used to create indexes on CCNumber and OCNumber:
     ValueExtractor[] valExt = new ValueExtractor[]{
          new ReflectionExtractor("getCCNumber"), new ReflectionExtractor("getValue")};
     ChainedExtractor chExt = new ChainedExtractor(valExt);
     Long value = new Long(0);
     Filter f = new NotEqualsFilter(chExt, value);
     ValueExtractor condExtractor = new ConditionalExtractor(f, chExt, true);
     cache.addIndex(condExtractor, false, null);The client code queries the cache with following code:
     ValueExtractor[] valExt1 = new ValueExtractor[]{
          new ReflectionExtractor("getCCNumber"), new ReflectionExtractor("getValue")};
     ChainedExtractor chExt1 = new ChainedExtractor(valExt1);
     EqualsFilter filter1 = new EqualsFilter(chExt1, ccnumber);
     ValueExtractor[] valExt2 = new ValueExtractor[]{
          new ReflectionExtractor("getOCNumber"), new ReflectionExtractor("getValue")};
        ChainedExtractor chExt2 = new ChainedExtractor(valExt2);
     EqualsFilter filter2 = new EqualsFilter(chExt2, ocnumber);
     AnyFilter anyFilter = new AnyFilter(new Filter[]{filter1, filter2})
     cache.entrySet(anyFilter);The observation is that for 20 client threads the application performs well(avg response time = 200ms) but as the number of client threads increases the application performance goes down disproportionately(query returns anywhere between 1000ms to 8000ms for 60 threads). I think this is because of the eden space filling up very fast when the number of client thread goes up. The number of collections per second goes up with the number of client threads. There are almost 2-3 ParNew collections every second when there are 60 client threads where as only 1 collection per second for 20 client threads. Even 100-200ms pause degrades the overall query performance.
My question is why coherence is creating so many objects that fills up eden so fast? Is there anything I need to do in my code?

Answer 1

Hi Coh,
The reason for so much garbage is that you are using ReflectionExtractors in you filters, I assume you do not have any indexes on your caches either. This means that each time you execute a query Coherence has to scan the cache for matches to the filter - like a full table scan in a DB. For each entry in the cache Coherence has to deserialize that entry into a real object then using reflection call the methods in the filters. Once the query is finished all these deserialized objects are garbage that needs to be collected. For a big cache this can be a lot of garbage.
You can change to POF extractors to save the deserialization step which should reduce the garbage quite a bit, although not eliminate it. You could also use indexes, which should eliminate pretty much more of the garbage you are seeing during queries.
JK