Distributed Cache : Performance issue; takes long to get data

Question

Hi there,
     I have set up a cluster on one a Linux machine with 11 nodes (Min & Max Heap Memory = 1GB). The nodes are connected through a multicast address / port number. I have configured Distributed Cache service running on all the nodes and 2 nodes with ExtendTCPService. I loaded a dataset of size 13 millions into the cache (approximately 5GB), where the key is String and value is Integer.
     I run a java process from another Linux machine on the same network, that makes use of the this cache. The process fetches around 200,000 items from the cache and it takes around 180 seconds ONLY to fetch the data from the cache.
     I had a look at the Performance Tuning > Coherence Network Tuning and checked the Publisher and Receiver Success rate and both were neardly 0.998 on all the nodes.
     It a bit hard to believe that it takes so long. May be I'm missing something. Would appreciate if you could advice me on the same?
     More info :
          a) All nodes are running on Java 5 update 7
          b) The java process is running on JDK1.4 Update 8
          c) -server option is enabled on all the nodes and the java process
          d) I'm using Tangosol Coherence 3.2.2b371
          d) cache-config.xml
                    <?xml version="1.0"?>
                    <!DOCTYPE cache-config SYSTEM "cache-config.dtd">
                    <cache-config>
                    <caching-scheme-mapping>
                    <cache-mapping>
                    <cache-name>dist-*</cache-name>
                    <scheme-name>dist-default</scheme-name>
                    </cache-mapping>
                    </caching-scheme-mapping>
                    <caching-schemes>
                    <distributed-scheme>
                    <scheme-name>dist-default</scheme-name>
                    <backing-map-scheme>
                         <local-scheme/>
                    </backing-map-scheme>
                    <lease-granularity>member</lease-granularity>
                    <autostart>true</autostart>
                    </distributed-scheme>
                    </caching-schemes>
                    </cache-config>
     Thanks,
     Amit Chhajed

Answer 1

Hi Amit,
     Is the java test process single threaded, i.e. you performed 200,000 consecutive cache.get() operations? If so then this would go a long ways towards explaining the results, as most of the time in all processes would be spent waiting on the network, and your results would come out to just over 1ms per operation. Please be sure to run with multiple test threads, and also it would be good to make use of the cache.getAll() call where possible to have a single thread fetching multiple items in parallel.
     Also you may need to do a some tuning on your cache server side. In general I would say that on a 1GB heap you should only utilize roughly 750 MB of that space for cache storage. Taking backups into consideration this means 375MB of data per JVM. So with 11 nodes, this would mean a cache capacity of 4GB. At 5GB of data each cache server will be running quite low on free memory, resulting in frequent GCs which will hurt performance. Based on my calculations you should use 14 cache servers to hold your 5GB of data. Be sure to run with -verbose:gc to monitor your GC activity.
     You must also watch your machine to make sure that your cache servers aren't getting swapped out. This means that your server machine needs to have enough RAM to keep all the cache servers in memory. Using "top" you will see that a 1GB JVM actually takes about 1.2 GB of RAM. Thus for 14 JVMs you would need ~17GB of RAM. Obviously you need to leave some RAM for the OS, and other standard processes as well, so I would say this box would need around 18GB RAM. You can use "top" and "vmstat" to verify that you are not making active use of swap space. Obviously the easiest thing to do if you don't have enough RAM, would be to split your cache servers out onto two machines.
     See http://wiki.tangosol.com/display/COH32UG/Evaluating+Performance+and+Scalability for more information on things to consider when performance testing Coherence.
     thanks,
     Mark