HashMap String, ConcurrentHashMap Long, Bool concurrency , out of memory

Question

Need a cache as hashmap<String,ConcurrentHashMap<Long, Boolean>> for 500,000 String keys,
each with ConcurrentHashMap of max size - million for bulk deletes (based on string key), statistics and concurrency??
where total record count of all concurrentHashMaps is about 20 million.
As the application is multi threaded, and we want fastest insert, update, and delete cache the only choice was a concurrentHashMap.
Methods such as put, putIfAbsent, delete are exposed for this cache.
For example, Is the following thread safe?
cache = new HashMap<String, ConcurrentHashMap<Long, Boolean>>
put(T key, value){
ConcurrentHashMap map;
if((map = getConcurrentHashMap(key.getStringValue())) != null){
     synchronized(cache){
          map = new ConcurrentHashMap<Long, Boolean>();
          cache.put(key.getStringValue()), map);
map.put(key.getLongValue(), false)
Even before we loaded a million records, we ran out of space. Why do we have so many, about 16 times the number of concurrent hash maps in heap and this is a bad sign, what other options do we have?
12355048     593042304.00     565.5692     java.util.concurrent.locks.ReentrantLock$NonfairSync
12355026     593041248.00     565.5682     java.util.concurrent.ConcurrentHashMap$Segment
12355026     396169112.00     377.8163     [Ljava.util.concurrent.ConcurrentHashMap$HashEntry;
772191     55597752     53.02215     java.util.concurrent.ConcurrentHashMap
Thanks
Krishna

Answer 1

jtahlborn wrote:
803891 wrote:
:), could you read again please.
Ofcourse, I do use putIfAbsent, but as there are many concurrent hashmaps, the only time I use synchronized is when I do not find a key without an initialized concurrentHashMap.i did read your post the first time, and it is still incorrect. the point of ConcurrentHashMap is to avoid using synchronized. besides that, your code has a timing hole if more than one thread try to access a new key for the first time. they could get 2 different sub-maps. (not to mention a logic bug). the correct way to lazy-load a ConcurrentMap (assuming that object construction for the values is cheap):
ConcurrentHashMap<Long, Boolean> map;
if((map = getConcurrentHashMap(key.getStringValue())) == null) {
map = new ConcurrentHashMap<Long, Boolean>();
ConcurrentHashMap<Long, Boolean> curMap = cache.putIfAbsebent(key.getStringValue()), map);
if(curMap != null) {
map = curMap;
The good thing about my cache is 99% of the keys (String) are eagerly known. The cache is a HashMap here, so are you suggesting this should also be a ConcurrentHashMap as well, otherwise your cache.putIfAbsent will still require a lock on cache itself, which is what I did below, is there anything wrong here:
public Boolean put(T key){
          String stringKey = key.getStringKey();
          return getLongCache(stringKey).put(key.getLongKey, true);
all ... methods use getLongCache() first
private ConcurrentHashMap<Long, Boolean> getLongCache(
               String stringKey) {
          ConcurrentHashMap<Long, Boolean> longCache;
          if((longCache = cache.get(key.getStringValue())) == null) {
               longCache = putIfAbsentLongCache(key.getStringKey);
          return longCache;
public ConcurrentHashMap<Long, Boolean> putIfAbsentLongCache(
               String stringKey){
          // acquire implicit client side lock as it is different from this lock
          synchronized(cache) {
               ConcurrentHashMap<Long, Boolean> longCache;
               if((longCache = cache.get(stringKey)) == null) {
                    longCache = new ConcurrentHashMap<Long, Boolean>();
                    cache.put(stringKey, longCache);
               return longCache;
     }>
No, the whole idea was to avoid the database and use it as a cache, what other alternatives do I have.i understand that. hope you are using a 64bit jvm on a box with lots o' memory. however, for most normal database scenarios, a good ORM (like hibernate) with the correct caching configured will handle the load you need. you would only need something like this for an extreme scenario, in which case you'll probably need a big box.Yeah, I did not really expect this, that is way too much memory to just setup about half a million concurrentHashMaps. But are there other choices such as writing my own map type?
Edited by: 803891 on Oct 21, 2010 8:24 AM