Evictor/UtilizationProfile locking - severe performance problem

Question

I have a database with 2.5 million entries, 10 - 20 bytes keys and 40kb values (about 200 GB overall).
je.properties:
je.maxMemory=419430400
je.evictor.lruOnly=false
je.env.isTransactional=false
je.cleaner.threads=40
je.evictor.nodesPerScan=100
I know I can't fit the data in the cache, but the tree should fit. The access pattern is random.
The situation is this:
Every get or put is causing an eviction. To do this the PrivateEvictor must be locked:
"NioProcessor-1" prio=3 tid=0x08d91c00 nid=0x47 waiting for monitor entry [0x70fcb000..0x70fcb970]
java.lang.Thread.State: BLOCKED (on object monitor)
     at com.sleepycat.je.evictor.Evictor.doEvict(Evictor.java:228)
     - waiting to lock <0x7f157c68> (a com.sleepycat.je.evictor.PrivateEvictor)
     at com.sleepycat.je.evictor.Evictor.doCriticalEviction(Evictor.java:269)
at com.sleepycat.je.dbi.CursorImpl.close(CursorImpl.java:711)
     at com.sleepycat.je.Cursor.close(Cursor.java:326)
Once it has a lock on the evictor it seems to need to lock the UtilizationProfile:
"NioProcessor-25" prio=3 tid=0x08ee4800 nid=0x5f waiting for monitor entry [0x70833000..0x70833970]
java.lang.Thread.State: BLOCKED (on object monitor)
     at com.sleepycat.je.cleaner.UtilizationProfile.putFileSummary(UtilizationProfile.java:787)
     - waiting to lock <0x7f55bf70> (a com.sleepycat.je.cleaner.UtilizationProfile)
     at com.sleepycat.je.cleaner.UtilizationProfile.flushFileSummary(UtilizationProfile.java:776)
     at com.sleepycat.je.cleaner.UtilizationTracker.evictMemory(UtilizationTracker.java:100)
     at com.sleepycat.je.evictor.PrivateEvictor.startBatch(PrivateEvictor.java:109)
     at com.sleepycat.je.evictor.Evictor.evictBatch(Evictor.java:302)
     at com.sleepycat.je.evictor.Evictor.doEvict(Evictor.java:244)
     - locked <0x7f157c68> (a com.sleepycat.je.evictor.PrivateEvictor)
     at com.sleepycat.je.evictor.Evictor.doCriticalEviction(Evictor.java:269)
But that is locked by a cleaner thread which holds onto the lock the whole time that it is deciding which file to delete:
"Cleaner-19" daemon prio=3 tid=0x08d35800 nid=0x2f runnable [0x71763000..0x71763970]
java.lang.Thread.State: RUNNABLE
     at java.util.TreeMap.successor(Unknown Source)
     at java.util.TreeMap$PrivateEntryIterator.nextEntry(Unknown Source)
     at java.util.TreeMap$EntryIterator.next(Unknown Source)
     at java.util.TreeMap$EntryIterator.next(Unknown Source)
     at com.sleepycat.je.cleaner.UtilizationProfile.getBestFileForCleaning(UtilizationProfile.java:344)
     - locked <0x7f55bf70> (a com.sleepycat.je.cleaner.UtilizationProfile)
     at com.sleepycat.je.cleaner.FileSelector.selectFileForCleaning(FileSelector.java:187)
     at com.sleepycat.je.cleaner.FileProcessor.doClean(FileProcessor.java:197)
     - locked <0x7f153340> (a com.sleepycat.je.cleaner.FileProcessor)
     at com.sleepycat.je.cleaner.FileProcessor.onWakeup(FileProcessor.java:138)
     at com.sleepycat.je.utilint.DaemonThread.run(DaemonThread.java:141)
     at java.lang.Thread.run(Unknown Source)
This seems to take quite a while.
So basically I have about worker 50 threads all waiting to lock the evictor and horrible throughput and response times.
Problems that I see:
1. If an eviction is in progress having other threads waiting to try to evict more stuff seems pointless.
2. It is impossible for evictions to happen in parallel.
3. The locking on UtilizationProfile to way too coarse: cleaner threads shouldn't be able to block evictions for long periods of time.
4. getBestFileForCleaning doesn't seem very efficient. Big TreeMaps are getting copied around and iterated over. If the data was kept in a priority queue cleaner threads might be able to quickly take the best file of the queue.
5. Ideally nothing would be being added to or evicted from the cache all the time. Because of my data access patterns I know that having the values in the cache is pointless, and get and puts could bypass the cache completely.

Answer 1

Hello Cormac,
If you're a customer, please contact me via email to take this offline (charles.lamb @ o.com).
Here are some suggestions, questions, and comments:
. Do you see this particular thread dump all the time or just occasionally? In other words, is it always the same or just certain parts of it? I know you're seeing the eviction part all the time, but what about the Cleaner Thread?
. One workaround you can try is to increase the amount of cache for the Utilization tracker. You can do this by changing the EnvironmentConfig.CLEANER_DETAIL_MAX_MEMORY_PERCENTAGE. It defaults to 2% so moving it to 5% might have a big impact.
. Your cache size is << your database size. Have you run DbCacheSize to see if you have sized your cache large enough to fit all of the internal tree nodes in memory? In general, you don't have a lot of cache so an increase in cache size may yield a significant increase in performance.
. You can look at the stats and if there is no cleaner backlog, you can reduce the number of cleaner threads from 40.
. You can try increasing the log file size from the default of 10MB to (say) 50 or 100MB.
Charles Lamb