Implementation & performance of StoredCollection.removeAll()

Question

I'm working with an app that uses StoredMap extensively. In one case, I have a StoredMap with about 10 million entries, and then roughly 10% of the entries with arbitrary keys need to be removed.
Using StoredMap.remove() works just fine, although it is somewhat slow. I have a large known sequence of keys to remove so I tried using StoredMap.keySet().removeAll(), and encountered OOM errors every time. I expected that at worst removeAll() would perform the same as iterating over remove(), and I was hoping that it would be much faster similar to how putAll() is faster than iterating over put(), presumably because multiple elements are inserted in a single transaction.
Looking at the source for StoredCollection.removeAll(), it appears that it attempts to iterate over the entire keySet, tests if the iterated key exists in the collection argument. My naive assumption would be that it should iterate over the collection argument, not the stored key set. The only time the existing approach might make sense is if the the collection argument is larger than the StoredCollection itself.
I'm also not sure why its generating OOM errors. The test below is with only a 64mb heap size, but even in my app with a multi gigabyte app I'm seeing OOM generated when using removeAll().
Below is a simple test class demonstrating the poor performance. With a StoredMap of 100,000 items, my results are:
removeUsingRemoveOnMap 92ms
removeUsingRemoveAllOnKeySet 735ms
With a StoredMap of 1,000,000 items, my results are:
removeUsingRemoveOnMap 1475ms
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
     at com.sleepycat.je.log.LogUtils.readByteArray(LogUtils.java:333)
     at com.sleepycat.je.tree.IN.readFromLog(IN.java:3438)
     at com.sleepycat.je.log.entry.INLogEntry.readEntry(INLogEntry.java:96)
     at com.sleepycat.je.log.LogManager.getLogEntryFromLogSource(LogManager.java:747)
     at com.sleepycat.je.log.LogManager.getLogEntry(LogManager.java:664)
     at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1215)
     at com.sleepycat.je.tree.Tree.getNextBinInternal(Tree.java:1419)
     at com.sleepycat.je.tree.Tree.getNextBin(Tree.java:1282)
     at com.sleepycat.je.dbi.CursorImpl.getNextWithKeyChangeStatus(CursorImpl.java:1629)
     at com.sleepycat.je.dbi.CursorImpl.getNext(CursorImpl.java:1499)
     at com.sleepycat.je.dbi.CursorImpl.getNextNoDup(CursorImpl.java:1688)
     at com.sleepycat.je.Cursor.retrieveNextAllowPhantoms(Cursor.java:2189)
     at com.sleepycat.je.Cursor.retrieveNext(Cursor.java:1987)
     at com.sleepycat.je.Cursor.getNextNoDup(Cursor.java:872)
     at com.sleepycat.util.keyrange.RangeCursor.doGetNextNoDup(RangeCursor.java:920)
     at com.sleepycat.util.keyrange.RangeCursor.getNextNoDup(RangeCursor.java:475)
     at com.sleepycat.collections.DataCursor.getNextNoDup(DataCursor.java:463)
     at com.sleepycat.collections.StoredIterator.move(StoredIterator.java:619)
     at com.sleepycat.collections.StoredIterator.hasNext(StoredIterator.java:151)
     at com.sleepycat.collections.StoredCollection.removeAll(StoredCollection.java:354)
     at com.sleepycat.collections.StoredCollection.removeAll(StoredCollection.java:330)
     at JETester.removeUsingRemoveAllOnKeySet(JETester.java:54)
     at JETester.main(JETester.java:37)
import com.sleepycat.bind.tuple.TupleBinding;
import com.sleepycat.collections.StoredMap;
import com.sleepycat.je.*;
import java.io.File;
import java.util.ArrayList;
import java.util.List;
public class JETester {
public static void main(final String[] args) throws DatabaseException {
final StoredMap<String,String> storedMap = setupMap(new File("c:\\temp"));
populateMap(storedMap, 100000);
removeUsingRemoveOnMap(storedMap);
removeUsingRemoveAllOnKeySet(storedMap);
private static void removeUsingRemoveOnMap(final StoredMap<String,String> storedMap) {
final long startTime = System.currentTimeMillis();
for (int i = 0; i < 100; i++) {
storedMap.remove(String.valueOf(i));
System.out.println("removeUsingRemoveOnMap " + (System.currentTimeMillis() - startTime) + "ms");
private static void removeUsingRemoveAllOnKeySet(final StoredMap<String,String> storedMap) {
final List<String> removalList = new ArrayList<String>();
for (int i = 100; i < 200; i++) {
removalList.add(String.valueOf(i));
final long startTime = System.currentTimeMillis();
storedMap.keySet().removeAll(removalList);
System.out.println("removeUsingRemoveAllOnKeySet " + (System.currentTimeMillis() - startTime) + "ms");
private static void populateMap(final StoredMap<String, String> storedMap, final int size) {
for (int i = 0; i <= size; i++ ) {
storedMap.put(String.valueOf(i),String.valueOf(i));
private static StoredMap<String,String> setupMap(final File databaseDirectory)
throws DatabaseException
final EnvironmentConfig envConfig = new EnvironmentConfig();
envConfig.setAllowCreate(true);
envConfig.setTransactional(true);
final Environment environment = new Environment(databaseDirectory, envConfig);
final DatabaseConfig dbConfig = new DatabaseConfig();
dbConfig.setAllowCreate(true);
dbConfig.setTransactional(true);
final Database database = environment.openDatabase(null, "test", dbConfig);
final TupleBinding<String> stringBinding = TupleBinding.getPrimitiveBinding(String.class);
return new StoredMap<String,String>(database, stringBinding, stringBinding, true);
}

Answer 1

Hi,
Thank you for posting a complete test!
I'll take a look at why removeAll is implemented the way it is later on, and report back here.
For now I just wanted to say that the OOME is probably caused by the large transaction. removeAll (if the database is transactional) must do the entire operation within a single transaction. Each record is locked, which takes up memory. In general, large transactions do take up a lot of memory. So if you don't need to delete the entire set of keys atomically, you're better off not to use removeAll (use remove instead).
To be honest, the bulk collections APIs (putAll, removeAll, etc) were not implemented for performance reasons, but mainly to adhere to the Java collections interface spec for compatibility. It would be great to make these also provide performance benefits, and we should continue to improve them -- your test will be useful for doing that, thanks for that. But you should know that a lot of effort has not been put into that aspect of their implementation so far.
--mark