Stranger performance profile

Question

I suppose this is pretty vague, but I will ask anyway... I have a couple of very large databases (6G or so apiece) on 64-bit linux using version 4.5 that I access and delete from using the java API in a single long transaction with multiple cursors in various loops-- it's sort of a graph search which is part of a bigger computation.
I'm finding that operations are many orders of magnitude slower in the midst of than what I expect, and to verify that it's not just because I have a huge database or inefficient algorithm, I've taken out the most problematic part and I run it on its own, and it runs hundreds or thousands of times faster. Cursor.getSearchBoth slows down from .08ms in the standalone case to 10ms; Cursor.delete, once it's already positioned, slows from .02ms to almost 5ms, and perhaps most bizarrely, Cursor.close slows from too small to measure to 6ms! I'm not even looking at commit time or anything right now, just performance of these cursors; again there's just one transaction here.
I don't see any evidence that the machine is thrashing, there's tons of free memory in the jvm, the cache misses per operation are extremely similar between the real app and the test app, and almost the whole database would fit in the cache I've set (though changing the cache size has negligible effect on all this). The key/data pairs are all quite small; 8 byte keys and ~40 byte values; I'm using the default page size. I guess I'm wondering how I can start to pin this down. These numbers really seem crazy, especially Cursor.close... what could I possibly be doing to make that run so slow?? If it helps, here is how I set up the environment, transactions, and cursors in both programs:
EnvironmentConfig config = new EnvironmentConfig();
config.setRunFatalRecovery(true);
config.setErrorStream(System.err);
config.setErrorPrefix("BerkeleyDB> ");
config.setCacheSize(40 * 1024 * 1024 * 1024L);
config.setLogRegionSize(512 * 1024);
config.setLogBufferSize(4 * 512 * 1024);
config.setMaxLocks(1000000);
config.setMaxLockObjects(1000000);
config.setMaxLockers(1000000);
config.setLockDetectMode(LockDetectMode.YOUNGEST);
File baseDirectory = new File(dir).getAbsoluteFile();
setDirectories(baseDirectory, config);
config.setAllowCreate(true);
config.setInitializeLocking(true);
config.setInitializeLogging(true);
config.setInitializeCache(true);
config.setTransactional(true);
config.setPrivate(true);
Environment dbEnv = new Environment(baseDirectory, config);
DatabaseConfig config = new DatabaseConfig();
config.setSortedDuplicates(true);
config.setAllowCreate(true);
config.setReadUncommitted(true);
config.setMode(0664);
config.setType(DatabaseType.BTREE);
config.setTransactional(true);
Database db = dbEnv.openDatabase(null, name,
null, config);
TransactionConfig transactionConfig = new TransactionConfig();
transactionConfig.setReadUncommitted(true);
Transaction txn = dbEnv.beginTransaction(null, transactionConfig);

Answer 1

Ugh.. OK, the best way to find a bug that's been driving you crazy all week is to publically whine about it on a forum. I had a cursor in a tight loop that I wasn't closing. I found this literally 5 minutes after posting... but hey, whatever it takes! People, take note! Don't mess with them cursors!