Scalability with Collections-API and transactions

Question

I'm using the BerkeleyDB such that a visualization which depends on some computations scales even if I just use the saved items to prune the underlying tree (a SunburstView).
Now I have just implemented the BerkeleyDB binding and use the Collection-API. I'm running a transaction after a specified amount of items have been appended to a List to save the list in a database:
<pre>
/** {@inheritDoc} */
@Override
public void diffListener(final EDiff paramDiff, final IStructuralItem paramNewNode,
final IStructuralItem paramOldNode, final DiffDepth paramDepth) {
mDiffs.add(new Diff(paramDiff, paramNewNode.getNodeKey(), paramOldNode.getNodeKey(), paramDepth));
mEntries++;
if (mEntries == AFTER_DIFFS) {
try {
mRunner.run(new PopulateDatabase(mDiffDatabase, mDiffs));
} catch (final Exception e) {
LOGWRAPPER.error(e.getMessage(), e);
mEntries = 0;
mDiffs = new LinkedList<>();
/** {@inheritDoc} */
@Override
public void diffDone() {
try {
mRunner.run(new PopulateDatabase(mDiffDatabase, mDiffs));
} catch (final Exception e) {
LOGWRAPPER.error(e.getMessage(), e);
mStart.countDown();
</pre>
Now I think about what's the right value for transactions? AFTER_DIFFS is currently set to 100 which means everytime the List has 100 elements it's saved in a database within a transaction and a new instance is created. I currently don't have time to empirically measure certain values, but maybe one can get a suggestion which might be a good value or if it really depends? I'm persisting a rather small Object with two long values, an enum (with just a few values and no methods) and another small Object with 2 int values, if that's of significance.
I think the transaction is bound to the current thread, maybe it would be also good to use an ExecutorService with a single thread and let the transaction, the Runner.run(TransactionWorker); run in this thread?
best regards,
Johannes

Answer 1

I changed the database to a temporary database, but I'm getting a horrible read-performance (with and without the temporary database setting). I thought the database wouldn't be persisted if the data isn't too big.
Some debugging output is:
02:13:53.906 [pool-4-thread-1] DEBUG o.t.g.view.model.TraverseCompareTree - stats: I/O: Log file opens, fsyncs, reads, writes, cache misses.
     bufferBytes=3,145,728
     endOfLog=0x0/0x439
     nBytesReadFromWriteQueue=0
     nBytesWrittenFromWriteQueue=0
     nCacheMiss=0
     nFSyncRequests=0
     nFSyncTimeouts=0
     nFSyncs=0
     nFileOpens=0
     nLogBuffers=3
     nLogFSyncs=0
     nNotResident=0
     nOpenFiles=0
     nRandomReadBytes=0
     nRandomReads=0
     nRandomWriteBytes=0
     nRandomWrites=0
     nReadsFromWriteQueue=0
     nRepeatFaultReads=0
     nSequentialReadBytes=0
     nSequentialReads=0
     nSequentialWriteBytes=0
     nSequentialWrites=0
     nTempBufferWrites=0
     nWriteQueueOverflow=0
     nWriteQueueOverflowFailures=0
     nWritesFromWriteQueue=0
Cache: Current size, allocations, and eviction activity.
     adminBytes=153
     avgBatchCACHEMODE=0
     avgBatchCRITICAL=0
     avgBatchDAEMON=0
     avgBatchEVICTORTHREAD=0
     avgBatchMANUAL=0
     cacheTotalBytes=8,265,140
     dataBytes=5,118,941
     lockBytes=318
     nBINsEvictedCACHEMODE=0
     nBINsEvictedCRITICAL=0
     nBINsEvictedDAEMON=0
     nBINsEvictedEVICTORTHREAD=0
     nBINsEvictedMANUAL=0
     nBINsFetch=343
     nBINsFetchMiss=0
     nBINsStripped=0
     nBatchesCACHEMODE=0
     nBatchesCRITICAL=0
     nBatchesDAEMON=0
     nBatchesEVICTORTHREAD=0
     nBatchesMANUAL=0
     nCachedBINs=234
     nCachedUpperINs=8
     nEvictPasses=0
     nINCompactKey=2
     nINNoTarget=0
     nINSparseTarget=8
     nLNsFetch=686
     nLNsFetchMiss=0
     nNodesEvicted=0
     nNodesScanned=0
     nNodesSelected=0
     nRootNodesEvicted=0
     nThreadUnavailable=0
     nUpperINsEvictedCACHEMODE=0
     nUpperINsEvictedCRITICAL=0
     nUpperINsEvictedDAEMON=0
     nUpperINsEvictedEVICTORTHREAD=0
     nUpperINsEvictedMANUAL=0
     nUpperINsFetch=343
     nUpperINsFetchMiss=0
     requiredEvictBytes=0
     sharedCacheTotalBytes=0
Cleaning: Frequency and extent of log file cleaning activity.
     cleanerBackLog=0
     fileDeletionBacklog=0
     nCleanerDeletions=0
     nCleanerEntriesRead=0
     nCleanerRuns=0
     nClusterLNsProcessed=0
     nINsCleaned=0
     nINsDead=0
     nINsMigrated=0
     nINsObsolete=0
     nLNQueueHits=0
     nLNsCleaned=0
     nLNsDead=0
     nLNsLocked=0
     nLNsMarked=0
     nLNsMigrated=0
     nLNsObsolete=0
     nMarkLNsProcessed=0
     nPendingLNsLocked=0
     nPendingLNsProcessed=0
     nRepeatIteratorReads=0
     nToBeCleanedLNsProcessed=0
     totalLogSize=1,074
Node Compression: Removal and compression of internal btree nodes.
     cursorsBins=0
     dbClosedBins=0
     inCompQueueSize=0
     nonEmptyBins=0
     processedBins=0
     splitBins=0
Checkpoints: Frequency and extent of checkpointing activity.
     lastCheckpointEnd=0x0/0x396
     lastCheckpointId=1
     lastCheckpointStart=0x0/0x16c
     nCheckpoints=0
     nDeltaINFlush=0
     nFullBINFlush=0
     nFullINFlush=0
Environment: General environment wide statistics.
     btreeRelatchesRequired=0
Locks: Locks held by data operations, latching contention on lock table.
     nLatchAcquireNoWaitUnsuccessful=0
     nLatchAcquiresNoWaitSuccessful=0
     nLatchAcquiresNoWaiters=0
     nLatchAcquiresSelfOwned=0
     nLatchAcquiresWithContention=0
     nLatchReleases=0
     nOwners=2
     nReadLocks=2
     nRequests=686
     nTotalLocks=2
     nWaiters=0
     nWaits=0
     nWriteLocks=0
Seems it hasn't got any cache misses at all and the other values seem to be ok, too?
best regards,
Johannes