Log cleaning takes a lot

Question

Hi,
I've been reading the forum but could not find a hint for solving my problem. I apologize if I missed something and the answer is already there.
My scenario for using BDB (je-4.0.103):
- a read-only environment with about 4000 databases. The environment uses about 100GB disk space. Log file size is 10MB. Cache size is 100MB.
- databases are opened only when needed and are closed after a few minutes of inactivity
- when in read-only mode cleaner, checkpointer and IN compressor threads are disabled
- once a day, the env is closed and re-opened in read/write mode for performing bulk data load, in deferred write mode, without transactions. About 0.5 - 1GB of data is imported daily (1 million key-value pairs).
- during load, IN compressor is enabled but cleaner and checkpointer are not
- log cleaning and checkpointing is invoked manually at the end of data import using the algorithm suggested in the docs
After several months of working properly without any signs of degradation the whole thing broke down suddenly. The log cleaning takes now hours instead of minutes and it does not seem to do its job: the environment is now 250GB and keeps growing. Restart also takes several hours.
Any help would be appreciated. If more details about my setup is needed, I can provide it.
Thank you!
Edited by: user9045772 on Sep 17, 2010 3:30 AM
Mentioned the size of the cache.

Answer 1

Hi Eric,
Thank you very much for your help.
I just tried increasing the cache from 100MB to 1GB and it made miracles. Log cleaning was finally completed after 2 hours. The db start time is now back to normal (less than 2 minutes).
I'm still concerned with the fact that log cleaning takes a lot, but at least it finishes after several hours. Will have a look at db space utilization after completing a new run and will tell you the results.
Until then, here are the env config and stats (after increasing the cache), maybe you spot something odd:
------------------ Env Config ------------------------------------
allowCreate=true
cacheSize=1048576000
txnNoSync=false
txnWriteNoSync=false
{je.log.fileMax=10485760,
je.env.runCheckpointer=false,
je.env.runCleaner=false,
je.log.totalBufferBytes=10485760,
je.cleaner.minFileUtilization=50,
je.env.isLocking=false,
je.cleaner.minUtilization=80,
je.sharedCache=true,
je.env.runINCompressor=true,
je.env.isTransactional=false,
je.env.isReadOnly=false,
je.log.faultReadSize=120960,
je.log.bufferSize=5242880,
je.maxMemory=10485760000,
je.log.numBuffers=2,
je.log.iteratorReadSize=120960}
------------------ Env Stats ------------------------------------
I/O: Log file opens, fsyncs, reads, writes, cache misses.
     bufferBytes=10,485,760
     endOfLog=0x0/0x0
     nBytesReadFromWriteQueue=0
     nBytesWrittenFromWriteQueue=0
     nCacheMiss=24,646
     nFSyncRequests=1
     nFSyncTimeouts=0
     nFSyncs=1
     nFileOpens=5,319
     nLogBuffers=2
     nLogFSyncs=3
     nNotResident=24,646
     nOpenFiles=100
     nRandomReadBytes=1,664,658,816
     nRandomReads=13,751
     nRandomWriteBytes=2,961,648
     nRandomWrites=1
     nReadsFromWriteQueue=0
     nRepeatFaultReads=2
     nSequentialReadBytes=53,787,475,644
     nSequentialReads=274,985
     nSequentialWriteBytes=0
     nSequentialWrites=0
     nTempBufferWrites=0
     nWriteQueueOverflow=0
     nWriteQueueOverflowFailures=0
     nWritesFromWriteQueue=0
Cache: Current size, allocations, and eviction activity.
     adminBytes=4,807,377
     cacheTotalBytes=21,728,916
     dataBytes=6,435,779
     lockBytes=0
     nBINsEvictedCACHEMODE=0
     nBINsEvictedCRITICAL=0
     nBINsEvictedEVICTORTHREAD=0
     nBINsEvictedMANUAL=0
     nBINsFetch=53,566
     nBINsFetchMiss=543
     nBINsStripped=0
     nEvictPasses=0
     nLNsFetch=90,991
     nLNsFetchMiss=24,096
     nNodesEvicted=0
     nNodesScanned=0
     nNodesSelected=0
     nRootNodesEvicted=0
     nSharedCacheEnvironments=1
     nUpperINsEvictedCACHEMODE=0
     nUpperINsEvictedCRITICAL=0
     nUpperINsEvictedEVICTORTHREAD=0
     nUpperINsEvictedMANUAL=0
     nUpperINsFetch=36,940
     nUpperINsFetchMiss=9
     requiredEvictBytes=0
     sharedCacheTotalBytes=21,728,916
Cleaning: Frequency and extent of log file cleaning activity.
     cleanerBackLog=7,441
     fileDeletionBacklog=0
     nCleanerDeletions=0
     nCleanerEntriesRead=1,400,054
     nCleanerRuns=4,845
     nClusterLNsProcessed=0
     nINsCleaned=0
     nINsDead=0
     nINsMigrated=0
     nINsObsolete=86
     nLNQueueHits=0
     nLNsCleaned=0
     nLNsDead=0
     nLNsLocked=0
     nLNsMarked=0
     nLNsMigrated=0
     nLNsObsolete=1,390,901
     nMarkLNsProcessed=0
     nPendingLNsLocked=0
     nPendingLNsProcessed=0
     nRepeatIteratorReads=0
     nToBeCleanedLNsProcessed=0
     totalLogSize=0
Node Compression: Removal and compression of internal btree nodes.
     cursorsBins=0
     dbClosedBins=0
     inCompQueueSize=0
     nonEmptyBins=0
     processedBins=15
     splitBins=0
Checkpoints: Frequency and extent of checkpointing activity.
     lastCheckpointEnd=0xae0c/0x74f5c6
     lastCheckpointId=1,590
     lastCheckpointStart=0xae0c/0x4c6dea
     nCheckpoints=1
     nDeltaINFlush=15
     nFullBINFlush=47
     nFullINFlush=51
Environment: General environment wide statistics.
     btreeRelatchesRequired=148
Locks: Locks held by data operations, latching contention on lock table.
     nLatchAcquireNoWaitUnsuccessful=0
     nLatchAcquiresNoWaitSuccessful=0
     nLatchAcquiresNoWaiters=0
     nLatchAcquiresSelfOwned=0
     nLatchAcquiresWithContention=0
     nLatchReleases=0
     nRequests=0
     nWaits=0
Edited by: user9045772 on Sep 17, 2010 4:12 AM
added env config and stats