Bdb cleaner fall behind a lot

Question

Hi,
Firstly, apologize for my poor english.
I have met some issues when we using bdb in our application, and need help.
We are using bdb in transaction mode in our application.
The issue we met is that:
the bdb cleaner is running, and I can saw bdb log files adding and deleting quickly, but the disk size is increasing about 4~5 G per week, and CPU is high.
Here is our application how to work:
We have one thread to write data to bdb, about 200 thousand data per hour, and most of them are actually update actions.
Also we run 2 cleaners to clean bdb.
The setting is as below:
je.maxMemory=1310720000
je.log.fileMax=5242880
je.nodeMaxEntries=1024
je.log.bufferSize=20971520
je.log.numBuffers=5
je.log.totalBufferBytes=104857600
je.cleaner.minUtilization=50
je.cleaner.minFileUtilization=25
je.cleaner.threads=2
je.cleaner.readSize=1024000
je.cleaner.bytesInterval=209715200
je.cleaner.lockTimeout=100000
je.cleaner.expunge=true
## checkpointer
je.checkpointer.highPriority=true
je.checkpointer.bytesInterval=20971520
je.env.backgroundReadLimit=5
je.env.backgroundWriteLimit=5
Does anybody have any suggestion about this issue?
Thanks,
Yu

Answer 1

First, I suggest increasing the number of cleaner threads until the cleaner catches up. This could take quite a long time, since it is so far behind. You may need to set a large number of cleaner threads. Use at least the same number of cleaner threads as there are app threads running concurrently, which access the JE environment. How many app threads are running concurrently?
Second, I suggest changing some of the config settings that are very different from anything we've tested before.
The log file size is small. Many users increase the size, but it is very rare to decrease it. I suggest you remove this parameter.
The nodeMaxEntries setting is extremely high. We've never tested with a setting this high. Please remove this parameter.
You're using 5 log buffers. More than 3 has no benefit, and wastes memory that could be used for the cache. The log buffer size should never be more than the file size, as this is also a waste of memory. Please use only 3 log buffers and set the buffer size no larger than the file size.
--mark