Takes too long to open a database

Question

Dear experts,
we are trying to load a lot of data into BDB JE 4.1.10, and at the moment total size of all log files is about 306Gb (~ 55mln records with keys length 15-20 bytes and values 10-30KB), total number of files is 1570.
Usage pattern: big batch updates once a week, all other time read-only with random reads.
So far the experience of using BDB JE is very good, but one annoying thing is that it takes ~ 30 min to open the database (new Environment(...) takes only 10 seconds, but dbEnv.openDatabase(...) is very slow). It is slow even after a clean shutdown, and even when there were previously no updates at all. Looking at the iostat, the application is very busy with reading from disk.
Is there a way to speed opening the database up? Or may be it is a reasonable time for such a big database?
Settings:
envConfig.setConfigParam(EnvironmentConfig.LOG_FILE_MAX, "" + (200 * MB));
envConfig.setConfigParam(EnvironmentConfig.LOG_FILE_CACHE_SIZE, "500");
envConfig.setConfigParam(EnvironmentConfig.LOG_WRITE_QUEUE_SIZE, "" + (24 * MB));
envConfig.setConfigParam(EnvironmentConfig.LOG_BUFFER_SIZE, "" + (8 * MB));
envConfig.setConfigParam(EnvironmentConfig.CHECKPOINTER_BYTES_INTERVAL, "" + (100 * MB));
envConfig.setConfigParam(EnvironmentConfig.CLEANER_MIN_AGE, "10");
envConfig.setConfigParam(EnvironmentConfig.CLEANER_MAX_BATCH_FILES, "10");
envConfig.setConfigParam(EnvironmentConfig.CLEANER_READ_SIZE, "" + (4 * MB));
envConfig.setConfigParam(EnvironmentConfig.CLEANER_LOOK_AHEAD_CACHE_SIZE, "" + (2 * MB));
envConfig.setConfigParam(EnvironmentConfig.LOG_FAULT_READ_SIZE, "" + (24 * KB));
envConfig.setConfigParam(EnvironmentConfig.LOG_ITERATOR_READ_SIZE, "" + (128 * KB));
envConfig.setConfigParam(EnvironmentConfig.EVICTOR_LRU_ONLY, "false");
envConfig.setConfigParam(EnvironmentConfig.EVICTOR_NODES_PER_SCAN, "200");
System: SUSE with 32Gb onboard and 2CPUs with 4 cores each, RAID0 on 4 disks 7200rpm. BDB JE has 75% for cache from 10GB available for the application.
Thanks in advance!

Answer 1

Hi ambber,
Thanks for your suggestion. A counter that must be updated atomically and logged periodically by JE would be accurate, but would add the same sort of complexity to JE that you mention was added to your application. More importantly, it would only address one type of aggregate "count" that an application may be interested in. Other applications, for example, need to find the number of records in a key range, where the range end points are not predetermined.
Another possibility is for JE to provide a very rough estimate of the record count, based on the depth of the Btree and the number of internal nodes below the root level of the tree. For example, if the tree is 5 levels deep, the tree has N nodes under the root level, 128 is the maximum size of each node, and nodes are on average 75% full, then the total number of records is very roughly:
N * ((128 * 0.75) ^^ 4)The advantage of this approach is that it requires no additional data to be logged, does not reduce concurrency, and a similar approach can be applied to a key range as well the entire database.
Because it would be a very rough estimate, it would mainly be useful for determining optimal query strategies.
How are you using the record count?
Just a thought.
--mark