Question about Berkeley DB Primary Index/Secondary Index in DPL package

Question

We are using DPL package for BDB access.
My question is about memory consumption by Primary or Secondary Index. We are using BDB only for Read Purposes
e.g. Let's say if we are using BDB to store 250GB of Raw Data (which will translate to .jdb files which will have all indexes). Each box is having around 32GB memory.
1) So when we call method store.getPrimaryIndex(....) or Create Secondary Index; how much size these Indexes will take up in memory ?
2) Is it advisable to keep these Indexes in memory all the time ?
Thanks,

Answer 1

I have a couple caveats to add on DbCacheSize.
1) It is an estimate, and for ordinary databases (without duplicate keys) is an upper bound.
2) For databases with duplicate keys (in the DPL these are MANY_TO_ONE and MANY_TO_MANY secondary indices), it is incorrect and can be used as a lower bound.
So it gives only a rough estimate. If you need an accurate number, write a DPL-based program that creates a realistic data set and look at EnvironmentStats.getCacheTotalBytes to see how much memory is used prior to any eviction occurring, which is indicated byEnvironmentStats.getNEvictPasses. Use a machine that has as much memory as possible, set the Java heap size and the JE cache size to the maximum possible, and create the data set as close as possible to the size you expect in production.
2) Is it advisable to keep these Indexes in memory all the time ?Yes. The FAQ has multiple entries on this topic in the performance section.
--mark