JE - Hangs after a period of time.

Question

Hi,
I am relatively new to the world of Berkeley DB so please forgive my seemingly meandering request for help.
I am writing an application which requires 8 hashtable or map structures. These data structures are prohibitively large and thus I require disk access. Persistence is not an actual requirement of the system, merely that the hashtable structures be arbitrarily scalable. The system I am running is single threaded, reading from a file and storing certain key/value pairs which are continuously read and infrequently updated (once off file analysis).
I have afforded 3G of heapspace to the system and given seven of the hashtables 5% cache size, and a more frequently accessed hashtable 25% cache.
After a given period of time, my application slows from processing approximately 2M input lines per minute to a single input line consistently every five minutes. This has continued for a number of hours. The time is not spent in the read or write methods of the hashtables (calls to which are in single figures) and is not used up by other parts of the code (I have tested the code without the hashtable features and it scales linearly: no slow down). Also, I have checked the garbage collection and it is not a problem with only minor collections running very quickly. CPU usage is minimal.
My question is simply, where is the time being spent? Also can anyone suggest a better configuration? I just want 8 simple hashtables that overflow onto disk as heap space requires. Performance is so bad, I question the utility of tweaking but instead would hope that someone points out any glaring mistakes I have made in my implementation.
The sizes of the hashtables at slow down are as follows:
299462, 174978, 3543871, 304504, 1173250, 557360, 1859511 (5% cache)
50429 (25% cache)
Key size is about 40 bytes. Value size is somewhere in the low hundreds of bytes range.
Version is je-3.2.76
The following is my constructor... No changes are made to the database or environment setup outside of this. The only other interaction with the databases are through simple get() put() methods.
public PersistantHashtable(String rootDir, int cachePercent, Class c) throws IOException, DatabaseException {
DatabaseConfig dbConfig = new DatabaseConfig();
dbConfig.setTransactional(false);
dbConfig.setAllowCreate(true);
dbConfig.setDeferredWrite(true);
Database catalogDb = env.openDatabase(null, "catalog", dbConfig);
ClassCatalog catalog = new StoredClassCatalog(catalogDb);
SerialBinding keyBinding = new SerialBinding(catalog, Node.class);
SerialBinding dataBinding = new SerialBinding(catalog, c);
_db = env.openDatabase(null, name, dbConfig);
ht = new StoredMap(db, keyBinding, dataBinding, true);
Some typical garbage collection output:
[GC 2906686K->2067991K(3040896K), 0.0108480 secs]
[GC 2906903K->2067724K(3040896K), 0.0106010 secs]
Thanks for your time. Hope you can help ;).

Answer 1

Thanks both for your time. I suppose my first direct
question is whether or not it is advisable from a
performance point of view to re-use the same
environment for each of the eight database
structures?Yes, the normal and best performing choice is to have a single environment for all databases.
Secondly, can anyone provide any insights into
whether I can expect more reliable performance from a
StoredMap or StoredSortedMap? I would expect that the
former is based on a HTree implementation and the
latter a BTree. I only need the functionality of the
former, but would be happy to use whichever in
anyone's experience is the more reliable.JE only provides a Btree. The StoredMap and StoredSortedMap have the same underlying storage -- both are in fact sorted since a Btree is always used.
My third question is as to whether the cache percent
parameter is reasonably well behaved and implemented
using strong references, or whether it still relies
on soft/weak references to be reclaimed by garbage
collection. Previous incarnations of the code got
hammered by garbage collection, so I would almost
prefer a system with little or no cache as opposed to
one with high object churn that hammers performance
because of garbage collection.The JE cache uses strong references. No weak/soft refs are used.
On the topic of using top to identify the
problem, I have found that WA is very high during the
slowdown, while CPU is extremely low. Thus, the wait
time for IO operation is very high. I have also
noticed that the slowdown also occurs earlier in the
file analysis, with irregular pauses earlier in the
job becoming more frequent as the job progresses. My
final question is: what operation is being triggered
in the database at these points in time? Are some
clean-up process running on the database when it
believes it to be idle and would it be
possible/advisable to turn it off?What is your JE cache size? I suspect it's much too small for what you're doing.
I will run some env.getStats(..) and see if I can
find out anything more.Please do print out the env stats after the slowdown occurs and post them.
Thanks,
--mark