Performance DB vs Java Edition

Question

Hi there,
we've been investigating wether a (and when which) NoSQL-DB outperforms a standard SQL solution for our usecase:
500.000 records bulk insert
2k - 5k per record
With 2 Indices
Using Java API
Multi OS: Linux, Windows, z/OS
Berkeley DB performed best (bulk insert, selects and deletes) on our Windows-Tests (with 2GB memory).
Unfortunately theres a huge gap between Berkley DB and its Java Edition performance.
For a 20.000 record chunk insert Berkeley DB (using BTREE) takes 2000 - 3500 ms. When the cache is full and the first cache misses appear the inserting time grows and stays at arround 8000 ms. After a while st_cacheMiss decreases again resulting in 2000 - 3500 ms per insert.
The JE starts with 4500 - 5000 ms per insert. After the first 200.000 records the cache is full and the nCacheMiss grows rapidly along with the inserting time (11s, 13s, 16s, ... 30s and even longer). There is no constancy like the non Java version shows to have.
Is there any way to reach the Berkeley DB's performance with the JE? Esp. the constant insert time.
We've tried already different CacheModes but default mode is the best.
Setting je.log.faultReadSize=5120 leads to a slighter better perfomance, but still the inserting gets slower and slower.
Since we need z/OS support we'd prefer to use the JE.
Thanks for any help.
Jonas

Answer 1

Hello Jonas,
Glad you found the main problem.
I don't find it's productive to compare the two products' performance as they are so different, but I can help with tuning BDB JE.
The first step is to take a look at the performance section of the FAQ:
http://www.oracle.com/technetwork/database/berkeleydb/je-faq-096044.html
If you have further questions on tuning please be sure to post:
+ Your EnvironmentConfig settings and je.properties (if any)
+ Your DatabaseConfig or StoreConfig settings.
+ A couple dumps of the EnvironmentStats covering the performance test time period, and be sure to clear the stats each time with StatsConfig.setClear(true).
This is just the starting info for doing any sort of tuning with JE.
I'm surprised that CacheMode.EVICT_LN didn't help, since moving the LNs out of cache immediately should reduce the burden on the evictor and Java GC. If you don't have enough cache to hold all LNs (see the FAQ entry on DbCacheSize), then EVICT_LN is usually beneficial. If you use EVICT_LN, be sure to also set EnvironmentConfig.CLEANER_LAZY_MIGRATION to false. Lazy migration retains LNs in cache and partially negates the benefits of EVICT_LN.
As described in the FAQ, be sure your cache size is large enough to hold the BINs.
The deferred-write mode is sometimes a win for bulk loading, but if you are doing pure insertions it may not help.
I suggest enlarging these ENVIRONMENT_CONFIG settings if you haven't already:
CLEANER_LOOK_AHEAD_CACHE_SIZE - try 1MB
CLEANER_READ_SIZE - try 1MB
See Durability if your database/store is transactional. Using NO_SYNC durability is much faster of course, but there are trade-offs. NO_SYNC is the default for a non-transactional database/store.
--mark