Facing low Performance when iterating records of database using cursor

Question

Hi ,
i inserted nearly 80,000,000 records into a database, by reading a file whose size is nealry 800MB, in 10 minutes.
when i am iterating the records using Cursor with Default lock mode , it is taking nearly 1 hour.
My Bdb details are as follows
Environment : Non transactional , non locking
Database : Deferred write.
Cache : 80% of JVM ( -Xms=1000M -Xmx=1200m )
Could you please explain why it is taking such a long time ? did i make any mistakes on settings ?
Thanks
nvseenu
Edited by: nvseenu on Jan 15, 2009 5:47 AM

Answer 1

Hello Gary,
StoredMap is a convenience API wrapper for a Database. It has the same performance and multi-threading characteristics as a Database. You don't need to synchronize a StoredMap, or use Database to get better performance.
The lock conflicts are the thing to focus on here. This is unrelated to the topic discussed earlier in this thread.
How many threads are inserting and how many performing queries?
What other work, other than inserting and reading, are these threads performing?
Does any thread keep an iterator (which is a cursor) open?
How large are the data items in the map?
What is the resolution of the timestamp? Milliseconds?
I don't think the exception you posted is complete. Please post the full exception including the cause exception.
I can't tell from the exception but it looks like multiple insertion threads are conflicting with each other, not with the query threads. If you test only the insertions (no queries), do the lock conflicts still occur?
One possibility is that multiple insertions threads are using the same timestamp as the key. Only one thread will be able to access that key at a time, the others will wait. Even so, I don't understand why it's taking so long to perform the inserts. But you can easily make the key unique by appending a sequence number -- use a two part key {timestamp, sequence}.
Please upgrade to JE 3.3 in order to take advantage of the improvements and so we can better support you. We're not actively working with JE 3.2, which is very outdated now.
--mark