Need suggestions for concurrent reads while deleting all entities

Question

Hello,
we need some suggestions for the following use case, as we cannot seem to find the correct transaction locking combination to meet our needs.
We are using BDB JE 4.1.10, and the entityStore is transactional. In a nutshell, BDB JE is used as a Cache for specific data that is pulled and updated regularly from a source Oracle DB. The application that holds BDB is a realtimel app, and response time is critical.
In order to avoid having to merge (Insert/update/delete) entities for some of the tables that are very static and small (a few hundred records that might change every day or so), we are trying to simply delete all records (with an EntityCursor loop, since there is no 'deleteAll' that we could find) and reinsert all data every 5 minutes, and then committing when the whole process is complete. This should always be very quick, but we are not immune to timeouts from the source Oracle DB, so the transaction can be long.
Ideally, while the delete/insert is happening, we want any concurrent read operations to return the old data (before the delete). We have made a test that locks the updater thread between the 'delete all' and 'update all' methods, so that the delete cursor is closed, but the Tx is not yet commited. We have tried the following for our reader thread:
1- If we get() a deleted entity with a LockMode.READ_UNCOMMITTED, we get a null entity back.
2- If we get() a deleted entity with LockMode.READ_COMMITTED, we get a LockTimeoutException.
We have also considered simply using truncate on the EntityStore, but this requires closing and reopening the DB, so this would not work for our need to always return a value, instantly.
Any ideas, suggestions on how we could do this in a simple manner?
Thanks,
Max

Answer 1

Hello Max,
I understand the issue you're describing.
I think the best performing solution by far is to load the new data into new databases while servicing reads from the old databases. When the load is complete, reads can be diverted to the new databases, and the old databases can then be removed.
Unfortunately, this is easier to do with the base API than with the DPL, because the DPL is managing the underlying (base API level) databases for you.
However, even using the DPL, this approach could be very straightforward if you happen to be keeping this data set (the one that needs to be reloaded) in a separate EntityStore, or you can change your application to do so. If so, you can create a new (empty) EntityStore with a different name and use this store for loading the new data. When the load is complete, divert reads to the new EntityStore object and remove all databases for the old EntityStore.
If this is impractical, please explain, and I'll try to suggest a different solution. If you cannot use a separate EntityStore for this particular data set, then the solution may be more complex.
--mark