LogFileNotFoundException from one index but not another in DPL

Question

Hi folks,
I have an entity type in the DPL called an Item. It has various secondary keys. When I try to use the "type" of the item to get all items of a particular type, I get a LogFileNotFoundException (the log file is clearly gone, and we have no processes that could delete them):
EntityIndex index = itemByType.subIndex(type.ordinal());
EntityCursor<ItemImpl> cur = index.entities();
try {
    for(ItemImpl i : cur) {
        items.add(i);
} finally {
    cur.close();
}The exception occurs in the implicit hasNext in the for loop:
Caused by: com.sleepycat.je.DatabaseException: (JE 3.3.75) fetchTarget of 0xf/0x6ccfd parent IN=80127 IN class=com.sleepycat.je.tree.DBIN lastFullVersion=0x21/0x61c37 parent.getDirty()=false state=0 com.sleepycat.je.log.LogFileNotFoundException: (JE 3.3.75) 0xf/0x6ccfd (JE 3.3.75) Couldn't open file /files/data/db/0000000f.jdb: /files/data/db/0000000f.jdb (No such file or directory)
     at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1230)
     at com.sleepycat.je.dbi.CursorImpl.fetchCurrent(CursorImpl.java:2320)
     at com.sleepycat.je.dbi.CursorImpl.getCurrentAlreadyLatched(CursorImpl.java:1427)
     at com.sleepycat.je.dbi.CursorImpl.getNextDuplicate(CursorImpl.java:1745)
     at com.sleepycat.je.Cursor.retrieveNextAllowPhantoms(Cursor.java:2187)
     at com.sleepycat.je.Cursor.retrieveNext(Cursor.java:1991)
     at com.sleepycat.je.SecondaryCursor.retrieveNext(SecondaryCursor.java:1497)
     at com.sleepycat.je.SecondaryCursor.getNextDup(SecondaryCursor.java:676)
     at com.sleepycat.util.keyrange.RangeCursor.doGetNextDup(RangeCursor.java:907)
     at com.sleepycat.util.keyrange.RangeCursor.getNext(RangeCursor.java:445)
     at com.sleepycat.persist.BasicCursor.next(BasicCursor.java:81)
     at com.sleepycat.persist.BasicIterator.hasNext(BasicIterator.java:50)Curiously, if I get the same data by using a different index (one indexed by a type/time composite key), I don't hit any exceptions.
Some possibly relevant data:
This occurred shortly after upgrading from .69 to .75. I didn't deploy the update, but I'm told that it worked initially after the update, but then the service was restarted a second time a day or so later (for other unrelated changes) and this exception started to happen sometime after the second restart.
Restarting the process doesn't seem to change anything.
There are about 64K "items" in the database, and about 66K of other types.
We have 15 other databases with the same schema but different data (and roughly the same amount of data as this one). They are all functioning fine, and have had roughly equal amount of load/churn on them as the corrupt database.
The log file in question exists for most of our other databases, but is missing in a few (which I think should mean that it is reasonable to expect that it could have been "garbage collected" on the corrupt database).
The database seems to be about 195MB so it is somewhat portable.
So I have two questions:
1) Is this likely to be a result of something that I did in my code? Obviously the snippet above didn't use a transaction explicitly, but most other code does, including any writes. (And my understanding is that an implicit transaction is used in that snippet if the DB is transactional otherwise)
2) Assuming that the log file in question wasn't spuriously deleted by some log file gnome, is there any way that we can recover from this in general? This may be the wrong question, but is it possible to rebuild an index that thinks an old log file is still there?
Thanks!
Jeff

Answer 1

Hello all,
I'd like to let everyone know that the problem reported by Jeff is the result of a JE bug that was introduced in JE 3.3.62 and recently fixed in JE 3.3.76. JE 3.3.76 is not yet published, and we don't have a date doing that yet, but if would like a copy of it now, please send me email -- mark.hayes at the obvious (oracle) .com.
Below is the change log entry for the bug. Although the bug occurs rarely, any app that configures duplicates (or a MANY_TO_XXX secondary key relationship) could be impacted. The result of the bug is data loss, and reverting to a backup is necessary.
Change log entry:
Fix a bug that caused LogFileNotFoundException in rare circumstances for an Environment having one or more Databases configured for duplicates (or one or more SecondaryIndexes with MANY_TO_XXX relationships). The bug only occurs when a single secondary/duplicate key value is associated with a large number of records/entities; specifically, the sub-Btree for a single duplicate key value must have at least 3 levels. SR #16712
Thanks, Jeff, for your help on this.
--mark