Weird error: "Cannot write file summary in a read-only environment"

Question

Disclaimer: We are using an old version of bdb - 3.2.76 but are not able to easily upgrade.
We see this occasionally in production. From looking at the code it looks like critical eviction is happening since more memory is used then the limit we set then eviction is happening but then failing since the env is read only. I'm just wondering how it got to this point. we set the max size (je.maxMemory) to 1572864 bytes.
This is not urgent but is adding noise to logs so any suggestions would be welcome...
<application specific exception>
…causedBy com.sleepycat.je.DatabaseException: (JE 3.2.76) Cannot write file summary in a read-only environment
at com.sleepycat.je.cleaner.UtilizationProfile.putFileSummary(UtilizationProfile.java:720)
at com.sleepycat.je.cleaner.UtilizationProfile.flushFileSummary(UtilizationProfile.java:708)
at com.sleepycat.je.cleaner.UtilizationTracker.evictMemory(UtilizationTracker.java:120)
at com.sleepycat.je.evictor.Evictor.evictBatch(Evictor.java:314)
at com.sleepycat.je.evictor.Evictor.doEvict(Evictor.java:253)
at com.sleepycat.je.evictor.Evictor.doCriticalEviction(Evictor.java:279)
at com.sleepycat.je.dbi.CursorImpl.close(CursorImpl.java:690)
at com.sleepycat.je.dbi.CursorImpl.close(CursorImpl.java:660)
at com.sleepycat.je.Cursor.endRead(Cursor.java:1820)
at com.sleepycat.je.Cursor.retrieveNextAllowPhantoms(Cursor.java:1616)
at com.sleepycat.je.Cursor.retrieveNext(Cursor.java:1397)
at com.sleepycat.je.Cursor.getNext(Cursor.java:456)
...

Answer 1

Hi Nick,
It is strange to me also that this is happening in a read-only env. Looking at the code, we don't prevent this sort of eviction in the 3.2 release or later releases. Yet we've never seen this problem before.
This sort of eviction only happens when writing is occurring. Of course, in a read-only env there is no writing. The only way I can think of for this to occur in a read-only env is when there is a very large recovery interval -- the amount of log between the last checkpoint start and the end of the log. This can occur if you didn't close the env cleaning when you last wrote to it, or if the last checkpoint just happened to be very large.
So one thing you could try is to ensure that you have a small checkpoint when you write to the log, before opening it in read-only mode. To do this, just before you call Environment.close, call Environment.checkpoint(new CheckpointConfig().setForce(true)). This will do one checkpoint before closing the env, and then another during the close. The last checkpoint should be small.
If you can't do this, or it isn't sufficient, you could increase the je.cleaner.detailMaxMemoryPercentage env config property. This is the max amount of memory (% of total cache) used for the information that is being evicted. Since nothing can really be evicted in read-only mode, there is no harm in setting this to a much higher value than the default (2). I would try 10. This doesn't reserve or waste any memory, it just prevents this sort of eviction from occurring until a larger part of the cache is used. This is probably the simplest solution.
A third approach is to simply accept that this sort of eviction will be attempted, but try to prevent the exception. If you're willing to rebuild JE (and you're not a supported Oracle customer -- please contact us if you are), then you could simply replace the 'throws' statement with a 'return null' in src/com/sleepycat/je/cleaner/UtilizationProfile.java:
    private synchronized PackedOffsets putFileSummary(TrackedFileSummary tfs)
        throws DatabaseException {
        if (env.isReadOnly()) {
            throw new DatabaseException
                ("Cannot write file summary in a read-only environment");
        }--mark