EnvironmentFailureException
Hello!
I've upgraded my systems to use 6.0.11 instead of 5.0.97. Everything works fine, storage is 20+ percent more compact and so on, BUT.
1) Under load (all the time) on production server, after every 25-40 minutes uptime the
> (JE 6.0.11) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 6.0.11) /usr/local/ae3/private/data/bdbj-lcl java.lang.AssertionError UNCAUGHT_EXCEPTION: Uncaught Exception in internal thread, unable to continue. Environment is invalid and must be closed.
> com.sleepycat.je.EnvironmentFailureException
> Environment invalid because of previous exception: (JE 6.0.11) /usr/local/ae3/private/data/bdbj-lcl java.lang.AssertionError UNCAUGHT_EXCEPTION: Uncaught Exception in internal thread, unable to continue. Environment is invalid and must be closed.
> com.sleepycat.je.EnvironmentFailureException
> null
> java.lang.AssertionError
: com.sleepycat.je.evictor.LRUEvictor.processTarget(LRUEvictor.java:2014)
: com.sleepycat.je.evictor.LRUEvictor.findParentAndRetry(LRUEvictor.java:2182)
: com.sleepycat.je.evictor.LRUEvictor.processTarget(LRUEvictor.java:2019)
: com.sleepycat.je.evictor.LRUEvictor.evictBatch(LRUEvictor.java:1689)
: com.sleepycat.je.evictor.LRUEvictor.doEvict(LRUEvictor.java:1538)
: com.sleepycat.je.evictor.Evictor$BackgroundEvictTask.run(Evictor.java:739)
: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
: java.lang.Thread.run(Thread.java:724)
error appears and floods everything, I have to restart the instance and then it works another 30 minutes approx.
Same thing was happening on non heavy loaded (all) servers during log file upgrade to new format, timing was random tough. None of the environments that are bigger than 1G size were upgraded 'in one go' - I had to restart several times because of the same error.
2) I had to reduce the number of cleaner threads to 1 (config.setConfigParam( EnvironmentConfig.CLEANER_THREADS, "1" )) - otherwise it was not starting AT ALL, on every database instance it was failing on with words like "expect BEING_CLEANED but CLEANED"
I am using low-level BDB JE API with cursors and byte arrays. No secondary databases. No DPL. Does anyone experiencing anything like that?
2nd issue regarding @ I had to reduce the number of cleaner threads to 1 (config.setConfigParam( EnvironmentConfig.CLEANER_THREADS, "1" )) - otherwise it was not starting AT ALL, on every database instance it was failing on with words like "expect BEING_CLEANED but CLEANED"@ is 100% reproducible.
The main issue with com.sleepycat.je.evictor.LRUEvictor is happening on one production server currently (57G data, 3/4 of that rotates per week (new data added, old data cleaned)), but it was happening on other servers while upgrading from previous log format.
My workaround was:
if (environment != null && !environment.isValid()) {
WorkerBdbj.LOG.event( "BDBJ-WORKER:FAILURE:FATAL",
"Environment is invalid!",
Convert.Throwable.toText( new IllegalStateException( "this:" + this + ", env:" + environment ) ) );
try {
environment.close();
} catch (final Throwable t) {
// ignore
Runtime.getRuntime().exit( -37 );
Thanks for -da:com.sleepycat.je.evictor.LRUEvictor hint, anyway I wouldn't be able to assert it is safe to do so without your reply!
Several times I noticed different stack traces (normally it is like in initial post).
> (JE 6.0.11) JAVA_ERROR: Java Error occurred, recovery may not be possible.
> com.sleepycat.je.EnvironmentFailureException
> null
> java.lang.AssertionError
: com.sleepycat.je.evictor.LRUEvictor.processTarget(LRUEvictor.java:2014)
: com.sleepycat.je.evictor.LRUEvictor.findParentAndRetry(LRUEvictor.java:2182)
: com.sleepycat.je.evictor.LRUEvictor.processTarget(LRUEvictor.java:2019)
: com.sleepycat.je.evictor.LRUEvictor.evictBatch(LRUEvictor.java:1689)
: com.sleepycat.je.evictor.LRUEvictor.doEvict(LRUEvictor.java:1538)
: com.sleepycat.je.evictor.Evictor.doCriticalEviction(Evictor.java:469)
: com.sleepycat.je.dbi.EnvironmentImpl.criticalEviction(EnvironmentImpl.java:2726)
: com.sleepycat.je.dbi.CursorImpl.criticalEviction(CursorImpl.java:624)
: com.sleepycat.je.Cursor.beginMoveCursor(Cursor.java:4217)
: com.sleepycat.je.Cursor.beginMoveCursor(Cursor.java:4237)
: com.sleepycat.je.Cursor.searchAllowPhantoms(Cursor.java:2795)
: com.sleepycat.je.Cursor.searchNoDups(Cursor.java:2647)
: com.sleepycat.je.Cursor.search(Cursor.java:2594)
: com.sleepycat.je.Cursor.search(Cursor.java:2579)
: com.sleepycat.je.Cursor.getSearchKey(Cursor.java:1698)
> (JE 6.0.11) JAVA_ERROR: Java Error occurred, recovery may not be possible.
> com.sleepycat.je.EnvironmentFailureException
> null
> java.lang.AssertionError
: com.sleepycat.je.evictor.LRUEvictor.processTarget(LRUEvictor.java:2014)
: com.sleepycat.je.evictor.LRUEvictor.findParentAndRetry(LRUEvictor.java:2182)
: com.sleepycat.je.evictor.LRUEvictor.processTarget(LRUEvictor.java:2019)
: com.sleepycat.je.evictor.LRUEvictor.evictBatch(LRUEvictor.java:1689)
: com.sleepycat.je.evictor.LRUEvictor.doEvict(LRUEvictor.java:1538)
: com.sleepycat.je.evictor.Evictor.doCriticalEviction(Evictor.java:469)
: com.sleepycat.je.dbi.EnvironmentImpl.criticalEviction(EnvironmentImpl.java:2726)
: com.sleepycat.je.dbi.CursorImpl.criticalEviction(CursorImpl.java:624)
: com.sleepycat.je.dbi.CursorImpl.close(CursorImpl.java:583)
: com.sleepycat.je.Cursor.endMoveCursor(Cursor.java:4269)
: com.sleepycat.je.Cursor.searchAllowPhantoms(Cursor.java:2811)
: com.sleepycat.je.Cursor.searchNoDups(Cursor.java:2647)
: com.sleepycat.je.Cursor.search(Cursor.java:2594)
: com.sleepycat.je.Cursor.search(Cursor.java:2579)
: com.sleepycat.je.Cursor.getSearchKeyRange(Cursor.java:1757)
Similar Messages
-
BDB v5.0.73 - EnvironmentFailureException: (JE 5.0.73) JAVA_ERROR
Hi there!
Im using java berkeley db as caching tool between two applications (backend application is quite slow so we cache some requests). We are caching xml data (one request xml can be ~4MB), if xml data exists we update the cache entry, if does not exists we insert new entry. there can be many concurrent hits same time.
Currently I tested with JMeter and with 10 Threads all works fine but if I increase to 20 Threads following error occurs:
+2013-05-14 15:31:15,914 [ERROR] CacheImpl - error occured while trying to get data from cache.+
com.sleepycat.je.EnvironmentFailureException: (JE 5.0.73) JAVA_ERROR: Java Error occurred, recovery may not be possible. fetchTarget of 0x11/0x1d1d parent IN=8 IN class=com.sleepycat.je.tree.IN lastFullVersion=0x1b/0x4cd lastLoggedVersion=0x1b/0x4cd parent.getDirty()=false state=0 fetchTarget of 0x11/0x1d1d parent IN=8 IN class=com.sleepycat.je.tree.IN lastFullVersion=0x1b/0x4cd lastLoggedVersion=0x1b/0x4cd parent.getDirty()=false state=0 fetchTarget of 0x11/0x1d1d parent IN=8 IN class=com.sleepycat.je.tree.IN lastFullVersion=0x1b/0x4cd lastLoggedVersion=0x1b/0x4cd parent.getDirty()=false state=0 fetchTarget of 0x11/0x1d1d parent IN=8 IN class=com.sleepycat.je.tree.IN lastFullVersion=0x1b/0x4cd lastLoggedVersion=0x1b/0x4cd parent.getDirty()=false state=0 fetchTarget of 0x11/0x1d1d parent IN=8 IN class=com.sleepycat.je.tree.IN lastFullVersion=0x1b/0x4cd lastLoggedVersion=0x1b/0x4cd parent.getDirty()=false state=0 fetchTarget of 0x11/0x1d1d parent IN=8 IN class=com.sleepycat.je.tree.IN lastFullVersion=0x1b/0x4cd lastLoggedVersion=0x1b/0x4cd parent.getDirty()=false state=0 fetchTarget of 0x11/0x1d1d parent IN=8 IN class=com.sleepycat.je.tree.IN lastFullVersion=0x1b/0x4cd lastLoggedVersion=0x1b/0x4cd parent.getDirty()=false state=0 fetchTarget of 0x11/0x1d1d parent IN=8 IN class=com.sleepycat.je.tree.IN lastFullVersion=0x1b/0x4cd lastLoggedVersion=0x1b/0x4cd parent.getDirty()=false state=0 fetchTarget of 0x11/0x1d1d parent IN=8 IN class=com.sleepycat.je.tree.IN lastFullVersion=0x1b/0x4cd lastLoggedVersion=0x1b/0x4cd parent.getDirty()=false state=0 fetchTarget of 0x11/0x1d1d parent IN=8 IN class=com.sleepycat.je.tree.IN lastFullVersion=0x1b/0x4cd lastLoggedVersion=0x1b/0x4cd parent.getDirty()=false state=0 fetchTarget of 0x11/0x1d1d parent IN=8 IN class=com.sleepycat.je.tree.IN lastFullVersion=0x1b/0x4cd lastLoggedVersion=0x1b/0x4cd parent.getDirty()=false state=0 fetchTarget of 0x11/0x1d1d parent IN=8 IN class=com.sleepycat.je.tree.IN lastFullVersion=0x1b/0x4cd lastLoggedVersion=0x1b/0x4cd parent.getDirty()=false state=0
+ at com.sleepycat.je.dbi.EnvironmentImpl.checkIfInvalid(EnvironmentImpl.java:1507)+
+ at com.sleepycat.je.Environment.checkEnv(Environment.java:2185)+
+ at com.sleepycat.je.Environment.beginTransactionInternal(Environment.java:1313)+
+ at com.sleepycat.je.Environment.beginTransaction(Environment.java:1284)+
+ at com.ebcont.redbull.bullchecker.cache.impl.CacheImpl.get(CacheImpl.java:157)+
+ at com.ebcont.redbull.bullchecker.handler.EndpointHandler.doPerform(EndpointHandler.java:132)+
+ at com.ebcont.redbull.bullchecker.WSCacheEndpointServlet.doPost(WSCacheEndpointServlet.java:86)+
+ at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)+
+ at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)+
+ at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)+
+ at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)+
+ at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)+
+ at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)+
+ at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)+
+ at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)+
+ at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)+
+ at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)+
+ at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)+
+ at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606)+
+ at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)+
+ at java.lang.Thread.run(Unknown Source)+
Caused by: java.lang.OutOfMemoryError: Java heap space
+2013-05-14 15:31:15,939 [ERROR] CacheImpl - error occured while trying to get data from cache.+
com.sleepycat.je.EnvironmentFailureException: (JE 5.0.73) JAVA_ERROR: Java Error occurred, recovery may not be possible. fetchTarget of 0x11/0x1d1d parent IN=8 IN class=com.sleepycat.je.tree.IN lastFullVersion=0x1b/0x4cd lastLoggedVersion=0x1b/0x4cd parent.getDirty()=false state=0 fetchTarget of 0x11/0x1d1d parent IN=8 IN class=com.sleepycat.je.tree.IN lastFullVersion=0x1b/0x4cd lastLoggedVersion=0x1b/0x4cd parent.getDirty()=false state=0 fetchTarget of 0x11/0x1d1d parent IN=8 IN class=com.sleepycat.je.tree.IN lastFullVersion=0x1b/0x4cd lastLoggedVersion=0x1b/0x4cd parent.getDirty()=false state=0 fetchTarget of 0x11/0x1d1d parent IN=8 IN class=com.sleepycat.je.tree.IN lastFullVersion=0x1b/0x4cd lastLoggedVersion=0x1b/0x4cd parent.getDirty()=false state=0 fetchTarget of 0x11/0x1d1d parent IN=8 IN class=com.sleepycat.je.tree.IN lastFullVersion=0x1b/0x4cd lastLoggedVersion=0x1b/0x4cd parent.getDirty()=false state=0 fetchTarget of 0x11/0x1d1d parent IN=8 IN class=com.sleepycat.je.tree.IN lastFullVersion=0x1b/0x4cd lastLoggedVersion=0x1b/0x4cd parent.getDirty()=false state=0 fetchTarget of 0x11/0x1d1d parent IN=8 IN class=com.sleepycat.je.tree.IN lastFullVersion=0x1b/0x4cd lastLoggedVersion=0x1b/0x4cd parent.getDirty()=false state=0 fetchTarget of 0x11/0x1d1d parent IN=8 IN class=com.sleepycat.je.tree.IN lastFullVersion=0x1b/0x4cd lastLoggedVersion=0x1b/0x4cd parent.getDirty()=false state=0 fetchTarget of 0x11/0x1d1d parent IN=8 IN class=com.sleepycat.je.tree.IN lastFullVersion=0x1b/0x4cd lastLoggedVersion=0x1b/0x4cd parent.getDirty()=false state=0 fetchTarget of 0x11/0x1d1d parent IN=8 IN class=com.sleepycat.je.tree.IN lastFullVersion=0x1b/0x4cd lastLoggedVersion=0x1b/0x4cd parent.getDirty()=false state=0 fetchTarget of 0x11/0x1d1d parent IN=8 IN class=com.sleepycat.je.tree.IN lastFullVersion=0x1b/0x4cd lastLoggedVersion=0x1b/0x4cd parent.getDirty()=false state=0 fetchTarget of 0x11/0x1d1d parent IN=8 IN class=com.sleepycat.je.tree.IN lastFullVersion=0x1b/0x4cd lastLoggedVersion=0x1b/0x4cd parent.getDirty()=false state=0
+ at com.sleepycat.je.dbi.EnvironmentImpl.checkIfInvalid(EnvironmentImpl.java:1507)+
+ at com.sleepycat.je.Environment.checkEnv(Environment.java:2185)+
+ at com.sleepycat.je.Environment.beginTransactionInternal(Environment.java:1313)+
+ at com.sleepycat.je.Environment.beginTransaction(Environment.java:1284)+
+ at com.ebcont.redbull.bullchecker.cache.impl.CacheImpl.get(CacheImpl.java:157)+
+ at com.ebcont.redbull.bullchecker.handler.EndpointHandler.doPerform(EndpointHandler.java:132)+
+ at com.ebcont.redbull.bullchecker.WSCacheEndpointServlet.doPost(WSCacheEndpointServlet.java:86)+
+ at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)+
+ at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)+
+ at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)+
+ at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)+
+ at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)+
+ at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)+
+ at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)+
+ at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)+
+ at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)+
+ at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)+
+ at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)+
+ at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606)+
+ at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)+
+ at java.lang.Thread.run(Unknown Source)+
after restarting the server I get following error while trying to get data from cache:
java.lang.OutOfMemoryError: Java heap space
+ at com.sleepycat.je.log.LogUtils.readBytesNoLength(LogUtils.java:365)+
+ at com.sleepycat.je.tree.LN.readFromLog(LN.java:786)+
+ at com.sleepycat.je.log.entry.LNLogEntry.readBaseLNEntry(LNLogEntry.java:196)+
+ at com.sleepycat.je.log.entry.LNLogEntry.readEntry(LNLogEntry.java:130)+
+ at com.sleepycat.je.log.LogManager.getLogEntryFromLogSource(LogManager.java:1008)+
+ at com.sleepycat.je.log.LogManager.getLogEntry(LogManager.java:848)+
+ at com.sleepycat.je.log.LogManager.getLogEntryAllowInvisibleAtRecovery(LogManager.java:809)+
+ at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1412)+
+ at com.sleepycat.je.tree.BIN.fetchTarget(BIN.java:1251)+
+ at com.sleepycat.je.dbi.CursorImpl.fetchCurrent(CursorImpl.java:2261)+
+ at com.sleepycat.je.dbi.CursorImpl.getCurrentAlreadyLatched(CursorImpl.java:1466)+
+ at com.sleepycat.je.dbi.CursorImpl.getNext(CursorImpl.java:1593)+
+ at com.sleepycat.je.Cursor.retrieveNextAllowPhantoms(Cursor.java:2924)+
+ at com.sleepycat.je.Cursor.retrieveNextNoDups(Cursor.java:2801)+
+ at com.sleepycat.je.Cursor.retrieveNext(Cursor.java:2775)+
+ at com.sleepycat.je.Cursor.getNextNoDup(Cursor.java:1244)+
+ at com.ebcont.redbull.bullchecker.cache.impl.BDBCacheImpl.getStoredKeys(BDBCacheImpl.java:244)+
+ at com.ebcont.redbull.bullchecker.CacheStatisticServlet.doPost(CacheStatisticServlet.java:108)+
+ at com.ebcont.redbull.bullchecker.CacheStatisticServlet.doGet(CacheStatisticServlet.java:74)+
+ at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)+
+ at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)+
+ at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)+
+ at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)+
+ at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)+
+ at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)+
+ at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)+
+ at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)+
+ at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)+
+ at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)+
+ at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)+
+ at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606)+
+ at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)+
my bdb configuration:
environmentConfig.setReadOnly(false);
databaseConfig.setReadOnly(false);
environmentConfig.setAllowCreate(true);
databaseConfig.setAllowCreate(true);
environmentConfig.setTransactional(true);
databaseConfig.setTransactional(true);
environmentConfig.setCachePercent(60);
environmentConfig.setLockTimeout(2000, TimeUnit.MILLISECONDS);
environmentConfig.setCacheMode(CacheMode.DEFAULT);
environment path: C:/tmp/berkeleydb
Tomcat JVM Parameters: Initial Memory Pool: 1024
Maximum Memory Pool: 2048
Server: Windows Server 2008
Memory: 8 GB
Edited by: 1005842 on 14.05.2013 07:22
Edited by: 1005842 on 14.05.2013 07:23
Edited by: 1005842 on 14.05.2013 07:37Hi,
The stack trace shows an OOME error due to running out of heap space.
Could you detail what is the exact Java version you are using, on what OS, and what are the JVM options, in particular the max heap size (-Xmx), you are using?
Also, what is the JE cache size you use? (if you do not set any of the MAX_MEMORY or MAX_MEMORY_PERCENT then the JE cache size will default to 60% of the JVM max heap size)
You should look into the way you are using transactions, cursors etc. It might be possible that you are using long running transactions that accumulate a large number of locks, or you might be opening more and more transactions without closing/completing them (by aborting or committing them). Is any of this the case for your application? You can check the lock and transaction statistics using Environment.getStats() and respectively Environment.getTransactionStats().
Aside properly ending/closing transactions and cursors, you should also examine your cache statistics to understand the memory profile. See the following documentation sections on this:
http://docs.oracle.com/cd/E17277_02/html/GettingStartedGuide/cachesize.html
http://www.oracle.com/technetwork/database/berkeleydb/je-faq-096044.html#HowcanIestimatemyapplicationsoptimalcachesize
http://www.oracle.com/technetwork/database/berkeleydb/je-faq-096044.html#WhyshouldtheJEcachebelargeenoughtoholdtheBtreeinternalnodes
Regards,
Andrei -
EnvironmentFailureException thrown while recovering the database!
While recovering the database, an EnvironmentFailureException with LOG_FILE_NOT_FOUND was thrown. The exception was thrown after some data was recovered, and the left data can not be recovered because of the EnvironmentFailureException.
I upgraded the je to 4.1.7, but the data still can not be recovered!
Caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 4.0.92) /home/admin/shopcenter/cdncleaner fetchTarget of 0x64/0x3b8f73 parent IN=8811763 IN class=com.sleepycat.je.tree.IN lastFullVersion=0xffffffff/0xffffffff parent.getDirty()=true state=0 LOG_FILE_NOT_FOUND: Log file missing, log is likely invalid. Environment is invalid and must be closed.
at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1241)
at com.sleepycat.je.tree.Tree.searchSubTreeInternal(Tree.java:1858)
at com.sleepycat.je.tree.Tree.searchSubTree(Tree.java:1682)
at com.sleepycat.je.tree.Tree.search(Tree.java:1548)
at com.sleepycat.je.dbi.CursorImpl.searchAndPosition(CursorImpl.java:2054)
at com.sleepycat.je.Cursor.searchInternal(Cursor.java:2088)
at com.sleepycat.je.Cursor.searchAllowPhantoms(Cursor.java:2058)
at com.sleepycat.je.Cursor.search(Cursor.java:1926)
at com.sleepycat.je.Cursor.getSearchKey(Cursor.java:1351)
at com.sleepycat.util.keyrange.RangeCursor.doGetSearchKey(RangeCursor.java:966)
at com.sleepycat.util.keyrange.RangeCursor.getSearchKey(RangeCursor.java:593)
at com.sleepycat.collections.DataCursor.doGetSearchKey(DataCursor.java:571)
at com.sleepycat.collections.DataCursor.initForPut(DataCursor.java:812)
at com.sleepycat.collections.DataCursor.put(DataCursor.java:752)
at com.sleepycat.collections.StoredContainer.putKeyValue(StoredContainer.java:322)
at com.sleepycat.collections.StoredMap.put(StoredMap.java:280)
at com.taobao.shopservice.picture.core.util.BdbStoredQueueImpl.offer(BdbStoredQueueImpl.java:118)
at com.taobao.shopservice.picture.core.service.CdnClearServiceImpl.clearCdnCache(CdnClearServiceImpl.java:45)
at sun.reflect.GeneratedMethodAccessor484.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:304)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
at com.taobao.shopservice.common.monitor.ProfileInterceptor.invoke(ProfileInterceptor.java:26)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy74.clearCdnCache(Unknown Source)
at com.taobao.shopservice.picture.core.service.PictureWriteServiceImpl.movePicturesToRecycleBin(PictureWriteServiceImpl.java:302)
at com.taobao.shopservice.picture.core.service.PictureWriteServiceImpl.deletePictures(PictureWriteServiceImpl.java:207)
at sun.reflect.GeneratedMethodAccessor483.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:304)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
at com.taobao.shopservice.common.monitor.ProfileInterceptor.invoke(ProfileInterceptor.java:26)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy77.deletePictures(Unknown Source)
at sun.reflect.GeneratedMethodAccessor482.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.taobao.hsf.rpc.tbremoting.provider.ProviderProcessor.handleRequest0(ProviderProcessor.java:222)
at com.taobao.hsf.rpc.tbremoting.provider.ProviderProcessor.handleRequest(ProviderProcessor.java:174)
at com.taobao.hsf.rpc.tbremoting.provider.ProviderProcessor.handleRequest(ProviderProcessor.java:41)
at com.taobao.remoting.impl.DefaultMsgListener$1ProcessorExecuteTask.run(DefaultMsgListener.java:131)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.FileNotFoundException: /home/admin/shopcenter/cdncleaner/00000064.jdb (No such file or directory)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:98)
at com.sleepycat.je.log.FileManager$1.<init>(FileManager.java:993)
at com.sleepycat.je.log.FileManager.openFileHandle(FileManager.java:992)
at com.sleepycat.je.log.FileManager.getFileHandle(FileManager.java:888)
at com.sleepycat.je.log.LogManager.getLogSource(LogManager.java:1073)
at com.sleepycat.je.log.LogManager.getLogEntry(LogManager.java:779)
at com.sleepycat.je.log.LogManager.getLogEntryAllowInvisibleAtRecovery(LogManager.java:743)
at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1225)
... 49 more
2011-03-24 00:00:27,967 INFO [org.quartz.core.JobRunShell] Job DEFAULT.cdnCleanerJobDetail threw a JobExecutionException:
org.quartz.JobExecutionException: Invocation of method 'clearCdn' on target class [class com.taobao.shopservice.picture.core.job.clearcdn.CdnCleaner] failed [See nested exception: com.sleepycat.je.EnvironmentFailureException: (JE 4.0.92) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 4.0.92) /home/admin/shopcenter/cdncleaner fetchTarget of 0x64/0x3b8f73 parent IN=8811763 IN class=com.sleepycat.je.tree.IN lastFullVersion=0xffffffff/0xffffffff parent.getDirty()=true state=0 LOG_FILE_NOT_FOUND: Log file missing, log is likely invalid. Environment is invalid and must be closed.]
at sun.reflect.GeneratedConstructorAccessor102.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:85)
at org.springframework.scheduling.quartz.MethodInvokingJobDetailFactoryBean$MethodInvokingJob.executeInternal(MethodInvokingJobDetailFactoryBean.java:283)
at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:86)
at org.quartz.core.JobRunShell.run(JobRunShell.java:203)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:520)
* Nested Exception (Underlying Cause) ---------------
com.sleepycat.je.EnvironmentFailureException: (JE 4.0.92) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 4.0.92) /home/admin/shopcenter/cdncleaner fetchTarget of 0x64/0x3b8f73 parent IN=8811763 IN class=com.sleepycat.je.tree.IN lastFullVersion=0xffffffff/0xffffffff parent.getDirty()=true state=0 LOG_FILE_NOT_FOUND: Log file missing, log is likely invalid. Environment is invalid and must be closed.
at com.sleepycat.je.EnvironmentFailureException.wrapSelf(EnvironmentFailureException.java:197)
at com.sleepycat.je.dbi.EnvironmentImpl.checkIfInvalid(EnvironmentImpl.java:1403)
at com.sleepycat.je.Database.checkEnv(Database.java:1772)
at com.sleepycat.je.Database.openCursor(Database.java:619)
at com.sleepycat.collections.CurrentTransaction.openCursor(CurrentTransaction.java:416)
at com.sleepycat.collections.MyRangeCursor.openCursor(MyRangeCursor.java:54)
at com.sleepycat.collections.MyRangeCursor.<init>(MyRangeCursor.java:30)
at com.sleepycat.collections.DataCursor.init(DataCursor.java:171)
at com.sleepycat.collections.DataCursor.<init>(DataCursor.java:59)
at com.sleepycat.collections.StoredContainer.getValue(StoredContainer.java:301)
at com.sleepycat.collections.StoredMap.get(StoredMap.java:241)
at com.taobao.shopservice.picture.core.util.BdbStoredQueueImpl.peek(BdbStoredQueueImpl.java:131)
at com.taobao.shopservice.picture.core.util.BdbStoredQueueImpl.poll(BdbStoredQueueImpl.java:169)
at com.taobao.shopservice.picture.core.job.clearcdn.CdnCleaner.clearCdn(CdnCleaner.java:194)
at sun.reflect.GeneratedMethodAccessor641.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:283)
at org.springframework.scheduling.quartz.MethodInvokingJobDetailFactoryBean$MethodInvokingJob.executeInternal(MethodInvokingJobDetailFactoryBean.java:272)
at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:86)
at org.quartz.core.JobRunShell.run(JobRunShell.java:203)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:520)
Caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 4.0.92) /home/admin/shopcenter/cdncleaner fetchTarget of 0x64/0x3b8f73 parent IN=8811763 IN class=com.sleepycat.je.tree.IN lastFullVersion=0xffffffff/0xffffffff parent.getDirty()=true state=0 LOG_FILE_NOT_FOUND: Log file missing, log is likely invalid. Environment is invalid and must be closed.
at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1241)
at com.sleepycat.je.tree.Tree.searchSubTreeInternal(Tree.java:1858)
at com.sleepycat.je.tree.Tree.searchSubTree(Tree.java:1682)
at com.sleepycat.je.tree.Tree.search(Tree.java:1548)
at com.sleepycat.je.dbi.CursorImpl.searchAndPosition(CursorImpl.java:2054)
at com.sleepycat.je.Cursor.searchInternal(Cursor.java:2088)
at com.sleepycat.je.Cursor.searchAllowPhantoms(Cursor.java:2058)
at com.sleepycat.je.Cursor.search(Cursor.java:1926)
at com.sleepycat.je.Cursor.getSearchKey(Cursor.java:1351)
at com.sleepycat.util.keyrange.RangeCursor.doGetSearchKey(RangeCursor.java:966)
at com.sleepycat.util.keyrange.RangeCursor.getSearchKey(RangeCursor.java:593)
at com.sleepycat.collections.DataCursor.doGetSearchKey(DataCursor.java:571)
at com.sleepycat.collections.DataCursor.initForPut(DataCursor.java:812)
at com.sleepycat.collections.DataCursor.put(DataCursor.java:752)
at com.sleepycat.collections.StoredContainer.putKeyValue(StoredContainer.java:322)
at com.sleepycat.collections.StoredMap.put(StoredMap.java:280)
at com.taobao.shopservice.picture.core.util.BdbStoredQueueImpl.offer(BdbStoredQueueImpl.java:118)
at com.taobao.shopservice.picture.core.service.CdnClearServiceImpl.clearCdnCache(CdnClearServiceImpl.java:45)
at sun.reflect.GeneratedMethodAccessor484.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:304)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
at com.taobao.shopservice.common.monitor.ProfileInterceptor.invoke(ProfileInterceptor.java:26)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy74.clearCdnCache(Unknown Source)
at com.taobao.shopservice.picture.core.service.PictureWriteServiceImpl.movePicturesToRecycleBin(PictureWriteServiceImpl.java:302)
at com.taobao.shopservice.picture.core.service.PictureWriteServiceImpl.deletePictures(PictureWriteServiceImpl.java:207)
at sun.reflect.GeneratedMethodAccessor483.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:304)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
at com.taobao.shopservice.common.monitor.ProfileInterceptor.invoke(ProfileInterceptor.java:26)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy77.deletePictures(Unknown Source)
at sun.reflect.GeneratedMethodAccessor482.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.taobao.hsf.rpc.tbremoting.provider.ProviderProcessor.handleRequest0(ProviderProcessor.java:222)
at com.taobao.hsf.rpc.tbremoting.provider.ProviderProcessor.handleRequest(ProviderProcessor.java:174)
at com.taobao.hsf.rpc.tbremoting.provider.ProviderProcessor.handleRequest(ProviderProcessor.java:41)
at com.taobao.remoting.impl.DefaultMsgListener$1ProcessorExecuteTask.run(DefaultMsgListener.java:131)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.FileNotFoundException: /home/admin/shopcenter/cdncleaner/00000064.jdb (No such file or directory)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:98)
at com.sleepycat.je.log.FileManager$1.<init>(FileManager.java:993)
at com.sleepycat.je.log.FileManager.openFileHandle(FileManager.java:992)
at com.sleepycat.je.log.FileManager.getFileHandle(FileManager.java:888)
at com.sleepycat.je.log.LogManager.getLogSource(LogManager.java:1073)
at com.sleepycat.je.log.LogManager.getLogEntry(LogManager.java:779)
at com.sleepycat.je.log.LogManager.getLogEntryAllowInvisibleAtRecovery(LogManager.java:743)
at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1225)
... 49 moreI mean that i can open the database and read some data from it, which was stored before the database was closed, and than the exception was thrown.
here is exception stack with JE 4.1.7:
com.sleepycat.je.EnvironmentFailureException: (JE 4.1.7) F:\job fetchTarget of 0x64/0x4735f7 parent IN=7847269 IN class=com.sleepycat.je.tree.IN lastFullVersion=0x66/0x927f09 parent.getDirty()=false state=0 LOG_FILE_NOT_FOUND: Log file missing, log is likely invalid. Environment is invalid and must be closed.
at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1337)
at com.sleepycat.je.tree.IN.fetchTargetWithExclusiveLatch(IN.java:1278)
at com.sleepycat.je.tree.Tree.getNextBinInternal(Tree.java:1358)
at com.sleepycat.je.tree.Tree.getPrevBin(Tree.java:1240)
at com.sleepycat.je.dbi.CursorImpl.getNextWithKeyChangeStatus(CursorImpl.java:1754)
at com.sleepycat.je.dbi.CursorImpl.getNext(CursorImpl.java:1617)
at com.sleepycat.je.Cursor.retrieveNextAllowPhantoms(Cursor.java:2488)
at com.sleepycat.je.Cursor.retrieveNext(Cursor.java:2304)
at com.sleepycat.je.Cursor.getPrev(Cursor.java:1190)
at com.ppsoft.bdb.test.Main.main(Main.java:52)
Caused by: java.io.FileNotFoundException: F:\job\00000064.jdb (No such file or directory)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:98)
at com.sleepycat.je.log.FileManager$1.<init>(FileManager.java:995)
at com.sleepycat.je.log.FileManager.openFileHandle(FileManager.java:994)
at com.sleepycat.je.log.FileManager.getFileHandle(FileManager.java:890)
at com.sleepycat.je.log.LogManager.getLogSource(LogManager.java:1074)
at com.sleepycat.je.log.LogManager.getLogEntry(LogManager.java:778)
at com.sleepycat.je.log.LogManager.getLogEntryAllowInvisibleAtRecovery(LogManager.java:742)
at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1320)
... 9 more -
EnvironmentFailureException question
Hi,
I'm relatively new to BDB... I've (part-)written a distributed crawler which uses BDB to store a persistent on-disk queue of URIs. This is the setup.
+/* Open a transactional Berkeley DB engine environment. */+
EnvironmentConfig envConfig = new EnvironmentConfig();
envConfig.setAllowCreate(true);
envConfig.setTransactional(true);
_env = new Environment(envDir, envConfig);+
+/* Open a transactional entity store. */+
StoreConfig storeConfig = new StoreConfig();
storeConfig.setAllowCreate(true);
storeConfig.setTransactional(true);
store = new EntityStore(env, this.getClass().getSimpleName(), storeConfig);+
+/* Primary index of the queue */+
urlIndex = store.getPrimaryIndex(String.class, URLObject.class);+
+/* Secondary index of the queue */+
countIndex = store.getSecondaryIndex(_urlIndex, Integer.class, "count");+
The machine is running Java 1.6.0_12 on Debian 5.0.4. The environment directory is on a local partition (in fact, everything is pretty much local as far as BDB can see).
This setup seems to work quite well. However, after about 30 hours or crawling one server (of eight) dies with the initial exception:
+<DaemonThread name="Cleaner-1"/> caught exception: com.sleepycat.je.EnvironmentFailureException: (JE 4.0.71) /data/webdb/may10/crawl/q java.io.IOException: Input/output error LOG_READ: IOException on read, log is likely invalid. Environment is invalid and must be closed.+
com.sleepycat.je.EnvironmentFailureException: (JE 4.0.71) /data/webdb/may10/crawl/q java.io.IOException: Input/output error LOG_READ: IOException on read, log is likely invalid. Environment is invalid and must be closed.
at com.sleepycat.je.log.FileManager.readFromFile(FileManager.java:1516)
at com.sleepycat.je.log.FileReader$ReadWindow.fillFromFile(FileReader.java:1116)
at com.sleepycat.je.log.FileReader$ReadWindow.fillNext(FileReader.java:1074)
at com.sleepycat.je.log.FileReader.readData(FileReader.java:759)
at com.sleepycat.je.log.FileReader.readNextEntryAllowExceptions(FileReader.java:315)
at com.sleepycat.je.cleaner.FileProcessor.processFile(FileProcessor.java:396)
at com.sleepycat.je.cleaner.FileProcessor.doClean(FileProcessor.java:236)
at com.sleepycat.je.cleaner.FileProcessor.onWakeup(FileProcessor.java:141)
at com.sleepycat.je.utilint.DaemonThread.run(DaemonThread.java:161)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.IOException: Input/output error
at java.io.RandomAccessFile.readBytes(Native Method)
at java.io.RandomAccessFile.read(RandomAccessFile.java:322)
at com.sleepycat.je.log.FileManager.readFromFileInternal(FileManager.java:1551)
at com.sleepycat.je.log.FileManager.readFromFile(FileManager.java:1506)
+... 9 more+
Exiting
Followed by a plethora of similar exceptions for different lookup threads:
Exception in thread "LookupThread-XXX" ...
Also, in the je.info.0 file in the environment directory, I found the following:
+SEVERE [data/webdb/may10/crawl/q]Halted log file reading at file 0x997 offset 0x1edcb offset(decimal)=126411 prev=0x1ed84:+
entry=BINDeltatype=22,version=7)
prev=0x1ed84
size=886
Next entry should be at 0x1f14f
I'm generally at a loss as to why this happened. There's no obvious cause (such as no disk space, etc.) and it seems more-so that the index is corrupted. At the time of the exception, there is 1.5GB in the environment directory (~150 x 9.6M *.jdb files).
I cannot really reproduce the error/create a test-case, so I don't know if updating JE would help -- I'd like to have as much information on the bug as possible, and have implemented as sure a fix as possible before I try restarting the crawl.
Any help/thoughts greatly appreciated.Aidan,
From the error message and the je.info logging you showed us, the complaint seems to be about the file /data/webdb/may10/crawl/q/00000997.jdb. But unfortunately, the Java IOException is not telling you much.
What's happened is that our daemon thread, the log cleaner, was going along, and it hit an IOException when reading that file. We just wrap the IOException, and send it back to up, and in this case, the exception is particularly uninformative ("Input/output exception"). Since we toString the exception, I think there's really not any more message. We've noticed that JDKs on different platforms can have more or less informative IOExceptions -- perhaps what's available underneath?
The string "LOG_READ: IOException on read, log is likely invalid. Environment is invalid and must be closed." is a wrapper from JE, and is potentially a bit alarmist. We take the conservative approach that if anything unexpected goes wrong with a read, we don't want to return bad data, and we shut down the whole environment. Other threads will note the invalidation of the environment and will also shut down. Tthough it can certainly always be a JE bug, in this case, it seems more like some kind of underlying and transient system issue.
JE does detect when it is able to do a read and it thinks the data is bad. In that case, you get an checksum exception. This is different, in that something killed off the read operation itself. Could something have sent an interrupt to the process? Though in those cases, we usually see an InterruptedException.
One thing you can do is to use the com.sleepycat.je.util.DbPrintLog utility to just read the afflicted file. Despite the JE suggestion of an invalid environment, in this case, it smells a little more like a transient platform I/O problem. You could do "java -jar je.jar DbPrintLog -h /data/webdb/may10/crawl/q -s 0x997 > temp.xml" and see if that file can be read. If it is successful, the utility will dump the contents of the log and you'll see a complete xml file, with no exception at the end. If it can, it would reinforce the likelihood that some transient IO incident occurred, and you may well be able to restart the environment with no problem.
Hope that give you a start,
Linda -
EnvironmentFailureException on opening EntityStore
Adding a new secondary key field to an entity class made it impossible to open EntityStore:
com.sleepycat.je.EnvironmentFailureException: (JE 4.1.6) UNEXPECTED_STATE: Unexpected internal state, may have side effects.
at com.sleepycat.je.EnvironmentFailureException.unexpectedState(EnvironmentFailureException.java:347)
at com.sleepycat.compat.DbCompat.unexpectedState(DbCompat.java:507)
at com.sleepycat.persist.impl.EnhancedAccessor.newInstance(EnhancedAccessor.java:104)
at com.sleepycat.persist.impl.ComplexFormat.checkNewSecKeyInitializer(ComplexFormat.java:475)
at com.sleepycat.persist.impl.ComplexFormat.initialize(ComplexFormat.java:451)
at com.sleepycat.persist.impl.Format.initializeIfNeeded(Format.java:542)
at com.sleepycat.persist.impl.ComplexFormat.initialize(ComplexFormat.java:334)
at com.sleepycat.persist.impl.Format.initializeIfNeeded(Format.java:542)
at com.sleepycat.persist.impl.ComplexFormat.initialize(ComplexFormat.java:334)
at com.sleepycat.persist.impl.Format.initializeIfNeeded(Format.java:542)
at com.sleepycat.persist.impl.PersistCatalog.init(PersistCatalog.java:454)
at com.sleepycat.persist.impl.PersistCatalog.<init>(PersistCatalog.java:221)
at com.sleepycat.persist.impl.Store.<init>(Store.java:186)
at com.sleepycat.persist.EntityStore.<init>(EntityStore.java:185)
Entity store opens fine if there are no changes to format at all or there is no @SecondaryKey annotation for the new field. Here is my entity class:
@Entity
public abstract class AbstractMessageEntity implements MessageEntity {
@PrimaryKey
private Long id;
and after adding new secondary key:
@Entity( version = 3 )
public abstract class AbstractMessageEntity implements MessageEntity {
@PrimaryKey
private Long id;
@SecondaryKey( relate = Relationship.MANY_TO_ONE )
private Long executionTime;
AbstractMessageEntity has 3 persistent subclasses, but they were not changed except increasing their version from 0 to 3.
I would greatly appreciate any workaround for this problem!Hi,
According to your given class information, I assume that you have an abstract class, AbstractMessageEntity, and three subclasses, called MessageEntity1, MessageEntity2, and MessageEntity3. All you want to store are those three subclasses, right?
Before addressing to your problem, I wanna point out that, the annotation of AbstractMessageEntity, which is @Entity should be changed to @Persistent. Because in DPL, you cannot define an entity subclass for another entity class. This is just a reminder, which is not related to your current errors.
Now come back to your error. Actually, I cannot reproduce your error. Below is the code I made and tried to use it to reproduce your error:
import static com.sleepycat.persist.model.Relationship.MANY_TO_ONE;
import java.io.File;
import com.sleepycat.je.Environment;
import com.sleepycat.je.EnvironmentConfig;
import com.sleepycat.persist.EntityStore;
import com.sleepycat.persist.PrimaryIndex;
import com.sleepycat.persist.StoreConfig;
import com.sleepycat.persist.model.AnnotationModel;
import com.sleepycat.persist.model.Entity;
import com.sleepycat.persist.model.EntityModel;
import com.sleepycat.persist.model.Persistent;
import com.sleepycat.persist.model.PrimaryKey;
import com.sleepycat.persist.model.SecondaryKey;
public class AddNewSecKeyTest {
private Environment env;
private EntityStore store;
private PrimaryIndex<Long, MessageEntity> primary;
public static void main(String args[]) {
AddNewSecKeyTest epc = new AddNewSecKeyTest();
epc.open();
epc.writeData();
epc.close();
private void writeData() {
primary.put(null, new MessageEntity(1));
private void getData() {
AbstractMessageEntity data = primary.get(1L);
private void close() {
store.close();
store = null;
env.close();
env = null;
private void open() {
EnvironmentConfig envConfig = new EnvironmentConfig();
envConfig.setAllowCreate(true);
File envHome = new File("./");
env = new Environment(envHome, envConfig);
EntityModel model = new AnnotationModel();
StoreConfig config = new StoreConfig();
config.setAllowCreate(envConfig.getAllowCreate());
config.setTransactional(envConfig.getTransactional());
config.setModel(model);
store = new EntityStore(env, "test", config);
primary = store.getPrimaryIndex(Long.class, MessageEntity.class);
@Persistent(version = 3)
static public abstract class AbstractMessageEntity {
AbstractMessageEntity(Long i) {
this.id = i;
private AbstractMessageEntity(){}
@PrimaryKey
private Long id;
@SecondaryKey( relate = MANY_TO_ONE )
private Long executionTime;
@Entity(version = 3)
static public class MessageEntity extends AbstractMessageEntity{
private int f1;
private MessageEntity(){}
MessageEntity(int i) {
super(Long.valueOf(i));
this.f1 = i;
}However, the above code can run successfully on my sandbox (with JE 4.1.6). I don't know how much difference between my code and your code (I mean the class hierarchy). So please post your class hierarchy according to my code, also the code about how you serialize your class.
Thanks.
Eric Wang
BDB JE Team -
LOG_FILE_NOT_FOUND when running cleaner manually after some data purge
I hit LOG_FILE_NOT_FOUND error when running cleaner manually after some data purge, I searched the forum, found someone also faced the same issue before, but cannot find any clue on how to fix it. Below is the error trace and followed by our configurations
Caused by: com.sleepycat.je.EnvironmentFailureException: (JE 4.1.6)
Environment must be closed, caused by:
com.sleepycat.je.EnvironmentFailureException: Environment invalid because of
previous exception: (JE 4.1.6) /scratch/tie/thirdeye/index/data-store
fetchTarget of 0x50f/0x3fb9dd6 parent IN=368491717 IN
class=com.sleepycat.je.tree.IN lastFullVersion=0x510/0x2ca7d18
parent.getDirty()=false state=0 LOG_FILE_NOT_FOUND: Log file missing, log is
likely invalid. Environment is invalid and must be closed.
at
com.sleepycat.je.EnvironmentFailureException.wrapSelf(EnvironmentFailureExcept
ion.java:196)
at
com.sleepycat.je.dbi.EnvironmentImpl.checkIfInvalid(EnvironmentImpl.java:1439)
at com.sleepycat.je.Environment.checkEnv(Environment.java:2117)
at com.sleepycat.je.Environment.checkpoint(Environment.java:1440)
at
com.oracle.thirdeye.datastore.DataStoreManager.clean(DataStoreManager.java:402
at
com.oracle.thirdeye.infostore.InfoStoreManager.clean(InfoStoreManager.java:301
... 11 more
Caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid
because of previous exception: (JE 4.1.10)
/scratch/tie/thirdeye/index/data-store fetchTarget of 0x50f/0x3fb9dd6 parent
IN=368491717 IN class=com.sleepycat.je.tree.IN
lastFullVersion=0x510/0x2ca7d18 parent.getDirty()=false state=0
LOG_FILE_NOT_FOUND: Log file missing, log is likely invalid. Environment is
invalid and must be closed.
at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1332)
at com.sleepycat.je.tree.IN.findParent(IN.java:2886)
at com.sleepycat.je.tree.Tree.getParentINForChildIN(Tree.java:881)
at com.sleepycat.je.tree.Tree.getParentINForChildIN(Tree.java:809)
at
com.sleepycat.je.cleaner.FileProcessor.findINInTree(FileProcessor.java:1152)
at com.sleepycat.je.cleaner.FileProcessor.processIN(FileProcessor.java:1090)
at
com.sleepycat.je.cleaner.FileProcessor.processFile(FileProcessor.java:538)
at com.sleepycat.je.cleaner.FileProcessor.doClean(FileProcessor.java:241)
at com.sleepycat.je.cleaner.Cleaner.doClean(Cleaner.java:463)
------------Configurations-------------------------
EnvironmentConfig.ENV_RUN_CLEANER -> false
EnvironmentConfig.CHECKPOINTER_HIGH_PRIORITY -> true
EnvironmentConfig.CLEANER_EXPUNGE -> false
Any hints are appreciated. I'm also working for Oracle CDC, feel free to call me at 861065151679 or drop me an email at [email protected] so that we can talk more in detail
AnferneeAnfernee, I will contact you via email.
--mark -
LOG_FILE_NOT_FOUND bug possible in current BDB JE?
I've seen references to the LOG_FILE_NOT_FOUND bug in older BDB JE versions (4.x and 5 <= 5.0.34(, however, I seem to be suffering something similar with 5.0.48.
I have a non-transactional, deferred-write DB that seems to have gotten itself into an inconsistent state. It was fine loading several million records, but after ~8 hours of operation, bailed out with:
com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 5.0.55) /tmp/data/index fetchTarget of 0x9f1/0x24d34eb parent IN=44832 IN class=com.sleepycat.je.tree.BIN lastFullVersion=0xdcf/0x5a96c91 lastLoggedVersion=0xdcf/0x5a96c91 parent.getDirty()=true state=0 LOG_FILE_NOT_FOUND: Log file missing, log is likely invalid. Environment is invalid and must be closed.
at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1429)
at com.sleepycat.je.tree.BIN.fetchTarget(BIN.java:1251)
at com.sleepycat.je.dbi.CursorImpl.fetchCurrent(CursorImpl.java:2229)
at com.sleepycat.je.dbi.CursorImpl.getCurrentAlreadyLatched(CursorImpl.java:1434)
at com.sleepycat.je.Cursor.searchInternal(Cursor.java:2716)
at com.sleepycat.je.Cursor.searchAllowPhantoms(Cursor.java:2576)
at com.sleepycat.je.Cursor.searchNoDups(Cursor.java:2430)
at com.sleepycat.je.Cursor.search(Cursor.java:2397)
at com.sleepycat.je.Database.get(Database.java:1042)
at com.xxxx.db.BDBCalendarStorageBackend.indexCalendar(BDBCalendarStorageBackend.java:95)
at com.xxxx.indexer.TicketIndexer.indexDeltaLogs(TicketIndexer.java:201)
at com.xxxx.indexer.DeltaLogLoader.run(DeltaLogLoader.java:87)
Caused by: java.io.FileNotFoundException: /tmp/data/index/000009f1.jdb (No such file or directory)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:216)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:101)
at com.sleepycat.je.log.FileManager$6.<init>(FileManager.java:1282)
at com.sleepycat.je.log.FileManager.openFileHandle(FileManager.java:1281)
at com.sleepycat.je.log.FileManager.getFileHandle(FileManager.java:1147)
at com.sleepycat.je.log.LogManager.getLogSource(LogManager.java:1102)
at com.sleepycat.je.log.LogManager.getLogEntry(LogManager.java:808)
at com.sleepycat.je.log.LogManager.getLogEntryAllowInvisibleAtRecovery(LogManager.java:772)
at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1412)
... 11 more
Subsequent opens/use on the DB pretty much instantly yield the same error. I tried upgrading to 5.0.55 (hence the ver in the output above) but still get the same error.
As a recovery attempt, I used DbDump to try to dump the DB, however, it failed with a similar error. Enabling salvage mode enabled me to successfuly dump it, however, reloading it into a clean environment by programmatically running DbLoad.load() (so I can setup my env) caused the following error (after about 30% of the DB has restored):
Exception in thread "main" com.sleepycat.je.EnvironmentFailureException: (JE 5.0.55) Node 11991 should have been split before calling insertEntry UNEXPECTED_STATE: Unexpected internal state, may have side effects. fetchTarget of 0x25/0x155a822 parent IN=2286 IN class=com.sleepycat.je.tree.IN lastFullVersion=0x3e/0x118d8f6 lastLoggedVersion=0x3e/0x118d8f6 parent.getDirty()=false state=0
at com.sleepycat.je.EnvironmentFailureException.unexpectedState(EnvironmentFailureException.java:376)
at com.sleepycat.je.tree.IN.insertEntry1(IN.java:2326)
at com.sleepycat.je.tree.IN.insertEntry(IN.java:2296)
at com.sleepycat.je.tree.BINDelta.reconstituteBIN(BINDelta.java:216)
at com.sleepycat.je.tree.BINDelta.reconstituteBIN(BINDelta.java:144)
at com.sleepycat.je.log.entry.BINDeltaLogEntry.getIN(BINDeltaLogEntry.java:53)
at com.sleepycat.je.log.entry.BINDeltaLogEntry.getResolvedItem(BINDeltaLogEntry.java:43)
at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1422)
at com.sleepycat.je.tree.Tree.searchSubTreeUntilSplit(Tree.java:1786)
at com.sleepycat.je.tree.Tree.searchSubTreeSplitsAllowed(Tree.java:1729)
at com.sleepycat.je.tree.Tree.searchSplitsAllowed(Tree.java:1296)
at com.sleepycat.je.tree.Tree.findBinForInsert(Tree.java:2205)
at com.sleepycat.je.dbi.CursorImpl.putInternal(CursorImpl.java:834)
at com.sleepycat.je.dbi.CursorImpl.put(CursorImpl.java:779)
at com.sleepycat.je.Cursor.putAllowPhantoms(Cursor.java:2243)
at com.sleepycat.je.Cursor.putNoNotify(Cursor.java:2200)
at com.sleepycat.je.Cursor.putNotify(Cursor.java:2117)
at com.sleepycat.je.Cursor.putNoDups(Cursor.java:2052)
at com.sleepycat.je.Cursor.putInternal(Cursor.java:2020)
at com.sleepycat.je.Database.putInternal(Database.java:1324)
at com.sleepycat.je.Database.put(Database.java:1194)
at com.sleepycat.je.util.DbLoad.loadData(DbLoad.java:544)
at com.sleepycat.je.util.DbLoad.load(DbLoad.java:414)
at com.xxxx.db.BDBCalendarStorageBackend.loadBDBDump(BDBCalendarStorageBackend.java:254)
at com.xxxx.cli.BDBTool.run(BDBTool.java:49)
at com.xxxx.cli.AbstractBaseCommand.execute(AbstractBaseCommand.java:114)
at com.xxxx.cli.BDBTool.main(BDBTool.java:69)
The only other slightly exotic thing I'm using is a custom partial BTree comparator, however, it quite happily loaded/updated literally tens of millions of records for hours before the FileNotFound error cropped up, so it seems unlikely that would be the cause.
Any ideas?
Thanks in advance,
fb.Thanks heaps to Mark for working through this with me.You're welcome. Thanks for following up and explaining it for the benefit of others. And I'm very glad it wasn't a JE bug!
My solution is to switch to using a secondary database for providing differentiated "uniqueness" vs "ordering".An index for uniqueness may be a good solution. But as you said in email, it adds significant overhead (memory and disk). This overhead can be minimized by keeping your keys (primary and secondary) as small as possible, and enabling key prefixing.
I'd also like to point out that adding a secondary isn't always the best choice. For example, if the number of keys with the same C1 value is fairly small, another way of checking for uniqueness (when inserting) is to iterate over them, looking for a match on C1:C3. The cost of this iteration may be less than the cost of maintaining a uniqueness index. To make this work, you'll have to use Serializable isolation during the iteration, to prevent another thread from inserting a key in that range.
If you're pushing the performance limits of your hardware, it may be worth trying more than one such approach and comparing the performance. If performance is not a big concern, then the additional index is the simplest approach to get right.
--mark -
Hello everyone,
We are using Berkeley DB Java Edition with our heavily-multithreaded application. Transactions have been enabled on the database.
While the system is busy, we occasionally face an orgy of LockTimeoutException, which can be resolved only by a restart. Note that the locks also usually prevent a clean close of the database/environment.
One such sample is:
com.sleepycat.je.LockTimeoutException: (JE 4.1.17) Lock expired. Locker 1668415747 -1_TH_CGTOutCylSMS5 for gOutMsgID 96970960_ReadCommittedLocker: waited for lock on database=sampleDatabase LockAddr:292555378 node=8759642048 type=READ grant=WAIT_NEW timeoutMillis=30000 startTime=1327243343323 endTime=1327243373324 Owners: [<LockInfo locker="1542681801 610308237201_TH_CGTOutCylSMS5 for gOutMsgID 96970930_Txn" type="WRITE"/>]
Waiters: [<LockInfo locker="711907833 -1_TH_CGTOutCylSMS5 for gOutMsgID 96970946_ReadCommittedLocker" type="READ"/>, <LockInfo locker="1403551794 -1_TH_CGTOutCylSMS5 for gOutMsgID 96970940_ReadCommittedLocker" type="READ"/>, <LockInfo locker="1369480950 -1_TH_CGTOutCylSMS5 for gOutMsgID 96970957_ReadCommittedLocker" type="READ"/>, <LockInfo locker="136598104 -1_TH_CGTOutCylSMS5 for gOutMsgID 96970970_ReadCommittedLocker" type="READ"/>, <LockInfo locker="144315285 -1_TH_CGTOutCylSMS5 for gOutMsgID 96970968_ReadCommittedLocker" type="READ"/>, <LockInfo locker="997957532 -1_TH_CGTOutCylSMS5 for gOutMsgID 96970975_ReadCommittedLocker" type="READ"/>, <LockInfo locker="824957254 -1_TH_CGTOutCylSMS5 for gOutMsgID 96970961_ReadCommittedLocker" type="READ"/>, <LockInfo locker="902142946 -1_TH_CGTOutCylSMS5 for gOutMsgID 96970967_ReadCommittedLocker" type="READ"/>, <LockInfo locker="1081945602 -1_TH_CGTOutCylSMS5 for gOutMsgID 96970977_ReadCommittedLocker" type="READ"/>, <LockInfo locker="1515693665 -1_TH_CGTOutCylSMS5 for gOutMsgID 96970981_ReadCommittedLocker" type="READ"/>, <LockInfo locker="812406941 -1_TH_CGTOutCylSMS5 for gOutMsgID 96970964_ReadCommittedLocker" type="READ"/>, <LockInfo locker="422429761 -1_TH_CGTOutCylSMS5 for gOutMsgID 96970966_ReadCommittedLocker" type="READ"/>, <LockInfo locker="196056145 -1_TH_CGTOutCylSMS5 for gOutMsgID 96970976_ReadCommittedLocker" type="READ"/>, <LockInfo locker="875714517 -1_TH_CGTOutCylSMS5 for gOutMsgID 96970971_Read
CommittedLocker" type="READ"/>, <LockInfo locker="1428454328 -1_TH_CGTOutCylSMS5 for gOutMsgID 96970969_ReadCommittedLocker" type="READ"/>, <LockInfo locker="1312200883 -1_TH_CGTOutCylSMS5 for gOutMsgID 96970985_ReadCommittedLocker" type="READ"/>, <LockInfo locker="1650669538 -1_TH_CGTOutCylSMS5 for gOutMsgID 96970986_ReadCommittedLocker" type="READ"/>, <LockInfo locker="1609173220 -1_TH_CGTOutCylSMS5 for gOutMsgID 96970983_ReadCommittedLocker" type="READ"/>, <LockInfo locker="2112741869 -1_TH_CGTOutCylSMS5 for gOutMsgID 96970982_ReadCommittedLocker" type="READ"/>, <LockInfo locker="210489138 -1_TH_CGTOutCylSMS5 for gOutMsgID 96970980_ReadCommittedLocker" type="READ"/>, <LockInfo locker="785375757 -1_TH_CGTOutCylSMS5 for gOutMsgID 96970973_ReadCommittedLocker" type="READ"/>, <LockInfo locker="1375853437 -1_TH_CGTOutCylSMS5 for gOutMsgID 96970978_ReadCommittedLocker" type="READ"/>, <LockInfo locker="2012288410 -1_TH_CGTOutCylSMS5 for gOutMsgID 96970988_ReadCommittedLocker" type="READ"/>, <LockInfo locker="1958833914 -1_TH_CGTOutCylSMS5 for gOutMsgID 96970987_ReadCommittedLocker" type="READ"/>, <LockInfo locker="1197284088 -1_TH_CGTOutCylSMS5 for gOutMsgID 96970984_Re
adCommittedLocker" type="READ"/>, <LockInfo locker="831894626 -1_TH_CGTOutCylSMS5 for gOutMsgID 96970979_ReadCommittedLocker" type="READ"/>, <LockInfo locker="1534437212 -1_TH_CGTOutCylSMS5 for gOutMsgID 96970911_ReadCommittedLocker" type="READ"/>, <LockInfo locker="155105261 -1_TH_CGTOutCylSMS5 for gOutMsgID 96970922_ReadCommittedLocker" type="READ"/>, <LockInfo locker="1956457816 -1_TH_CGTOutCylSMS5 for gOutMsgID 96970919_ReadCommittedLocker" type="READ"/>, <LockInfo locker="2064394295 -1_TH_CGTOutCylSMS5 for gOutMsgID 96970928_ReadCommittedLocker" type="READ"/>, <LockInfo locker="1529115133 -1_TH_CGTOutCylSMS5 for gOutMsgID 96970936_ReadCommittedLocker" type="READ"/>, <LockInfo locker="901236682 -1_TH_CGTOutCylSMS5 for gOutMsgID 96970943_ReadCommittedLocker" type="READ"/>, <LockInfo locker="882752083 -1_TH_CGTOutCylSMS5 for gOutMsgID 96970937_ReadCommittedLocker" type="READ"/>]
This has today resulted in, during shutdown:
com.sleepycat.je.EnvironmentFailureException: (JE 4.1.17) Txn undo for LSN=0x4c5d59/0x2fd8da UNEXPECTED_EXCEPTION: Unexpected internal Exception, may have side effects.
Problem: 376426 locks left
java.lang.OutOfMemoryError
Java heap space
java.util.Arrays.copyOf(Arrays.java:2882)
java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
java.lang.StringBuilder.append(StringBuilder.java:119)
java.lang.StringBuilder.append(StringBuilder.java:115)
com.sleepycat.je.txn.LockManager.dumpToStringNoLatch(LockManager.java:1205)
com.sleepycat.je.txn.LockManager.dumpToString(LockManager.java:1187)
com.sleepycat.je.txn.LockManager.dump(LockManager.java:1177)
com.sleepycat.je.dbi.EnvironmentImpl.checkLeaks(EnvironmentImpl.java:1786)
com.sleepycat.je.dbi.EnvironmentImpl.doClose(EnvironmentImpl.java:1642)
com.sleepycat.je.dbi.DbEnvPool.closeEnvironment(DbEnvPool.java:348)
com.sleepycat.je.dbi.EnvironmentImpl.close(EnvironmentImpl.java:1480)
com.sleepycat.je.Environment.closeInternal(Environment.java:460)
com.sleepycat.je.Environment.close(Environment.java:321)
and also an entry in the je.log:
120122 19:55:18:205 SEVERE [prog/citrine/db]LNUndo success=false node=46893596705 lsn=0x4c5d59/0x2fd8da found=false replaced=false inserted=false index=-1
We are using 4.1.17 (also previously used 4.1.10 with the same problem) on java 1.6.0u27, 64-bit, on CentOS. The size of the database is over 250 GB.
We am currently running a database.verify() to ensure that the data isnt corrupt.
Our questions are:
* The locks seem to be read locks. While i am aware that the Java Edition does not have MVCC, is there any other mechanism that will give similar 'locklessness'?
* Should increasing the setting of LOCK_N_LOCK_TABLES to say, 500, reduce the chances of this issue?
* Are there any other lock-specific settings we should be aware of? We have set the reads to READ_COMMITTED.
and more immediately:
* Is database.verify sufficient to verify that there is no corruption of data? (It has been running for over half an hour now) There is only one database in the environment.
Awaiting your response,
Sushant A.I have some additional comments.
The SEVERE log message that you describe should be accompanied by a stack trace. If you have one, please send it. If there was no stack trace, then it is possible this is due to an OutOfMemoryError. This is likely due to the large number of locks that have accumulated.
Increasing the number of JE lock tables will not solve this problem. The number of lock tables is increased for better concurrency at a lower level. It won't avoid lock timeouts.
In the stack trace you sent, one thread holds a write lock, and many threads are waiting on a read lock. Lock timeouts can occur because the writer holds the lock (does not commit/abort the txn) for the timeout interval, but this seems unlikely since it looks like you have set the timeout to 30s.
The other reason for the exception is a deadlock. Deadlocks are described in the doc link I sent you earlier. If a deadlock occurs, you'll need to retry the operation after closing the cursors and aborting the txn.
Because many (300k) locks have accumulated, I strongly suspect your app is not closing cursors and/or txns when an exception occurs. The first thing is to check for that.
--mark -
Environment is invalid and must be closed (after upgrading to 4.1.7)
Hi,
I started getting this exception (stack trace below) after updating to 4.1.6 and eventually to 4.1.7 as well
Creating the environment and accessing it works fine the first time around and further attempts to access the environment will succeed as long as the applications server is not restarted.
Any idea what could be the problem?
Thanks in advance,
Javier
com.sleepycat.je.EnvironmentFailureException: (JE 4.1.7) /data/bdb/dbEnv/comp_environment_1 This environment's log file has no root. Since the root is the first entry written into a log at environment creation, this should only happen if the initial creation of the environment was never checkpointed or synced. Please move aside the existing log files to allow the creation of a new environment LOG_INTEGRITY: Log information is incorrect, problem is likely persistent. Environment is invalid and must be closed.
at com.sleepycat.je.recovery.RecoveryManager.findLastCheckpoint(RecoveryManager.java:438)
at com.sleepycat.je.recovery.RecoveryManager.recover(RecoveryManager.java:164)
at com.sleepycat.je.dbi.EnvironmentImpl.finishInit(EnvironmentImpl.java:534)
at com.sleepycat.je.dbi.DbEnvPool.getEnvironment(DbEnvPool.java:217)
at com.sleepycat.je.Environment.makeEnvironmentImpl(Environment.java:229)
at com.sleepycat.je.Environment.<init>(Environment.java:211)
at com.sleepycat.je.Environment.<init>(Environment.java:165)Javier,
I'll explain what sequence of events can make this exception happen, and perhaps you can figure out why this occurs in your application.
The JE log files hold different kind of log entries, one of which is a type called the environment root. The root holds metadata about the environment as a whole, and JE needs to find the latest version of the root every time an environment is recovered (that is, re-opened). There should always be at least one version in the log.
When an environment is created, the first version of the root is written to the log almost immediately, and the log is checkpointed. The only way we anticipate seeing this particular error is if the environment creation was interrupted, after that first 00000000.jdb file was created, and before the root was written out, which is a pretty small window. This would occur before the environment constructor had returned.
Does that sound like it matches your scenario?
Linda -
Problem when open the Java Berkeley DB
I'm using the Java Berkeley DB version: je-6.2.31.jar
- My database has about 4 GB of info.
- I use a Replicated Environment (but now I'm currently pointing the -nodeHost and -helperHost to the same IP and port)
- I'm using a StoredSortedMap to store some key/value info. (sorted duplicates is set to true)
I use the database correctly for some time.
Normaly I open the database and close many times;
There is enough disk, and I already tested it in different HDs, (even with a SSD)
I casually receive the bellow error, it's not more possible to open the data base:
com.sleepycat.je.EnvironmentFailureException: (JE 6.2.31) Problem creating output files in: E:\data\database/je.info UNEXPECTED_EXCEPTION: Unexpected internal Exception, may have side effects.
at com.sleepycat.je.EnvironmentFailureException.unexpectedException(EnvironmentFailureException.java:330)
at com.sleepycat.je.dbi.EnvironmentImpl.initFileHandler(EnvironmentImpl.java:1484)
at com.sleepycat.je.dbi.EnvironmentImpl.<init>(EnvironmentImpl.java:474)
at com.sleepycat.je.rep.impl.RepImpl.<init>(RepImpl.java:267)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
at com.sleepycat.je.dbi.DbEnvPool.loadRepImpl(DbEnvPool.java:242)
at com.sleepycat.je.dbi.DbEnvPool.getEnvironment(DbEnvPool.java:178)
at com.sleepycat.je.Environment.makeEnvironmentImpl(Environment.java:251)
at com.sleepycat.je.Environment.<init>(Environment.java:232)
at com.sleepycat.je.Environment.<init>(Environment.java:188)
at com.sleepycat.je.rep.ReplicatedEnvironment.<init>(ReplicatedEnvironment.java:573)
at com.sleepycat.je.rep.ReplicatedEnvironment.<init>(ReplicatedEnvironment.java:443)
at com.sleepycat.je.rep.ReplicatedEnvironment.<init>(ReplicatedEnvironment.java:507)
Caused by: java.io.IOException: Couldn't get lock for E:\data\database/je.info
at java.util.logging.FileHandler.openFiles(FileHandler.java:422)
at java.util.logging.FileHandler.<init>(FileHandler.java:394)
at com.sleepycat.je.util.FileHandler.<init>(FileHandler.java:85)
at com.sleepycat.je.dbi.EnvironmentImpl.initFileHandler(EnvironmentImpl.java:1478)
... 24 moreI got the same behavior again:
I open and close the DB many times about 50 times.
OBS: I interrupt the java process abruptly not by API.
I think this is not a reason to get the files easily corrupted...
com.sleepycat.je.EnvironmentFailureException: (JE 6.2.31) Problem creating output files in: E:\data\database/je.info UNEXPECTED_EXCEPTION: Unexpected internal Exception, may have side effects.
at com.sleepycat.je.EnvironmentFailureException.unexpectedException(EnvironmentFailureException.java:330)
at com.sleepycat.je.dbi.EnvironmentImpl.initFileHandler(EnvironmentImpl.java:1484)
at com.sleepycat.je.dbi.EnvironmentImpl.<init>(EnvironmentImpl.java:474)
at com.sleepycat.je.rep.impl.RepImpl.<init>(RepImpl.java:267)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
at com.sleepycat.je.dbi.DbEnvPool.loadRepImpl(DbEnvPool.java:242)
at com.sleepycat.je.dbi.DbEnvPool.getEnvironment(DbEnvPool.java:178)
at com.sleepycat.je.Environment.makeEnvironmentImpl(Environment.java:251)
at com.sleepycat.je.Environment.<init>(Environment.java:232)
at com.sleepycat.je.Environment.<init>(Environment.java:188)
at com.sleepycat.je.rep.ReplicatedEnvironment.<init>(ReplicatedEnvironment.java:573)
at com.sleepycat.je.rep.ReplicatedEnvironment.<init>(ReplicatedEnvironment.java:443)
at com.sleepycat.je.rep.ReplicatedEnvironment.<init>(ReplicatedEnvironment.java:507)
Caused by: java.io.IOException: Couldn't get lock for E:\data\database/je.info
at java.util.logging.FileHandler.openFiles(FileHandler.java:422)
at java.util.logging.FileHandler.<init>(FileHandler.java:394)
at com.sleepycat.je.util.FileHandler.<init>(FileHandler.java:85)
at com.sleepycat.je.dbi.EnvironmentImpl.initFileHandler(EnvironmentImpl.java:1478)
... 24 more -
ChecksumExceptions in read only mode
I'm using SleepycatJE 3.3.87 to read from multiple BDBs that are in read-only mode. Recently I've started get ChecksumExceptions after my application runs for awhile. I tried upgrading to 5.0.58 and rebuilt my BDBs, but no luck. The error I am seeing is:
(JE 5.0.58) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 5.0.58) /apollo/env/timex/bdb com.sleepycat.je.log.ChecksumException: Location 0xf/0x6533b expected 112421487 got 3809318525 LOG_CHECKSUM: Checksum invalid on read, log is likely invalid. Environment is invalid and must be closed. fetchTarget of 0xf/0x6533b parent IN=14282 IN class=com.sleepycat.je.tree.IN lastFullVersion=0xf/0x9720a lastLoggedVersion=0xf/0x9720a parent.getDirty()=false state=0 { com.amazon.timex.classification.ClassificationException: (JE 5.0.58) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 5.0.58) /apollo/env/timex/bdb com.sleepycat.je.log.ChecksumException: Location 0xf/0x6533b expected 112421487 got 3809318525 LOG_CHECKSUM: Checksum invalid on read, log is likely invalid. Environment is invalid and must be closed. fetchTarget of 0xf/0x6533b parent IN=14282 IN class=com.sleepycat.je.tree.IN lastFullVersion=0xf/0x9720a lastLoggedVersion=0xf/0x9720a parent.getDirty()=false state=0
Any help?Hi,
I'm using SleepycatJE 3.3.87 to read from multiple BDBs that are in read-only mode. Recently I've started get ChecksumExceptions after my application runs for awhile. I tried upgrading to 5.0.58 and rebuilt my BDBs, but no luck. The error I am seeing is: I have a few questions.
Is the exception in a read-only env? And is there a process writing at the same time?
When you say you've upgraded and rebuilt your databases, do you mean that you started again from an empty environment directory? How long did it take for the error to occur again after you rebuilt the databases?
Please run the DbVerifyLog utility on the environment and post the complete stack trace along with a listing (ls -l) of the environment directory.
Thanks,
--mark -
Merging Log Files with same name
Hello,
I have log files with same name 00000000.jdb (because created at different machines). I would like to merge data stored in these two files.
I tried below approaches which do not work:
1. rename one of the files -> doesn't work as file header has file name info which doesn't match with renamed value
2. db dump/load -> dump and load works without error. (i have to dump Entity related Database and Format related Database -- from -l option) However when i try to read the file, it gives error.
Exception in thread "main" com.sleepycat.je.EnvironmentFailureException: (JE 5.0.103) Catalog could not be refreshed, may indicate corruption, errorFormatId=52 nFormats=51, . UNEXPECTED_EXCEPTION: Unexpected internal Exception, may have side effects.
I am using EntityStore and thus Annotated Entities.
Does any one has any insights?
Thanks.Yuvaraj,
As you discovered, it is not possible to use .jdb files from one environment in a different environment. The .jdb files use an internal format with relationships between files that form the Btree.
I'm glad to hear you were able to use DbDump and DbLoad successfully.
--mark -
Unexpected get for VLSN: X already waiting for VLSN: X-1
Hi all,
This early morning the following exception has been thrown on our production environment (Solaris 10 + JRE 6u17 64 bits + JE 4.1.10) when performing a transaction commit.
com.sleepycat.je.EnvironmentFailureException: (JE 4.1.10) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 4.1.10) correosbe02A(5):/bdb/Core unexpected get for VLSN: 37,054,546already waiting for VLSN: 37,054,545 UNEXPECTED_STATE_FATAL: Unexpected internal state, unable to continue. Environment is invalid and must be closed.
at com.sleepycat.je.EnvironmentFailureException.wrapSelf(EnvironmentFailureException.java:196)
at com.sleepycat.je.dbi.EnvironmentImpl.checkIfInvalid(EnvironmentImpl.java:1455)
at com.sleepycat.je.Transaction.checkEnv(Transaction.java:756)
at com.sleepycat.je.Transaction.commit(Transaction.java:224)
at eswf.zk.taskexecutor.TaskCallable.call(TaskCallable.java:47)
at eswf.zk.taskexecutor.TaskCallable.call(TaskCallable.java:20)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 4.1.10) correosbe02A(5):/bdb/Core unexpected get for VLSN: 37,054,546already waiting for VLSN: 37,054,545 UNEXPECTED_STATE_FATAL: Unexpected internal state, unable to continue. Environment is invalid and must be closed.
at com.sleepycat.je.EnvironmentFailureException.unexpectedState(EnvironmentFailureException.java:391)
at com.sleepycat.je.rep.vlsn.VLSNIndex.setupWait(VLSNIndex.java:550)
at com.sleepycat.je.rep.vlsn.VLSNIndex.waitForVLSN(VLSNIndex.java:502)
at com.sleepycat.je.rep.vlsn.VLSNIndex.awaitConsistency(VLSNIndex.java:1747)
at com.sleepycat.je.rep.impl.RepImpl.awaitVLSNConsistency(RepImpl.java:1378)
at com.sleepycat.je.recovery.Checkpointer.doCheckpoint(Checkpointer.java:712)
at com.sleepycat.je.recovery.Checkpointer.onWakeup(Checkpointer.java:507)
at com.sleepycat.je.utilint.DaemonThread.run(DaemonThread.java:162)
... 1 common frames omittedReviewing our application log there is not any previous exception and at that moment our business logic was performing several concurrent transactions.
Scenario:
- Replicated Environment with 1 master and 4 replicas.
- new Durability(Durability.SyncPolicy.SYNC, Durability.SyncPolicy.SYNC, Durability.ReplicaAckPolicy.SIMPLE_MAJORITY)
- 14735 tasks to complete.
- Each task opens a bdbje transaction.
- Up to 4 concurrent task.
- When the exception was thrown there was 4777 tasks completed and successfully commited.
After an environment restart (in fact was an application restart) everything goes fine and pending tasks was successfully completed.
Any help?
BR,
/César.Cesar,
After reviewing our code, we've found a problem in this area with the assertion that is being checked. The problem is transient and there is no change to the persistent data. After closing and reopening the environment, your environment will indeed be correct and fully consistent.
However, obviously it's still a bug, because you are getting this exception which forces you to close and reopen. We'll work on fixing that, and will let you know when a fix is available.
Our apologies,
Linda -
Adding Replication to an existing App
We're looking to add replication to our Bdb JE enabled application. We use the Direct Persistence Layer. So far, I've swapped out our use of the Environment object with that of a ReplicatedEnvironment object and wrapped our transactions with code similar to je.rep.quote.RunTransaction example. So basically, the application assumes that the local environment is master and replicates out to the group.
This works, however, when I bring the app down, the other nodes elect a new master and when I bring the app back up and try to write, the write fails because the local environment is now a Replica. This is disconcerting to me as I don't see a manual way of forcing the local environment to be the master when it comes up (after having sync'ed with the other nodes).
An alternative would be to move all persistence out of my application into remote nodes. I'm looking at the example code under je/rep. The example shows an HARouter class that can forward write requests to the master. However it looks rather low level, with the client application needing to marshall the requests to the server and the DPL code presumably would reside on the server nodes. My client code would need to change significantly to be able to write to an out of process node and I would have to keep the server DPL in sync with the client (my data's lifetime is one day, so schema changes are irrelevant).
Am I correct in this assessment, or are there APIs to more seamlessly forward writes while still offering the client the DPL API? If this is the case, are there APIs to control which node is master?
Thanks,
Adam
Edited by: 833022 on Feb 1, 2011 11:13 AMAdam,
You're right that BDB JE does not currently provide APIs to forward writes seamlessly. There's two ways to think about this.
If high availability is a priority, then you'd like your application to be able to flexibly handle the fact that mastership may have moved to any node in your group. If it can do that, you''ll be able to support writes with the least amount of disruption and delay in the event of a failover in the system. Alternatively, an application may find it preferable to have the writes serviced from a specific node. Replication is serving more like a backup, and the application is willing to incur some overhead to have mastership moved to a specific node.
In the HA-is-a-priority case, there are two options for directing write operations. You've looked at one of them, the HA aware router based on com.sleepycat.je.rep.Monitor, which directs write requests to the master. That's the HARouter example. A second option is to set up an HA unaware router, which uses the com.sleepycat.je.rep.StateChangeListener, to track the node that is the master, and redirect operations to that node. That's the SimpleRouter example.
With both options, your application would still have to do the work of forwarding. When using StateChangeListener, the code in the server nodes does it directly. The StockQuotesRMIForwarding example illustrates how to uses RMI to add forwarding to your application.
If you'd like to take the approach where you mandate who is the master, you gain some application ease, but do reduce how available your application is, in the case where the targeted master node goes down. In BDBJE 4.2, which is not yet available, we have both an API and command line utility that transfers mastership to the specified node. I've appended the description. Unfortunately, while I think it would serve your needs, it's not yet in a released version. In BDBJE 4.1, your main option would be to use ReplicationMutableConfig.NODE_PRIORITY to skew the mastership toward the desired node. If you do that, and specify the initialElectionPolicy parameter in the ReplicatedEnvironment constructor to require that all nodes in the group participate in the first election, you could ensure that your desired node is always the master. But be careful -- if that node were down, your application would stall, unable to elect a new master!
Hope this helps,
Linda
In JE 4.2, available in the future:
ReplicationGroupAdmin.transferMastership
public void transferMastership(String nodeName, int retries, int retryInterval, TimeUnit timeUnit)
throws EnvironmentFailureException,
InterruptedException,
MasterStateException,
MasterTransferFailureException,
MemberNotFoundException,
UnknownMasterException
Transfer mastership of the replication group from the currentl master to a specified replica.
This command is forwarded to the original, current master. The original master will request that the specified replica perform a syncup so its data is up to date with the original master. The period allowed for this syncup is specified by je.rep.catchupMasterTimeout, defined in ReplicationMutableConfig.CATCHUP_MASTER_TIMEOUT. When the replica is sufficiently caught up, the original master will send out messages to notify all nodes that the specified replica will be the new master.
The original master shuts down after the mastership transfer is successfully done. A MasterReplicaTransitionExcepiton will be thrown when users try to do further operations on the original master. At that point, the original master will have to be closed and reopened to resume life as a replica.
In the event of a network partition, it's possible that the original master is isolated from other nodes, and its transfer notification message is not seen by those nodes. In that case, the new master may not be the node specified in this API. To detect such unexpected transactions, this method will check to see if the specified replica is elected as a master
Edited by: Linda Lee on Feb 1, 2011 3:42 PM
Edited by: Linda Lee on Feb 1, 2011 3:53 PM -
BerkeleyDB LOG_INTEGRITY errors
Greetings!
We run BerkeleyDB embedded in our java application. Everything runs fine normally. Trouble happens if we run two instances of the same application. The folders where database files are created are separate for both. But it seems that when one stops something deletes the database files of the running one and it crashes.
A little more detail on how we run things:
init.d script (1) --> Starts a python module as root (2) ---> Starts a Java program as a regular user (3) ---> Forks off java programs that use individual instances of BerkeleyDB (4)
So if we start (3) manually things seem to run fine even if we have two instances of 4. It is only when the full thing runs that the error happens - the first process runs fine, the second process starts and completes and then the first process crashes because the entire content of its database folder is gone.
One more observation - if we set up the HOME environment variable to "/root" before starting (3) above - things do not fail. We cannot do that for obvious reasons, but this is something we found by chance.
The stack trace is attached below. If anyone can throw some light on what may be causing this that will be very very helpful.
Regards,
Arnab.
Caused by: com.sleepycat.je.EnvironmentFailureException: (JE 5.0.103) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 5.0.103) db com.sleepycat.je.EnvironmentFailureException: (JE 5.0.103) db java.io.FileNotFoundException: db/00000001.jdb (No such file or directory) LOG_INTEGRITY: Log information is incorrect, problem is likely persistent. UNEXPECTED_EXCEPTION_FATAL: Unexpected internal Exception, unable to continue. Environment is invalid and must be closed.
at com.sleepycat.je.EnvironmentFailureException.wrapSelf(EnvironmentFailureException.java:246)
at com.sleepycat.je.dbi.EnvironmentImpl.checkIfInvalid(EnvironmentImpl.java:1618)
at com.sleepycat.je.dbi.CursorImpl.checkEnv(CursorImpl.java:2859)
at com.sleepycat.je.Cursor.checkEnv(Cursor.java:4181)
at com.sleepycat.je.Cursor.close(Cursor.java:517)
at com.sleepycat.je.Database.putInternal(Database.java:1584)
at com.sleepycat.je.Database.put(Database.java:1448)
at com.veracode.webscan.scanner.BerkeleyDbStorage$Db.put(BerkeleyDbStorage.java:585)
at com.veracode.webscan.scanner.BerkeleyDbStorage$Db.put(BerkeleyDbStorage.java:615)
at com.veracode.webscan.scanner.BerkeleyDbStorage$Db.access$3400(BerkeleyDbStorage.java:568)
at com.veracode.webscan.scanner.BerkeleyDbStorage$NodeProviderImpl.setProp(BerkeleyDbStorage.java:850)
at com.veracode.webscan.scanner.BerkeleyDbStorage$Node.setProp(BerkeleyDbStorage.java:148)
at com.veracode.dscancommon.util.BaseMutablePropsAsMutableProps.setProp(BaseMutablePropsAsMutableProps.java:62)
... 34 more
Caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 5.0.103) db com.sleepycat.je.EnvironmentFailureException: (JE 5.0.103) db java.io.FileNotFoundException: db/00000001.jdb (No such file or directory) LOG_INTEGRITY: Log information is incorrect, problem is likely persistent. UNEXPECTED_EXCEPTION_FATAL: Unexpected internal Exception, unable to continue. Environment is invalid and must be closed.
at com.sleepycat.je.EnvironmentFailureException.unexpectedException(EnvironmentFailureException.java:351)
at com.sleepycat.je.log.LogBufferPool.writeBufferToFile(LogBufferPool.java:443)
at com.sleepycat.je.log.LogBufferPool.writeDirty(LogBufferPool.java:370)
at com.sleepycat.je.log.LogManager.multiLog(LogManager.java:426)
at com.sleepycat.je.log.LogManager.log(LogManager.java:350)
at com.sleepycat.je.tree.IN.logInternal(IN.java:3547)
at com.sleepycat.je.tree.IN.optionalLog(IN.java:3494)
at com.sleepycat.je.tree.IN.splitInternal(IN.java:2737)
at com.sleepycat.je.tree.IN.split(IN.java:2535)
at com.sleepycat.je.tree.Tree.forceSplit(Tree.java:1992)
at com.sleepycat.je.tree.Tree.searchSubTreeSplitsAllowed(Tree.java:1768)
at com.sleepycat.je.tree.Tree.searchSplitsAllowed(Tree.java:1318)
at com.sleepycat.je.tree.Tree.findBinForInsert(Tree.java:2246)
at com.sleepycat.je.dbi.CursorImpl.putInternal(CursorImpl.java:875)
at com.sleepycat.je.dbi.CursorImpl.put(CursorImpl.java:816)
at com.sleepycat.je.Cursor.putAllowPhantoms(Cursor.java:2397)
at com.sleepycat.je.Cursor.putNoNotify(Cursor.java:2352)
at com.sleepycat.je.Cursor.putNotify(Cursor.java:2260)
at com.sleepycat.je.Cursor.putNoDups(Cursor.java:2149)
at com.sleepycat.je.Cursor.putInternal(Cursor.java:2117)
at com.sleepycat.je.Database.putInternal(Database.java:1580)
... 41 more
Caused by: com.sleepycat.je.EnvironmentFailureException: (JE 5.0.103) db java.io.FileNotFoundException: db/00000001.jdb (No such file or directory) LOG_INTEGRITY: Log information is incorrect, problem is likely persistent.
at com.sleepycat.je.log.FileManager$LogEndFileDescriptor.getWritableFile(FileManager.java:2981)
at com.sleepycat.je.log.FileManager$LogEndFileDescriptor.access$200(FileManager.java:2682)
at com.sleepycat.je.log.FileManager.writeLogBuffer(FileManager.java:1607)
at com.sleepycat.je.log.LogBufferPool.writeBufferToFile(LogBufferPool.java:411)
... 60 more
Caused by: java.io.FileNotFoundException: db/00000001.jdb (No such file or directory)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(Unknown Source)
at java.io.RandomAccessFile.<init>(Unknown Source)
at com.sleepycat.je.log.FileManager$DefaultRandomAccessFile.<init>(FileManager.java:3181)
at com.sleepycat.je.log.FileManager$6.createFile(FileManager.java:3209)
at com.sleepycat.je.log.FileManager.openFileHandle(FileManager.java:1278)
at com.sleepycat.je.log.FileManager.makeFileHandle(FileManager.java:1245)
at com.sleepycat.je.log.FileManager.access$1500(FileManager.java:76)
at com.sleepycat.je.log.FileManager$LogEndFileDescriptor.getWritableFile(FileManager.java:2959)
... 63 moreBased on your description I'm guessing that either your script/app or something external is deleting the files. It is very unlikely that JE is deleting all the files in the environment directory. We've never had a report of such a problem anyway, and it's difficult for me to imagine how it could happen. For example, JE never cleans/deletes the two most recent files.
I should also mention that we do not certify the product on Windows, so there is a possibility that you're running into a file locking issue that we don't know about. Normally JE uses a lock file to ensure that only one JE process (JVM instance) can write to a given directory. If you are accidentally allowing two JVMs to write to the same directory, and Windows file locking is misbehaving, that might cause the problem. However, I think this is unlikely.
--mark
Maybe you are looking for
-
How can I access mail on my mac from my iPad?
I have recently converted to imac and ipad mini from Windows and so am still a newcomer to planet apple. I think I have eventually sorted myself out but with one exception in mail. I have transferred my outlook for mac filing system to mail using OL
-
HT1349 how do i transfer my music back to my computer
how do i transfer the music from my iphone back to my computer
-
Raw thumbnails will not display in Bridge CS3
I am using Bridge CS3 and Photoshop Elements 7 (the OS is Vista). I can open raw files OK via Bridge, and PSE will launch correctly as well, but the thumbnails in Bridge are only the file icons for the raw files, never images. .jpg files are fine.
-
Problem in reset cleared item by FBRA
Hi Experts Please see the below screen shot, I want to reverse the doc. no. 1500001135. But since this is a cleared item with clrn doc no 1500001178. First I have to reset doc no 1500001178. I am trying it with FBRA but reset option is not available
-
Can't get my back up from my 4s have a new 5s and disabled my 4s so need to get it and the back up icloud