Corruption on Windows 7 ?

Question

I've recently moved my development machine to 64bit Windows 7 build 7600 and Sun 32bit JDK 1.6.0_20.
I've seen DbChecksumException 3 times now when working on a dataset of about 20gb in size. I didn't track it the first two times, but the exception from the third instance is below. The first two times I removed the log files and reloaded them from a bulk import to clear the error. I'm not sure, but I think the other exceptions occured in my program threads and not the cleaner thread.
This is on a transaction-enabled database using the collection APIs. Has anyone seen similar issues on Windows 7? What information would be helpful to troubleshoot this?
<DaemonThread name="Cleaner-1"/> caught exception: com.sleepycat.je.log.DbChecksumException: (JE 3.3.87) Location 0x23/0x3fb8ee expected 4034848955 got 3710585889
com.sleepycat.je.log.DbChecksumException: (JE 3.3.87) Location 0x23/0x3fb8ee expected 4034848955 got 3710585889
     at com.sleepycat.je.log.ChecksumValidator.validate(ChecksumValidator.java:96)
     at com.sleepycat.je.log.FileReader.validateChecksum(FileReader.java:593)
     at com.sleepycat.je.log.FileReader.readNextEntry(FileReader.java:314)
     at com.sleepycat.je.cleaner.FileProcessor.processFile(FileProcessor.java:386)
     at com.sleepycat.je.cleaner.FileProcessor.doClean(FileProcessor.java:233)
     at com.sleepycat.je.cleaner.FileProcessor.onWakeup(FileProcessor.java:138)
     at com.sleepycat.je.utilint.DaemonThread.run(DaemonThread.java:141)
     at java.lang.Thread.run(Thread.java:619)

Answer 1

After the checksum exception, are you able to still open the database with an application or is the environment corrupted permanently?In this case, the enviroment and all databases open and close happily. It's just the cleaner thread that seems to be having problems. I'm not sure what problems I would have if I kept using the app, I havent really tested anything much after I spotted the error.You'd eventually run into it if you read that piece of data from the log with your app.
>
BTW, the verifyChecksums just checks the checksums when they're being written out as well as after they're written out. It's interesting that it's not being detected at write time.Perhaps I should try adding a read value after write during my population routine, it probably will be happy too. I suspect windows odditties. My current theory, which I base on absolutely nothing, is that for a jdb file to exhibit this problem it has to be evicted from the OS disk cache. Maybe I'll try a 2gb data set and a reboot in between....If you do a read after write you'll have to make sure that the data you're reading is not in the cache, otherwise it will just take the cached data.
DbPrintLog will accomplish this, but, it will do it by reading every record in the log, whether or not they are "live". You could do a DbDump, or use your own app to do a read of the entire database. Those programs will only read "live" data. It would be interesting to know if the thing it is stumbling upon is live or dead. That said, since you're doing a load, it is probably live.
If you do DbPrintLog -s 0x439 on the corrupted log environment that you have does it blow out at the same lsn? 0x439/0x78a1a0. Are you able to give us that file (0x00000439.jdb) or does it have confidential info in it? Is 00000439.jdb the last file or are there more after it?DbPrintLog barfs at the same place. 439.jdb is faiirly elderly among the 2800 log files, d84 is the youngest.
In a previous post you said that you are setting CLEANER_MIN_UTILIZATION:
environmentConfig.setConfigParam(EnvironmentConfig.CLEANER_MIN_UTILIZATION, String.valueOf(10* 1000 * 1000));
Do we really accept a value of 10,000,000 for a percentage without throwing an exception at y?Busted. I goofed during copy and paste, the values on those lines are actually constants from another class so I pasted in the values for readability. The real value for that line should be 80. The other mistake I just noticed is I'm on 1.6.0_16, not 1.6.0_20.OK, no problem. That explains that.
>
Also, it seems like 10MB of cache is pretty small for a 20GB databaseTrue, and only 64mb heap. But this should result in slowness, or possibly an OOM (with potential for subsequent corruption), but I'm just seeing the corruption. Now that I'm getting closer towards repeatability, maybe I'll try with bigger heap/cache. It basically takes an overnight data load for a test cycle now, I have to use the machine during the day for other stuff.I expect that this will be squirmy. You may be able to reproduce it on a smaller run some of the time, but not as often. Just a guess.
I'm not sure about sending the log file / app, I'll have to check on that...
I'm still focusing on trying make a reproducible test case. So far there isn't any real impact to this problem since the only place it exists is on my laptop. It's just a curiosity, at least until it happens on a real machine :)I'd like to have you try 4.0 on this if that's ok with you. I can't post the download link to it on the forum, but I can email it to you. If that still shows the bug, then I'll start giving you debugging jars built on 4.0.
What is bothersome is that with the verify checksums enabled we are doing a read-after-write (see src/com/sleepycat/je/log/FileManager.java:1344).
Edited by: Charles Lamb on Oct 22, 2009 9:07 AM