Log sequence error - possible causes?

Question

We're using C++, DBXML 2.1.7, with underlying Berkeley 4.3.28 - core 5 linux (2.6.16.28). We use transacted write's, with no nesting of transactions. We've been running with this version of DBXML for some time and this is the first time we've seen any sort of data corruption with the database.
In this case, the database server was shutdown, and the system restarted - on restart, the database server core dumped. Repeated attempts to restart the database gave the same failure. We enabled error output for the Berkeley DB and we get the following errors:
Finding last valid log LSN: file: 1 offset 8234100
Recovery starting from [1][7965842]
Log sequence error: page LSN 1 1664073; previous LSN 1 5236280
Recovery function for LSN 1 8228918 failed on forward pass
PANIC: Invalid argument
PANIC: fatal region error detected; run recovery (repeated several times)
followed by a segfault in libdb_cxx-4.3.so.
So I have 2 questions, the first (and most important) being - how can the log file get corrupted? Is this an OS/file system problem? Or could we have a problem in our database server? It's relatively simple - there is a single thread for read's/write's, and a separate "checkpoint" thread that periodically calls the txn_checkpoint function. Something was just changed on the system that has to do with mirroring, specifically on the partition that holds our database, but I don't know the details (I can get the info, though).
The second question - why is Berkeley choking on the error path, instead of causing a database panic? Granted, in this situation it would appear that we're hosed either way, but a panic is at least a little more user-friendy than a core dump. Looking at the core file, it appears that we've entered the error handling portion of dbenv_open, and the mp_handle of the environment object is NULL - we fail in the call to __dbenv_refresh because of that. (If you're interested, we rebuilt Berkeley with debug symbols - I can give you a stack trace with details for the segfault.)
Oh, the startup flags for the database server are: DB_CREATE|DB_INIT_LOCK|DB_INIT_LOG|DB_INIT_MPOOL|DB_INIT_TXN|DB_RECOVER|DB_THREAD
Thanks!
Wendy

Answer 1

Thanks Michael -
Here's the stack trace from one of the core files, generated with the debug version of the library:
#0 0xb7d97c3a in __dbenv_refresh (dbenv=0x80b5430, orig_flags=1024,
rep_check=0) at ../dist/../env/env_open.c:722
#1 0xb7d9993d in __dbenv_open (dbenv=0x80b5430,
db_home=0x80b5334 "/pivot3/repository/xml", flags=188513, mode=432)
at ../dist/../env/env_open.c:415
#2 0xb7d1fe78 in DbEnv::open (this=0xbfa94488,
db_home=0x80b5334 "/pivot3/repository/xml", flags=188513, mode=0)
at ../dist/../cxx/cxx_env.cpp:442
#3 0x0804e0e8 in main (argc=134537448, argv=0xbfa94488) at dbserver.cxx:179
As to the "why's": the database files are never moved (in normal operation - I did copy them to a similar system for debug, but the failure is the same on both systems), we always run recovery when we restart the server, we never physically access the environment outside of the server, and there is only one database server on the system, so no "cross contamination" from another server (basically, we're using DBXML as an embedded database to store configuration information - we have it running on multiple, identical systems, and this is the first time in over 2 years that we've seen any sort of database corruption).
So based on what you've said, the only other real possibility is if something happened to the partition holding the log file (all of the database files are on the same partition). Would there be any smoking guns we could look for on the physical system that might indicate what happened? The system that the failure originally occurred on is still in the 'last booted' state, in case there was anything that we could look for. The partition mirroring changes were just made this past week, which is why we thought it might be something at a lower level.
I just ran db_printlog and the entry for 1664073 looks like this:
[1][1664073]__bam_repl: rec: 58 txnid 80000047 prevlsn [0][0]
fileid: 21
pgno: 3
lsn: [1][8228918]
indx: 3
isdeleted: 0
orig: 0xc
repl: 0x11
prefix: 8
suffix: 23
The other record (5236280) does not exist in the log file, based on the output from db_printlog.
If there's anything else from the log print that you need, just let me know.
Thanks!