Eseutil failing to repair store.

Question

Hey all, after a sleepless night I could really do with some help.
I have a 2003 exchange server where the raid controller for the array that the exchange database lives on momentarily failed (I subsequently
got the array to read again). After rescuing the edb file and copying it to another drive using the same folder structure as initially, I attempted to mount the database it found that it fails with errors that the .edb is corrupt.
Needing to rescue this database I ran exeutil /r on the necessary files on a faster machine (the exchange server was going
to take 8 hours), this one happened to be 64 bit. Sadly this terminated at 99% saying that “jcb.dll” or one of it’s dependencies is missing and to copy the file over and click OK to try again.
I searched the forums and general internets and found the many articles dictating which files it needed. I then copied them to c:/Program
Files/exchsrvr/bin, and the directory I was running the repair from, and added the new exchsrvr path to the system variables for good measure.
Nothing I did seemed to work, so I began to worry that as it was a dll error and the computer I was running this on now was 64bit that either
that was the issue, or that not having the files had corrupted my file mid repair.
I took 2 more fresh copies of the database, logs and stm file and put one lot of them on an Exchange 2010 box to try the repair from there.
First I ran eseutil /mh to check the database condition and indeed found that it reported a Dirty shutdown state requires about 15 log files to be played into the store. I then ran eseutil /ml and it returned that the log files are there and are in good health.
Good so far.
I then ran eseutil /r and replayed the logs into the database, this completed successfully – yay me!
Checking the shutdown state again, it still reports “dirty shutdown”. /sigh
I then ran eseutil /p on the database and it ran along fine, but then fails reporting an error that “jcb.dll” is missing and to copy the
file over and click OK to try… again.
We’ve done this before… I copy the file (as well as ese.dll, exchmem.dll and exosal.dll) and click OK, but it does not continue, just returns
the same error and it does not clear. In order to get out of the loop I have to close the command prompt. Result – still a dirty shutdown. 
Where I am left is that I have the slow 2003 box running /r now (85%), but as for the 2010 attempt I can’t run eseutil /d against it due to the shutdown state nor can I run it with the /g switch to check consistency. 
Internets, how can I get this database to mount?
Normally I would restore from a backup but Murphy has stepped up and decided that my backups to capture Exchange data for the last few weeks
fail quietly due to a remote agent error with Backup Exec. So … no good backup to restore from. 
I need to rescue this database, even with some data loss it would be better than restoring from a very old, questionable backup.
Suggestions?

Answer 1

So first, lets keep /r (recovery) and /p (repair or as I like to call it "salvage") separate.  In one of the latter paragraphs you say /r now 85%, but usually recovery is pretty quick, so maybe that's /p?  Or perhaps you do incremental backups
(and as you said, haven't done backup in a while), and you're doing A LOT of logs to replay (that is possible unnecessary if you've lost or clobbered your checkpoint file).
The recovery/r that "completed successfully" (but with a dirty DB), did you maybe run with /i?  If you have a single DB attached / mounted to a single log stream (i.e. NO OTHER DBs attached / mounted to that log stream), then you should never run /r(ecovery)
with the /i, as it will only lead to tears.  If you have multiple DBs attached to a log stream, then the /i option can be necessary at times (Exchange 2007 never has this case, but 2003 did allow multiple DBs to be attached).
Further warning: Once you run /r xxx /i and get success (BUT did not really get success b/c the file is still dirty), you should consider the log files "ruined", and recopy the original log files, because when /r finishes successfully it updates the log
files, and those last updates, will probably make re-tries with different /r options behave different than if you had the original log files in place (so probably best to make a 3rd copy of the log files you will be using for recovery).  You
hopefully can understand how this can cause massive confusion as the operations might behave different.
eseutil /r is very sensitive to paths, and some of the paths it uses are stored in the trx log files.  So even if you copy the DB + log files to another box, if the DB isn't at the same path and same drive letter, /r can not find the DB.  Esp.
when using /i, which allows /r to skip DBs not at the path /r-ecovery expects them to be at (probably what happened to you above).  Consider using the /d option on /r(ecovery) if it exists in Ex2k3 when trying to recovery the DB on other drives or
paths.
Here is my typical approach (to recovery a clean DB on a different machine at a different path):
Make sure the server is offline or the DB is dismounted to avoid any interference while you xcopy in steps 2 and 3.
xcopy priv01.edb (the DB) to the new path
xcopy e01*.log (the logs) to the _same_ new path as the DB.
Optionally trim the unneeded log files from the new path (or at step 3 copy only the needed log files) ... the needed log files are the ones from eseutil /mh priv01.edb see the MinRequired log generation, to the Max Committed (in Ex2k7+) or Max Required
(in Ex2k3), and anything above or below these numbers is not needed ... though you should seriously consider keeping the larger log generation files over the max as that just saves more data.
cd into the new path
Run: "eseutil /r e01 /d."  Yes that's a dot at the end.
This is predicated on eseutil.exe having the /d option, which I really don't know about for Ex2k3.  The /d option says to eseutil even when you're running /r and you find a path to the DB listed in the trx log files, please try to find the DB file at
this alternate path instead.  The dot for "/d.", says that this alternate path to look in should be the current working directory which you cd'd into in step 5. ;-)
At that point you should have a clean DB to get your DB recovered.
Alternatively if you have to run /p(repair) then you could just copy the DB to another drive on the box, and run /g directly on the box you have Exchange installed on right?  Then it should surely work.
Cheers,
Brett Shirley [MSFT]
ESE Developer (Yes, I work on the actual ESE DB engine, eseutil.exe, ese.dll, etc)
LCA makes me say this: This posting is provided "AS IS" with no warranties, and confers no rights.
And as a member who has only added features to Exchange/ESE since 2003, I'll note that Exchange 2003 I _think_ is no longer supported. ;-)