Reconstruct problems after shutdown / startup

Question

Yesterday we had an air conditioning outage in our data center, so I had to shutdown our iMS system. I forgot to do a stop-msg on the mail server, so it apparently did not get shut down very cleanly. When I booted back up later in the day, I had to run fsck on the data partition (which is on an external disk array) many times before it would boot up successfully.
I was able to start up the messaging server fine, but I monitored the logs and found errors like this one (from the imap log):
[30/Jul/2006:22:00:02 -0500] opal imapd[813]: Store Error: Unable to get quota root user/sch736: Unknown code ____ 248
[30/Jul/2006:22:00:02 -0500] opal imapd[813]: Store Error: Unable to add new mailbox entry user/sch736: System I/O error. Administrator, check server log for details.
[30/Jul/2006:22:00:02 -0500] opal imapd[813]: Store Error: Unable to auto create mailbox user/sch736: System I/O error. Administrator, check server log for details.
I was getting similar errors all over the place, so I decided to rebuild the mboxlist.
running reconstruct -r -f doesn't do anything at all
reconstruct -m gives me the following:
partition primary is at /opt/luminis/iplanet/server5/msg-opal/store/partition/primary
user/aab753: fixed quota root usage
user/abm909: cannot fix quota root usage: I/O error
and puts the following in the default log:
[31/Jul/2006:08:52:47 -0500] opal reconstruct[25053]: General Notice: ./reconstruct -m
[31/Jul/2006:08:52:47 -0500] opal reconstruct[25053]: Store Critical: Unable to open index file for user/aab753/School/Comm 170 group: No such file or directory
[31/Jul/2006:08:52:47 -0500] opal reconstruct[25053]: Store Critical: Mailbox database error: folder.db: page 2204 doesn't exist, create flag not set
[31/Jul/2006:08:52:52 -0500] opal reconstruct[25053]: Store Critical: Mailbox database error: folder.db: page 2204 doesn't exist, create flag not set
[31/Jul/2006:08:52:52 -0500] opal reconstruct[25053]: Store Critical: Mailbox database error: folder.db: page 2204 doesn't exist, create flag not set
reconstruct -n -r runs for a while but then dies at
user/abm909/Sent
and puts the following in default log
[31/Jul/2006:08:57:02 -0500] opal reconstruct[25066]: General Notice: ./reconstruct -n -r
[31/Jul/2006:08:57:11 -0500] opal reconstruct[25066]: Store Critical: Mailbox database error: folder.db: page 2204 doesn't exist, create flag not set
I also was receiving the following error while running reconstruct -r
cannot fix quota root usage: I/O error
but I think that is because I had copied the files out of the mboxutil folder at the time.
Any ideas on how to fix what's going on?
Thanks,
Greg

Answer 1

HI Jay,
Sorry I disappeared for a while. This server is part
of a SunGard Higher Education Luminis install, so I
was dealing with their support for a while trying to
get things resolved.I understand.
>
We got things working ok by following your steps from
http://swforum.sun.com/jive/thread.jspa?threadID=53262
&messageID=204166
A. stop the messaging server. Confirm that ALL
processes related to Messaging are stopped. Use ps
-ef, and grep for the path it's installed in. Kill -9
any processes that are still there.
B. Remove all contents of the mboxlist directory.
C. Start the server up. It will work normally, but
users will only have access to INBOX folders.
D. Run reconstruct -m to fix folder access.
However, after doing that, some users were getting
the following error in the Luminis IMAP client:
[error: We could not load the Email Center because
one of the folders
was invalid. Error given was: MailException: type=2
(EXCEPTION_InvalidFolder), description: Error getting
subfolder, Sent ]
And some users can get in fine, but are missing their
personal subfolders.
So they had me run these steps:
1) Stop the messaging server in its entirety
(./stop-msg and check for related processes to make
sure they've stopped running. On NT/Windows 2000,
if they're set to start up automatically change them
to Manual in the Services control panel)
) Make a backup of the store directory, as well as
the msg-instance/store/mboxlist folder contents. It
is recommended you copy the contents of mboxlist out
to an easily accessible backup directory -- if later
required to restore from backup, this will make the
mboxlist folder readily available instead of having
to do a full restore.
3) run reconstruct -m
4) run a script to move all store.idx to
store.idx.old -- See the steps below on how to do
this.
5) start up store by doing:
./start-msg store
and wait until store posted all transactions in the
mboxlist/log.* files -- sometimes this may take 1-2
hours, because it has to go through every transaction
that hasn't completed the update on the mailbox
database
6) run reconstruct -r
7) stop-msg store
8) backup and move out the contents of
iplanet/server5/msg-instance/store/mboxlist to a
place we can get back to later
9) run reconstruct -m again
10) ran "start-msg store" and let it complete, once
again - this usually doesn't take as long unless new
transactions were posted
11) start the rest of the messaging server processes
-- by running ./start-msg, and let it ignore the
store process since that is already running, or start
up all other services manually (i.e., pop, http,
imap, etc)
Which I did, until step 9, when it gave me the
following error:
reconstruct: cannot list quota roots: System I/O
error. Administrator, check server log for details
So I started the messaging server and then
reconstruct -m ran successfully, but we're back in
the exact same boat with the same users having the
same problems.
I can fix the users getting the IMAP error by moving
all the store.idx files and doing a recontruct -r on
the user. However, I don't know who all is affected
and don't really want to do that one by one.
I don't know how to fix the subfolder problem.Each folder has a store.idx file.
reconstruct -r -f may fix all the problems at one go. If not, parse the log files for affected users, and remove store.idx files, and reconstruct.
You MAY need to apply some later hotfix/patch, as some work has been done in the reconstruct area. I don't remember exactly what version you're on.
>
>
Thank you so much for any help you can provide. I
hope I am giving you good information to go off of.
Thanks,
Greg