IPlanet Mesaging Server 5.1 SP1 (startup hangs on STORED)

Question

I've got a case open with support, but I'm reaching a desperate point here as we've been without mail for over 24 hours now. I restored our entire Messaging Server file system from backup yesterday after a failed disk took out the system. (Sidenote: the restore from tape worked up to a point but it didn't seem to set owner:group on some of the restored files. Is there any resource that explains which dirs should be owned by which users for Messaging Server to work correctly?) After restore, I tried to start up the Store Daemon portion of Messaging Server with the following command:
/iplanet/server5/msg-servername/start-msg store
After a long period of waiting, I got the following error:
bash-2.03# ./start-msg store
/iplanet/server5: Starting STORE daemon .................failed.
ERROR: Failed to find process file after 90 seconds:
/iplanet/server5/msg-servername/config/pidfile.store
Unable to verify if the STORE daemon started.
Check the log file(s) under: /iplanet/server5/msg-servername/log/default
ERROR: The following services failed to start, or were already:
STORE daemon
After that I went to the log location and found a 'default' log file that was two days old:
bash-2.03# ls -lsa
total 4884
2 drwxr-xr-x 2 root other 512 Mar 24 18:42 .
2 drwxr-xr-x 9 root other 512 Mar 24 09:24 ..
576 -rw------- 1 mailsrv nsgroup 286187 Mar 22 20:50 default
I assumed that something must be wrong with the store directory, so I looked around a bit. I found that permissions on that dir were set for everything to be owned by root:sys. I assume this is wrong since most of the utils that operate on the directory have to be run by the mailsrv:nsgroup user/group combo, so I ran a chown -R mailsrv:nsgroup on the store directory (about 12-13 gigs of mail in it). I tried to start the store daemon again and got the same error. So I went a little farther and started a reconstruct -r. That process started at about 11:30PM EST last night and it's still running. Using sar -u and sar -d to monitor CPU and disk use, I'm seeing very negligible use. Is reconstruct -r still actually doing anything? Is there any other way to tell? I saw in the docs that a reconstruct can take a long time, but they had an 18 hour period for an 85 gig mail store. Just for basis of comparison, this is running on an Enterprise 220R at Solaris 8 02/02REV. Any suggestions would be extremely appreciated.

Answer 1

It turned out to be owner:group problems. There was only one log.* file in the SERVEROOT/store/mboxlist directory. The problem was resolved by using 'truss' with the appropriate options when starting the store daemon with './msg-start store':
truss -fear all -w all -o stored_truss.txt ./start-msg store
Then I used grep to scan the file for the EACC string. This helped me figure out which dirs needed to be owned by the user and group that the mail server processes run under. (Thanks to [email protected] for the suggestion of 'truss')
At the moment I am running a 'reconstruct -r' and it seems to be doing what it's supposed to. When I ran it last night, I couldn't remember if the store daemon needed to be running or not, but apparently it does.