Major Mail Service Panic after upgrade! Customer about to kill me!

Question

Me again.
As mentioned in another thread, I had to upgrade an Xserve from 10.3.9 to 10.4.3 for a customer. When I ran the upgrade I discovered that the volume containing the mail database was damaged. I wasted a day getting the data off this onto another drive, then did the upgrade. Because the mail database had been moved and might have the odd missing file I used mailbfr -o and mailbfr -f at this point. Now, all client mail folders are back as they should be but all new mail is stuck in the queue. Outgoing mail can be sent successfully.
Now, I know an amateur like me shouldn't be messing with the config files willy-nilly to try and fix this without a clear idea of what's going on. However, given that the customer and his 20 staff are going mad waiting for their mail I tried to get it right by directly comparing their config files with the equivalents on my own 10.4.3 server which is set up more or less identically and is working.
I will get lynched if I can't make this work soon. Here's postconf -n and a snippet of the smtp log. Please someone look at this and save my life!!
mail:/var/log root# postconf -n
alias_maps = hash:/etc/aliases,hash:/var/mailman/data/aliases
always_bcc =
command_directory = /usr/sbin
config_directory = /etc/postfix
content_filter = smtp-amavis:[127.0.0.1]:10024
daemon_directory = /usr/libexec/postfix
debugpeerlevel = 2
enableserveroptions = yes
inet_interfaces = all
luser_relay = postmaster
mail_owner = postfix
mailboxsizelimit = 0
mailbox_transport = cyrus
mailq_path = /usr/bin/mailq
manpage_directory = /usr/share/man
mapsrbldomains = dun.dnsrbl.net,relays.ordb.org,bl.csma.biz,bl.spamcop.net,sbl-xbl.spamhaus.org
messagesizelimit = 0
mydestination = $myhostname,localhost.$mydomain,mail.mandm.uk.com,192.168.1.100,mandm.uk.com,re ed-munkenbeck.com,212.18.249.218,mail.reed-munkenbeck.com
mydomain = mandm.uk.com
mydomain_fallback = localhost
myhostname = mail.mandm.uk.com
mynetworks = 127.0.0.0/8
mynetworks_style = host
newaliases_path = /usr/bin/newaliases
ownerrequestspecial = no
queue_directory = /private/var/spool/postfix
readme_directory = /usr/share/doc/postfix
recipient_delimiter = +
relayhost =
sample_directory = /usr/share/doc/postfix/examples
sendmail_path = /usr/sbin/sendmail
setgid_group = postdrop
smtpdclientrestrictions = permit_mynetworks hash:/etc/postfix/smtpdreject rejectrblclient bl.csma.biz rejectrblclient dun.dnsrbl.net rejectrblclient relays.ordb.org rejectrblclient bl.spamcop.net rejectrblclient sbl-xbl.spamhaus.org permit
smtpdenforcetls = no
smtpdpw_server_securityoptions = cram-md5
smtpdrecipientrestrictions = permitsasl_authenticated,permit_mynetworks,reject_unauthdestination,permit
smtpdsasl_authenable = yes
smtpdtlsloglevel = 0
smtpduse_pwserver = yes
smtpdusetls = no
unknownlocal_recipient_rejectcode = 550
virtualmailboxdomains =
mail:/var/log root#
smtp log:
Jan 5 12:35:17 mail postfix/master[20503]: daemon started -- version 2.1.5
Jan 5 12:35:17 mail postfix/qmgr[20506]: 3C1983B7F0D82: from=<[email protected]>, size=5704, nrcpt=1 (queue active)
Jan 5 12:35:17 mail postfix/qmgr[20506]: 8C51A3B7F13CA: from=<[email protected]>, size=4047, nrcpt=1 (queue active)
Jan 5 12:35:17 mail postfix/qmgr[20506]: 91B7D3B7F13A8: from=<[email protected]>, size=2638, nrcpt=1 (queue active)
Jan 5 12:35:17 mail postfix/qmgr[20506]: CF9FB3B7F13AF: from=, size=5145, nrcpt=1 (queue active)
Jan 5 12:35:17 mail postfix/pipe[20510]: 3C1983B7F0D82: to=<[email protected]>, relay=cyrus, delay=4489, status=deferred (temporary failure. Command output: couldn't connect to lmtpd: Connection refused_ 421 4.3.0 deliver: couldn't connect to lmtpd_ )
Jan 5 12:35:17 mail postfix/pipe[20517]: 8C51A3B7F13CA: to=<[email protected]>, relay=cyrus, delay=2301, status=deferred (temporary failure. Command output: couldn't connect to lmtpd: Connection refused_ 421 4.3.0 deliver: couldn't connect to lmtpd_ )
Jan 5 12:35:17 mail postfix/pipe[20514]: CF9FB3B7F13AF: to=<[email protected]>, relay=cyrus, delay=2307, status=deferred (temporary failure. Command output: couldn't connect to lmtpd: Connection refused_ 421 4.3.0 deliver: couldn't connect to lmtpd_ )
Jan 5 12:35:17 mail postfix/pipe[20515]: 91B7D3B7F13A8: to=<[email protected]>, relay=cyrus, delay=2307, status=deferred (temporary failure. Command output: couldn't connect to lmtpd: Connection refused_ 421 4.3.0 deliver: couldn't connect to lmtpd_ )
Other symptoms since upgrade: Server Admin refuses to turn on Mailing Lists. There's one list that was in use before the upgrade. If I click 'Enable Mailing Lists' and press Save, the box goes unchecked again as soon as the screen refreshes. Mailaccess.log (as shown in SA as the IMAP log)remains empty. Like the Mailing Lists issue above, I can't change the log detail levels; they just reset to what they were before.
Anything else anyone wants to know I'll post it.
Xserve G5 2.0DP, 2Gb RAM, 10.4.3 (u/g from 10.3.9)   Mac OS X (10.4)  
G4 and Xserve   Mac OS X (10.4)  

Answer 1

aha. a-bloody-ha.
I'll live to fight another day. The log entries about the stage. folder got me thinking and I realised that there was something wrong with the permissions. because I'd had to 'rescue' the mail database by moving to another volume, the permissions weren't right. Now, I had run mailbfr and watched it say it was fixing the ownerships and permissions on all this; I shouldn't have taken this for granted. The stage. folder was owned by me (admin) with read-only for everyone else. I changed it (heck, I changed the whole directory tree) to read-write for everyone and instantly mail started pouring in to everyone's accounts.
Everything is now working normally. It still bothers me that the whole shebang seems so fragile - especially in so far as the Server Admin app has at best only a tenuous grasp on the config files it's supposed to be supporting. Looking back over the past two days I can see that the basic setup was right in the first place but the manual changes I've been making in the various conf and cf files were ones that SA was supposed to be able to do for me. The whole point of having Server Admin>Settings>Logging is so you can set the log level right there - not in bloody pico via the bloody command line. Sorry - I'm tired and a little punch-drunk. It also bothers me that, having learned my humble skills on AppleShare 6 (and MacDNS), we're now in a place where none of the key components of the server are actually made by Apple, documented by Apple, or properly supported by Apple. Since Mail Service isn't a closed box with 'Apple' written on it, but a collection of dozens of interconnected third-party processes working in tandem, there really ought to be a 'master document' that shows the complete flow through the system. Most of you guys are proper experts and I'm truly grateful you're there - but I'm not afraid to admit that I didn't actually know that every message had to pass though a folder called stage. on its way to delivery and it would have nice to have been able to find that information in one of Apple's manuals. If I'd known I might have looked at it sooner.
The customer in this case is a firm of architects; they're creatives; non-technical; Apple-lovers precisely because they love the ease of use, the style and the non-pointy-headed approach of Apple's kit. I'm reminded that when this customer bought this Xserve 18 months ago, I made them buy the 3 year service contract for it. But when during the initial setup NAT wouldn't work properly I called Apple for help and they told me that 'setup problems aren't covered'. Good Grief. 450 quid. And let's not forget 700 quid for the 10.4 software.
Now, having saved myself from being lynched by the customer I'm probably going to be strung up by the very people on this list that people like me have to rely on when we get into trouble.
So, to save you all the trouble - yes, I'm a rank amateur; yes, I should do more homework; yes, I promise to keep googling and searching and reading and playing with my in-house machine... I don't promise to do a course on it since 'MacOs Server Essentials' is $2,000 to start with, and I don't have that kind of money.
Think I'd better go have a lie down.