Mail Server suddenly stopped working

Question

My mail server has stopped working, and I can't figure out why. This is a brand new MacMini with the latest Maverics Server (preinstalled) that I just setup last month. (We migrated from an Xserve running 10.6.) Up to this point, mail has worked flawlessly.
This afternoon I used the Server app to create a new user. I then went about setting them up for ssh access and installing a public/private keypair (this is for a git repository). Around the same time, my mail server simply stopped working. As far as I know, no other configuration changes were made.
When I go into the Mail service in the Server app the status perpetually says "Starting...". The 'postfix status' command says that postfix is not running. I've checked every log and error file, and I can only find three anomolies:
The SMTP log file reports:
Feb 11 01:31:06 mail.gloaming.com postfix/postmap[7152]: warning: /Library/Server/Mail/Data/scanner/amavis/local_domains.db: duplicate entry: "mail.gloaming.com"
This warning has been logged before, several weeks earlier, so I assume it's not a serious problem.
The other warning is:
Feb 11 00:10:49 mail.gloaming.com postfix/postfix-script[23141]: warning: not owned by _postfix: /Library/Server/Mail/Data/mta/./guid_device_maps.plist
Other threads seem to indicate that this warning is benign.
The only other problem that looks really suspicious (to me) is in the system.log:
Feb 11 01:22:52 mail.gloaming.com ocspd[4947]: ERROR: unable to get the receiver data from the DB!
I'm not sure what that means, but it doesn't sound good.
Some threads point a finger at the virus and junk mail scanners. If I disable all mail filtering, the mail server status will finally change to "Serving mail at mail.gloaming.com". But it isn't. All attempts to get mail from the server are refused.
(Oh, and I've restarted the server several times.)
The postconf command tells me that the postfix system isn't accepting outside connections:
inet_interfaces = loopback-only
So it would appear that Server is setting this. Any attempt to change it back to "= all" fails, as the server software overwrites it with "= loopback-only" at its first possible opportunity. (Editing the main.cf file and reloading postfix doesn't fix it either.)
Here's the really strange part. The server just started do this today. I recovered a backup of the main.cf file from yesterday and diffed it with the one that's not working:
mail:postfix root# diff /Library/Server/Mail/Config/postfix/main.cf /Users/admin/Desktop/main.cf
675c675
< inet_interfaces = loopback-only
> inet_interfaces = all
682c682
< smtpd_helo_restrictions = reject_non_fqdn_helo_hostname reject_invalid_helo_hostname
> smtpd_helo_restrictions = reject_invalid_helo_hostname reject_non_fqdn_helo_hostname
692c692
< mydestination = $myhostname, localhost.$mydomain, localhost, mail.gloaming.com
> mydestination = $myhostname, localhost.$mydomain, localhost, mail.gloaming.com, $mydomain
712d711
< postscreen_dnsbl_sites = zen.spamhaus.org*2
As you can see, the Server software has deliberatly started setting inet_interfaces to loopback-only and I can't figure out how to get it to set it back to "all".
Some of the other threads have suggested "wipe and reinstall", but I've spent three weeks setting up this sever and shipping it to a co-location facility—starting over really isn't an option.
I'd really just want to know why the Server software is suddently configuring the mail server so it won't receive any mail!

Answer 1

The mystery continues.
This morning, I decided to take a fresh look at what's going on by recovering a backup copy of my entire /Library/Server/Mail/Config directory from last month. Looking at the item capture list, it shows that none of the files in my Mail/Config directory have changed since late last month.
Here are the disturbing changes.
In Server/Mail/Config/dovecot/dovecot.conf, there is one different line.
working:
     protocols = imap pop3 lmtp sieve
now:
     protocols = lmtp sieve
So clearly, the server doesn't think we should be running IMAP or POP. Which is probably the root of the problem.
In Server/Mail/Config/postfix/mail.cf, there are a number of minor/cosmetic differences, but the problematic ones appear to be
working:
     inet_interfaces = all
     mydestination = $myhostname, localhost.$mydomain, localhost, mail.gloaming.com, $mydomain
now:
     inet_interfaces = loopback-only
     mydestination = $myhostname, localhost.$mydomain, localhost, mail.gloaming.com
Again, this configuration would imply that there are no public mail services.
Some other odd change occured in the Server/Mail/Config/postfix/main.cf file
working:
     smtpd     pass  -       -       n       -       -       smtpd
     pickup    fifo  n       -       n       60      1       pickup
          -o content_filter=smtp-amavis:[127.0.0.1]:10024
now:
     smtpd     pass  -       -       n       -       -       smtpd
          -o receive_override_options=no_address_mappings
     pickup    fifo  n       -       n       60      1       pickup
          -o content_filter=
Reading through some other posts, I also found another inexplicable change reported by other users having similar problems. The /System/Library/LaunchDaemons/org.postfix.master.plist suddenly changed.
working:
    <key>OnDemand</key>
    <false/>
    <key>ProgramArguments</key>
    <array>
        <string>master</string>
        <string>-c</string>
        <string>/Library/Server/Mail/Config/postfix</string>
    </array>
now:
    <key>OnDemand</key>
    <true/>
    <key>ProgramArguments</key>
    <array>
        <string>master</string>
        <string>-c</string>
        <string>/Library/Server/Mail/Config/postfix</string>
        <string>-e</string>
        <string>60</string>
    </array>
The postfix service is now "OnDemand" with a new, and myserious, "-e 60" option added. (I can find no documentation for an -e option in the postfix control tool.)
Ultimately, I "fixed" the problem by restoring the backup copy of my entire /Library/Server/Mail/Config folder, the /System/Library/LaunchDaemon/org.postfix.master.plist file, and immedately restarting the server. The mail servers started and everything appears to be normal.
But this whole experience has made me very learly of the Server app. Why did these configuration changes occur? What settings or circumstances caused them to spontaniously change to an, essentually, unusable state? I'm now afriad of making any changes via Server.app for fear it will decide I shouldn't have a working mail server again.
If anyone has any insights, I'd ver very interested in hearing them.
Message was edited by: James Bucanek