Authentication freezing occasionally in OD

Question

We're having problems with our two Xserve G4s which are both running OS X Tiger Server 10.4.6. Call the first server "filesrvr" and the other "mailsrvr" (which in addition to mail also provides web and file sharing services). Both are running behind a firewall with mailsrvr on a DMZ visible to the Internet. The filesrvr is the DNS and OD Master, and the mailsrvr a DNS slave and OD replica. The mailsrvr is operating in a beta capacity at the moment while we address these problems.
As far as I can tell, the DNS and basic aspects of OD are working correctly: both systems have proper forward and reverse local DNS names, and the hostname is being set correctly on both systems. We had problems with a prior incarnation of the servers because DNS and OD were set up in a funny way and in the wrong order: this led to all sorts of lookupd-related crashes, but the servers have been wiped clean since and carefully rebuilt from scratch. We haven't seen anything that looks like the prior problem and both servers are much more stable now.
That being said, there still appear to be two problems, both on the mail server side. Any suggestions would help:
1) Every few days, OD-related authentication freezes up for a period of time. Sometimes it will come back on its own within fifteen or twenty minutes, at other times we must do a hard restart to reset the problem. The affected services are: Tiger Server Admin (port 311), SMTP, POP3, IMAP, SSH, AFP, SMB, and FTP. Console logins are also rejected during these periods. I'll supply details for what I think is happening here with log listings.
2) Several times a day (say two to three times a day), the Tiger Server Admin (servermgrd on port 311) becomes non-responsive. It doesn't matter whether you run the Server Admin client application locally on the Tiger Server, or via another machine. It also doesn't seem to matter if you have an active session already running or not. It just seems to cut in and out on its own. The good news is that usually within ten minutes it always comes back up on its own. I don't find anything in the logs related to servermgrd during these outages. I also don't see anything 'funny' about the instance of servermgrd running at the time.
I'm not sure if these two problems are related. The authentication issue seems more serious since it affects the availability of mail service to our users which needs to be up and running at all times.
I'll supply details below, but the bottom line is that I suspect memberd and/or lookupd (or something they both depend on -- like the NetInfo database) is not working quite right. I've been reading up on postings related to these two daemons and wonder what would be an appropriate course of attack.
In <http://discussions.apple.com/thread.jspa?threadID=492829>, one recommendation is to run "/sbin/fsck -y" in safe mode looking for any problems, and then to manually rebuild the NetInfo database. Obviously memberd and lookupd depend on their caches and the databases they make use of to be in working order. The suggestion is to check for file system issues and rebuilt the NetInfo database.
In <http://discussions.apple.com/thread.jspa?threadID=132009> the recommended course of action is to flush the memberd cache using "sudo memberd -r" from time to time (elsewhere it is suggested to do this once an hour via a crontab entry, and also to issue a "/usr/sbin/lookupd -flushcache"). The idea here is to make sure the caches are kept fairly fresh.
I'm a bit torn about how to attack this problem -- it seems a no-brainer to:
1) Run Disk Utility "Verify Disk" and "Verify Permissions" checks on filesrvr and mailsrvr.
2) Run "/sbin/fsck -y" on filesrvr and mailsrvr until no problems are reported.
3) Insert the following line into the root user's crontab to clear out the memberd and lookupd caches once an hour:
@hourly /usr/sbin/memberd -r; /usr/sbin/lookupd -flushcache
I'm a little hesitant to rebuild the NetInfo database right now. There's a number of special accounts defined there that will simply be a pain to rebuild. My thinking is that I'll make the three changes above and stop just short of rebuilding NetInfo. If authentication fails again on mailsrvr in a similar way then I'll run the Disk Utility and fsck fixes again but also go one step further and rebuild NetInfo.
Does anyone have thoughts for other things to try?
<details>
We've seen the first type of failure (authentication stops working) four times in the past two weeks, and it's happened in a number of different ways. Here are some of the details, with log snippets (the only interesting activity is in /var/log/system.log):
The first time authentication stopped working was on June 6th at about 3:15am. I happened to be awake at the time working on something else -- checking for mail periodically but otherwise not logged onto mailsrvr. We use InterMapper to probe all of the server ports and make login checks, and I received a flurry of failures (on the ports listed above). The system had been running without incident for about 36 hours since the last reboot, with only a few mail users accessing the system plus InterMapper's probes every two minutes or so, and not much else going on. In the logs (below), it's clear that the original memberd daemon (process 55) exited abnormally. Launchd created a new instance of memberd (process 23431) which stayed up about eleven minutes before it also crashed. Launchd fired up one more instance of memberd (process 23901), which then stayed up for several days until we took it down for a backup. It was when this final instance of memberd came up that authentication started working again. I also verified that I could not log in via SSH while authentication services were down.
Here's a log snippet (I've omitted log entries which are clearly part of normal operations -- like Cyrus's checkpointing of the mail database every thirty minutes and InterMapper's probes):
Jun 6 03:13:44 mailsrvr memberd[55]: Fatal error -1 submitting to kernel (1: Operation not permitted)\n
Jun 6 03:13:44 mailsrvr memberd[23431]: memberd starting up
Jun 6 03:15:18 mailsrvr sshd[23399]: fatal: Timeout before authentication for <<my IP address>>
Jun 6 03:24:34 mailsrvr memberd[23431]: Fatal error -1 submitting to kernel (1: Operation not permitted)\n
Jun 6 03:24:34 mailsrvr launchd: Server 0 in bootstrap 1103 uid 0: "/usr/sbin/memberd -x"[23431]: exited with status: 1
Jun 6 03:24:34 mailsrvr memberd[23901]: memberd starting up
On June 12th, we suffered a similar failure. All of the authentication-based services listed above suddenly stopped responding. At the time, I had an open SSH connection running on root@mailsrvr, looking into unrelated issues, so while authentication was down I probed around a bit. I noticed right off that there was nothing funny in the logs as before -- no smoking gun indicating the memberd daemon. In fact, there was nothing in the logs to indicate that there was any problem at all!
However, I noticed from my bash session that if I did an "ls", the shell would list files without problem, but if I did an "ls -l", the command would hang until I issued a CTRL+C. I surmised that the reverse lookup of the user ID or group ID wasn't working correctly, either because memberd or lookupd or something deep in the system related to user or group names wasn't running right. Since this hangup occurred during business hours, it was necessary to restart the server to get it back online ASAP, so unfortunately I couldn't spend more time investigating the state of the system.
The next glitch occurred on June 14th. Once again, all of the authentication-based services when down. The only spurious entry in our logs was related to servermgrd (which provides service on the Tiger Server port 311). I don't think this daemon was the cause of the problem, but an indicator that something else was going on at the time. At the time of the failure (about 16:02), we were receiving FTP probes from mainland China (a host at 219.136.191.93) at intervals of 75 seconds or so to keep under the radar. I also don't think this was the cause of the problem. By 16:10 the problem had cleared itself up without my interaction or the need to restart the server.
Jun 14 16:01:39 mailsrvr ftpd[14832]: FTP LOGIN REFUSED (PASS before USER) FROM 219.136.191.93 [219.136.191.93]
Jun 14 16:05:11 mailsrvr servermgrd: [41] error in getAndLockContext: flock(servermgr_netboot) FATAL time out
Jun 14 16:05:11 mailsrvr servermgrd: [41] process will force-quit to avoid deadlock
Jun 14 16:05:11 mailsrvr launchd: com.apple.servermgrd: exited with exit code: 1
Jun 14 16:05:11 mailsrvr launchd: com.apple.servermgrd: 9 more failures without living at least 60 seconds will cause job removal
Jun 14 16:05:21 mailsrvr servermgrd: servermgr_dns: hostname and DNS entries for this server are synchronized
Jun 14 16:09:26 mailsrvr ftpd[15037]: FTP LOGIN REFUSED (PASS before USER) FROM 219.136.191.93 [219.136.191.93]
Jun 14 16:09:27 mailsrvr servermgrd: TIME-CHECK: NSLXStandardRegisterService took 255.824166 seconds!\n
The next day (June 15th), we saw a similar outage which lasted a bit less than fifteen minutes. Our friends in China had been probing our FTP port for several hours before this occurred:
Jun 15 12:48:28 mailsrvr ftpd[10559]: FTP LOGIN REFUSED (PASS before USER) FROM 219.136.191.93 [219.136.191.93]
Jun 15 12:49:30 mailsrvr servermgrd: [15310] error in getAndLockContext: flock(servermgr_netboot) FATAL time out
Jun 15 12:49:30 mailsrvr servermgrd: [15310] process will force-quit to avoid deadlock
Jun 15 12:49:30 mailsrvr launchd: com.apple.servermgrd: exited with exit code: 1
Jun 15 12:49:30 mailsrvr launchd: com.apple.servermgrd: 9 more failures without living at least 60 seconds will cause job removal
Jun 15 12:49:36 mailsrvr servermgrd: servermgr_dns: hostname and DNS entries for this server are synchronized
Jun 15 12:49:44 mailsrvr ftpd[10600]: FTP LOGIN REFUSED (PASS before USER) FROM 219.136.191.93 [219.136.191.93]
Jun 15 12:50:53 mailsrvr ftpd[10635]: FTP LOGIN REFUSED (PASS before USER) FROM 219.136.191.93 [219.136.191.93]
Jun 15 12:55:31 mailsrvr servermgrd: [11272] error in getAndLockContext: flock(servermgr_dhcp) FATAL time out
Jun 15 12:55:31 mailsrvr servermgrd: [11272] process will force-quit to avoid deadlock
Jun 15 12:55:44 mailsrvr launchd: com.apple.servermgrd: exited with exit code: 1
Jun 15 12:55:44 mailsrvr launchd: com.apple.servermgrd: 9 more failures without living at least 60 seconds will cause job removal
Jun 15 12:56:03 mailsrvr sshd[11398]: fatal: Timeout before authentication for <<my IP address>>
Since June 15th we've been running without incident, I'm happy to report. It's possible that the failures have been related to periods in which a number of changes have been made to the LDAP and NetInfo databases via Workgroup Manager. I'm going to keep a log from now on to see if there's a correlation there. Since June 15th, we've made only minimal changes.
</details>
1.42GHz Mac Mini   Mac OS X (10.4.6)   1GB RAM, SuperDrive, Airport

Answer 1

Ah, forgot to add that I'm running version 1.1, if that makes a difference. Sounds like it's been causing trouble...