10.6.2 grinds to a halt after a few hours

Question

I've seen a few recent similar threads, but none were exactly the same so I figured I'd start my own. We have recently setup a MacMini Server for local IMAP mail, afp, ftp, and web services, those are the only services it is running. DNS and DHCP are provided via a third party server we do not control, but all lookups appear to be providing the expected information.
The problem has happened to us twice now, both times from a clean install. We would set it up with 3 admin users and a few test accounts and all services worked as expected for days.
The first failed iteration we then used Passenger to add 269 more accounts - those accounts worked as expected for a few days but then the server ground to a halt. We could still ping it, but it stopped responding to IMAP or web requests, and apps running on the server (such as Console, Server Admin, even Activity Monitor) would sporadically show up as "Not Responding" until eventually it was entirely unresponsive (other than you could still successfully ping it) and we had to force a power switch shutdown. After that it repeatably showed the same behavior every 2-4 hours, services work fine at the beginning and then get slower until they become unresponsive.
The second failed iteration we guessed that perhaps the Passenger user import introduced a corruption so we had been manually entering users one at a time. Yesterday we got up to 174 users and it began exhibiting the same behavior. Every 2-4 hours it just dies.
Activity Monitor shows no excessive process behavior, CPU usage, memory usage, or network usage.
Exhaustive Console searching is revealing no seemingly pertinent information other than one message in the system.log which I think is a symptom of the problem rather than the cause:
"dovecot\[59\]: imap-login: Timeout waiting for handshake from auth server."
This message was logged hundreds of times before we eventually force the machine to shut down.
I'm out of ideas at this point...

Answer 1

Jeremy made me aware of this thread. I think I have the same problem (see this for more info: http://discussions.apple.com/thread.jspa?threadID=2299067&tstart=0 ) and am stuck too.
I made a small script which basically watches the thread count of the DirectoryService every 5 seconds and restarts DirectoryService if it reaches a certain treshold. But even with a very low treshold level (I went as low as 18 threads in the end), it wasn't able to 'catch' DirectoryService when it went bezerk.
After that I did enable OD debugging and found a continious stream of -14292 errors (eNotHandledByThisNode) on the fileserver. This seems odd because all the regular logs show nothing special. I combed through Open Directory again and found that the machine records for the 3 servers (1 ODM, 2 ODRs) aren't correct. The hardware addresses of the three servers are mixed up. I can trace this back to when I installed the servers. I had two servers where I could work on at that time and imaged the final setups for three servers. So next thing I'll try is to correct these records and see if it changes things.