Potential VM growth Err,Cyrus, and Crashing Servers...

Question

On my Leopard Server (10.5.3), when an IMAP client makes lots of connections in a short period of time ( 35+ in one minute ), I will get the following errors in my system.log:
Potential VM growth in DirectoryService since client PID: 0, has 550 open references when the warning limit is 500.
If the DirectoryService and ServerManager processes are not restarted, this will spiral out of control, and Open Directory, then my server will crash. There is a little script that tails and grep's the system.log looking for these errors, and restarts both of these services (plugleak.pl):
See
http://davidleber.net/?p=216
It does work and keeps my server from crashing due to this Open Directory bug (which has been around since Tiger).
Why do clients try to make this many connections? One user had a PC with a weak intermittent wireless connection, and I guess his Outlook client kept making connections when the wireless connection fluctuated. This actually happened when the user was out of the office, but left his laptop running with Outlook open. For some reason, his laptop would switch over to the wireless connection while connected to our LAN, so we disabled his wireless (never takes the laptop home anyway), and this seemed to fix the problem. Unfortunately I did not put the plugleak.pl in a cronjob at this time, and my server eventually crashed, requiring a hard reboot (over a weekend).
Yesterday, I was migrating a user from our POP server, and uploading her local PC mail folders when this happened again, twice. She has hundereds of folders, and when Apple Mail creates a new folder and uploads the files, it creates a new IMAP connection for each folder. The "Potential VM" error messages started popping up and plugleak.pl did it' s job, saving the day.
Is there any way to configure Cyrus to limit the number of connections a client can make? Is this something that would be good to do? I noticed the following in the /etc/cyrus.conf:
LIMITS {
imaplimit value=0
I can't seem to find any documentation on what this value does. Does this limit the connections per user, or total connections? It seems that IMAP is very connection intensive.
Regards,
Flatrack
Oops, should be in the Leopard Server Mail forum.
Message was edited by: flatrack

Answer 1

Transcript of the server crash a few weeks ago with the recent.
Here's the first occurrence of the VM error message. John Doe is the user with Outlook, a funky wireless network connection, and Outlook checking mail every one min:
May 30 21:08:33 flatracks_server imap[12668]: login: flatracks_server [10.0.0.1] johndoe user logged in
May 30 21:08:33 flatracks_server imap[12485]: login: flatracks_server [10.0.0.1] johndoe user logged in
May 30 21:09:35 flatracks_server imap[12675]: login: flatracks_server [10.0.0.1] johndoe user logged in
May 30 21:09:35 flatracks_server imap[12662]: login: flatracks_server [10.0.0.1] johndoe user logged in
May 30 21:10:19 flatracks_server imap[12674]: login: bda144b.bis.na.blackberry.com [206.53.151.144] joesmith user logged in
May 30 21:10:37 flatracks_server imap[12713]: login: flatracks_server [10.0.0.1] johndoe user logged in
May 30 21:10:37 flatracks_server imap[12485]: login: flatracks_server [10.0.0.1] johndoe user logged in
May 30 21:10:39 flatracks_server imap[12662]: login: flatracks_server [10.0.0.1] ham user logged in
May 30 21:11:39 flatracks_server imap[12724]: login: flatracks_server [10.0.0.1] johndoe user logged in
May 30 21:11:39 flatracks_server imap[12674]: login: flatracks_server [10.0.0.1] johndoe user logged in
May 30 21:11:44 flatracks_server imap[12675]: login: flatracks_server [10.0.0.1] ham user logged in
May 30 21:12:41 flatracks_server imap[12735]: login: flatracks_server [10.0.0.1] johndoe user logged in
May 30 21:12:41 flatracks_server imap[12485]: login: flatracks_server [10.0.0.1] johndoe user logged in
May 30 21:12:50 flatracks_server imap[12760]: login: flatracks_server [10.0.0.1] ham user logged in
May 30 21:13:25 flatracks_server imap[12774]: login: bda144b.bis.na.blackberry.com [206.53.151.144] joesmith user logged in
May 30 21:13:43 flatracks_server imap[12736]: login: flatracks_server [10.0.0.1] johndoe user logged in
May 30 21:13:43 flatracks_server imap[12724]: login: flatracks_server [10.0.0.1] johndoe user logged in
May 30 21:14:45 flatracks_server imap[12780]: login: flatracks_server [10.0.0.1] johndoe user logged in
May 30 21:14:45 flatracks_server imap[12774]: login: flatracks_server [10.0.0.1] johndoe user logged in
May 30 21:14:59 flatracks_server DirectoryService[26]: Potential VM growth in DirectoryService since client PID: 0, has 525 open references when the warning limit is 500.
May 30 21:15:29: --- last message repeated 2 times ---
Around 1 hour later.....
May 30 22:10:42 flatracks_server imap[13506]: login: flatracks_server [10.0.0.1] johndoe user logged in
May 30 22:10:42 flatracks_server imap[13657]: login: flatracks_server [10.0.0.1] johndoe user logged in
May 30 22:11:44 flatracks_server imap[13694]: login: flatracks_server [10.0.0.1] johndoe user logged in
May 30 22:11:44 flatracks_server imap[13695]: login: flatracks_server [10.0.0.1] johndoe user logged in
May 30 22:11:44 flatracks_server imap[13529]: login: flatracks_server [10.0.0.1] ham user logged in
May 30 22:12:46 flatracks_server DirectoryService[26]: Potential VM growth in DirectoryService since client PID: 0, has 625 open references when the warning limit is 500.
May 30 22:12:46 flatracks_server imap[13703]: login: flatracks_server [10.0.0.1] johndoe user logged in
May 30 22:12:46 flatracks_server imap[13657]: login: flatracks_server [10.0.0.1] johndoe user logged in
May 30 22:12:50 flatracks_server imap[13694]: login: flatracks_server [10.0.0.1] ham user logged in
May 30 22:13:48 flatracks_server DirectoryService[26]: Potential VM growth in DirectoryService since client PID: 0, has 625 open references when the warning limit is 500.
Now things get interesting...
May 30 23:39:40: --- last message repeated 2 times ---
May 30 23:39:45 flatracks_server DirectoryService[26]: Potential VM growth in DirectoryService since client PID: 0, has 775 open references when the warning limit is 500.
May 30 23:39:45 flatracks_server imap[15168]: login: flatracks_server [10.0.0.1] johndoe user logged in
May 30 23:39:45 flatracks_server imap[15130]: login: flatracks_server [10.0.0.1] johndoe user logged in
May 30 23:40:39 flatracks_server com.apple.launchd[15209] (com.apple.launchctl.Background): fork() failed, will try again in one second: Resource temporarily unavailable
May 30 23:40:47 flatracks_server DirectoryService[26]: Potential VM growth in DirectoryService since client PID: 0, has 775 open references when the warning limit is 500.
May 30 23:40:47 flatracks_server imap[15146]: login: flatracks_server [10.0.0.1] johndoe user logged in
May 30 23:40:47 flatracks_server DirectoryService[26]: Potential VM growth in DirectoryService since client PID: 0, has 775 open references when the warning limit is 500.
May 30 23:40:47 flatracks_server imap[15139]: login: flatracks_server [10.0.0.1] johndoe user logged in
May 30 23:41:09 flatracks_server com.apple.launchd[15220] (com.apple.launchctl.Background): fork() failed, will try again in one second: Resource temporarily unavailable
May 30 23:41:39 flatracks_server com.apple.launchd[15231] (com.apple.launchctl.Background): fork() failed, will try again in one second: Resource temporarily unavailable
May 30 23:42:09 flatracks_server com.apple.launchd[15239] (com.apple.launchctl.Background): fork() failed, will try again in one second: Resource temporarily unavailable
May 30 23:42:39 flatracks_server com.apple.launchd[15248] (com.apple.launchctl.Background): fork() failed, will try again in one second: Resource temporarily unavailable
May 30 23:42:50 flatracks_server imap[15146]: login: flatracks_server [10.0.0.1] johndoe user logged in
May 30 23:43:09 flatracks_server com.apple.launchd[15255] (com.apple.launchctl.Background): fork() failed, will try again in one second: Resource temporarily unavailable
May 30 23:43:39 flatracks_server com.apple.launchd[15265] (com.apple.launchctl.Background): fork() failed, will try again in one second: Resource temporarily unavailable
May 30 23:43:51 flatracks_server imap[7730]: login: bda365c.bis.na.blackberry.com [67.223.68.76] janesmith user logged in
May 30 23:44:09 flatracks_server com.apple.launchd[15273] (com.apple.launchctl.Background): fork() failed, will try again in one second: Resource temporarily unavailable
May 30 23:44:39 flatracks_server com.apple.launchd[15279] (com.apple.launchctl.Background): fork() failed, will try again in one second: Resource temporarily unavailable
This goes on all night.
May 31 00:03:12 flatracks_server com.apple.launchd[15601] (com.apple.launchctl.Background): fork() failed, will try again in one second: Resource temporarily unavailable
May 31 00:03:42 flatracks_server com.apple.launchd[15606] (com.apple.launchctl.Background): fork() failed, will try again in one second: Resource temporarily unavailable
May 31 00:04:09 flatracks_server postfix/pipe[15347]: warning: pipecommandread: read time limit exceeded
May 31 00:04:12 flatracks_server com.apple.launchd[15620] (com.apple.launchctl.Background): fork() failed, will try again in one second: Resource temporarily unavailable
May 31 00:04:42 flatracks_server com.apple.launchd[15625] (com.apple.launchctl.Background): fork() failed, will try again in one second: Resource temporarily unavailable
My boss calls me Saturday, can't log into mail. I SSH in, stop and restart the Mail service.
May 31 08:53:31 flatracks_server com.apple.launchd[24084] (com.apple.launchctl.Background): fork() failed, will try again in one second: Resource temporarily unavailable
May 31 08:53:46 flatracks_server com.apple.launchd[1] (org.postfix.master[1113]): Exited: Terminated
May 31 08:53:46 flatracks_server postfix/postfix-script[24091]: fatal: the Postfix mail system is not running
May 31 08:53:49 flatracks_server org.clamav.clamd[1058]: Socket file removed.
May 31 08:53:49 flatracks_server org.clamav.clamd[1058]: Pid file removed.
May 31 08:53:49 flatracks_server org.clamav.clamd[1058]: --- Stopped at Sat May 31 08:53:49 2008
May 31 08:53:49 flatracks_server com.apple.launchd[1] (org.amavis.amavisd[1060]): Stray process with PGID equal to this dead job: PID 23758 PPID 1 perl
May 31 08:53:49 flatracks_server com.apple.launchd[1] (edu.cmu.andrew.cyrus.master[1054]): Stray process with PGID equal to this dead job: PID 23955 PPID 1 master
May 31 08:53:49 flatracks_server com.apple.launchd[1] (edu.cmu.andrew.cyrus.master[1054]): Stray process with PGID equal to this dead job: PID 23415 PPID 1 master
May 31 08:53:49 flatracks_server com.apple.launchd[1] (edu.cmu.andrew.cyrus.master[1054]): Stray process with PGID equal to this dead job: PID 22890 PPID 1 master
May 31 08:53:49 flatracks_server com.apple.launchd[1] (edu.cmu.andrew.cyrus.master[1054]): Stray process with PGID equal to this dead job: PID 22617 PPID 1 master
That seems to work, we can log into mail again, but I don't notice these in the logs...
May 31 08:53:49 flatracks_server com.apple.launchd[1] (edu.cmu.andrew.cyrus.master[1054]): Stray process with PGID equal to this dead job: PID 15204 PPID 1 imapd
May 31 08:53:49 flatracks_server com.apple.launchd[1] (edu.cmu.andrew.cyrus.master[1054]): Stray process with PGID equal to this dead job: PID 15160 PPID 1 lmtpd
May 31 08:54:01 flatracks_server com.apple.launchd[24142]: Background: Bug: launchdcorelogic.c:1882 (23459):3: (ji = jobmgrfind_bypid(jm, upid, true)) != NULL
May 31 08:54:01 flatracks_server com.apple.launchd[24142]: Background: Bug: launchdcorelogic.c:1913 (23459):2: jr != NULL
May 31 08:54:01 flatracks_server com.apple.launchd[24142]: Background: Bug: launchdcorelogic.c:1920 (23459):2: len == sizeof(kp)
May 31 08:54:01 flatracks_server com.apple.launchd[24142]: Bug: launchdcorelogic.c:5777 (23459):2: j != NULL
May 31 08:54:01 flatracks_server com.apple.launchd[24142]: Background: Bug: launchdcorelogic.c:1882 (23459):2: (ji = jobmgrfind_bypid(jm, upid, true)) != NULL
May 31 08:54:01 flatracks_server com.apple.launchd[24142]: Background: Bug: launchdcorelogic.c:1913 (23459):2: jr != NULL
May 31 08:54:01 flatracks_server com.apple.launchd[24142]: Background: Bug: launchdcorelogic.c:1920 (23459):2: len == sizeof(kp)
May 31 08:54:01 flatracks_server com.apple.launchd[24142]: Bug: launchdcorelogic.c:5777 (23459):2: j != NULL
May 31 08:54:01 flatracks_server deliver[24005]: connect(/Volumes/MailStore/Mail/Database/socket/lmtp) failed: Connection refused
May 31 08:54:01 flatracks_server deliver[23999]: connect(/Volumes/MailStore/Mail/Database/socket/lmtp) failed: Connection refused
May 31 08:54:02 flatracks_server master[24146]: process started
Not sure what the Background Bug thing is, but it can't be good. Sunday morning I get a text message telling me that the IMAP server is not responsive (something I set up on a Linux box to let me know when this sort of thing occurs. Good thing I did, and it works great). So I SSH in, and 'shutdown now'.
Jun 1 11:58:55 flatracks_server shutdown[41200]: SHUTDOWN_TIME: 1212346735 947810
Jun 1 11:58:56 flatracks_server com.apple.SystemStarter[99]: Stopping HighPoint RAID Management Server
Jun 1 11:58:56 flatracks_server com.apple.SystemStarter[99]: Stopping IP Failover services
Jun 1 11:58:56 flatracks_server com.apple.SystemStarter[99]: Starting Retrospect Client
Jun 1 11:58:56 flatracks_server com.apple.SystemStarter[99]: Disabling Network Address Translation
Jun 1 11:58:56 flatracks_server com.apple.SystemStarter[99]: cat: /var/run/failoverd.pid: No such file or directory
Jun 1 11:58:56 flatracks_server com.apple.SystemStarter[99]: kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
Jun 1 11:58:56 flatracks_server SystemStarter[99]: IP Failover (41205) did not complete successfully
Jun 1 11:58:56 flatracks_server com.apple.SystemStarter[99]: cat: /var/run/heartbeatd.pid: No such file or directory
Jun 1 11:58:56 flatracks_server com.apple.SystemStarter[99]: kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
Jun 1 11:58:56 flatracks_server com.apple.SystemStarter[99]: net.inet.ip.forwarding: 0 -> 0
Jun 1 11:58:56 flatracks_server com.apple.SystemStarter[99]: ipfw: rule 10 does not exist
Jun 1 11:58:56 flatracks_server com.apple.SystemStarter[99]: /Applications/Retrospect Client.app/Contents/Resources/pitond: already running
Jun 1 11:59:02 flatracks_server SystemStarter[99]: The following StartupItems failed to properly start:
Jun 1 11:59:02 flatracks_server SystemStarter[99]: /System/Library/StartupItems/IPFailover
Jun 1 11:59:02 flatracks_server SystemStarter[99]: - execution of Startup script failed
The shutdown hangs at this point, and I have to drive 110 miles to push the button.
Jun 1 15:40:04 localhost com.apple.launchctl.System[2]: 12300 allow tcp from any to any established
Jun 1 15:40:04 localhost com.apple.launchctl.System[2]: 12301 allow tcp from any to any out
System restarts, but my Open Directory DB is weird. Two users are missing, and I cant see the mail accounts in Server Manager. At this point, I boot from a backup disk and rebuild my system disk from the backup with Carbon Copy Cloner.
Two weeks later, I'm in the process of uploading a migrated users email in AppleMail. She has hundreds of folders, and the IMAP client (AppleMail) seems to make a connection for each folder it uploads. It looks like it's making 1-2 connections per second, this is only a small exerpt.
Jun 18 09:33:07 flatracks_server imap[16439]: login: flatracks_server [10.0.0.1] janedoe user logged in
Jun 18 09:33:08 flatracks_server imap[16440]: login: flatracks_server [10.0.0.1] janedoe user logged in
Jun 18 09:33:08 flatracks_server imap[16443]: login: flatracks_server [10.0.0.1] janedoe user logged in
Jun 18 09:33:09 flatracks_server imap[16444]: login: flatracks_server [10.0.0.1] janedoe user logged in
Jun 18 09:33:09 flatracks_server imap[16441]: login: flatracks_server [10.0.0.1] janedoe user logged in
Jun 18 09:33:11 flatracks_server imap[16445]: login: flatracks_server [10.0.0.1] janedoe user logged in
Jun 18 09:33:12 flatracks_server imap[16442]: login: flatracks_server [10.0.0.1] janedoe user logged in
Jun 18 09:33:12 flatracks_server imap[16446]: login: flatracks_server [10.0.0.1] janedoe user logged in
Jun 18 09:33:13 flatracks_server imap[16450]: login: flatracks_server [10.0.0.1] janedoe user logged in
Jun 18 09:33:13 flatracks_server imap[16451]: login: flatracks_server [10.0.0.1] janedoe user logged in
Jun 18 09:33:13 flatracks_server imap[16457]: login: flatracks_server [10.0.0.1] johndoe user logged in
Jun 18 09:33:13 flatracks_server imap[16459]: login: flatracks_server [10.0.0.1] janedoe user logged in
Jun 18 09:33:13 flatracks_server imap[16458]: login: flatracks_server [10.0.0.1] johndoe user logged in
Jun 18 09:33:13 flatracks_server imap[16462]: login: flatracks_server [10.0.0.1] janedoe user logged in
Jun 18 09:33:14 flatracks_server imap[16463]: login: flatracks_server [10.0.0.1] janedoe user logged in
Jun 18 09:33:14 flatracks_server imap[16464]: login: flatracks_server [10.0.0.1] janedoe user logged in
Jun 18 09:33:15 flatracks_server imap[16461]: login: flatracks_server [10.0.0.1] janedoe user logged in
Jun 18 09:33:16 flatracks_server imap[16465]: login: flatracks_server [10.0.0.1] janedoe user logged in
Jun 18 09:33:16 flatracks_server imap[16466]: login: flatracks_server [10.0.0.1] janedoe user logged in
Jun 18 09:33:16 flatracks_server imap[16467]: login: flatracks_server [10.0.0.1] janedoe user logged in
Jun 18 09:33:17 flatracks_server imap[16468]: login: flatracks_server [10.0.0.1] janedoe user logged in
Jun 18 09:33:17 flatracks_server DirectoryService[15022]: Potential VM growth in DirectoryService since client PID: 0, has 525 open references when the warning limit is 500.
Jun 18 09:33:17 flatracks_server imap[16469]: login: flatracks_server [10.0.0.1] janedoe user logged in
Jun 18 09:33:17 flatracks_server DirectoryService[15022]: Potential VM growth in DirectoryService since client PID: 0, has 525 open references when the warning limit is 500.
Jun 18 09:33:17 flatracks_server imap[16470]: login: flatracks_server [10.0.0.1] janedoe user logged in
Jun 18 09:33:19 flatracks_server imap[16471]: login: flatracks_server [10.0.0.1] janedoe user logged in
After many more of these, plugleak.pl does it's thing (it runs out of cron every 10 min).
Jun 18 08:40:00 flatrack PIMEDIA_PLUGLEAK[15017]: Checking logs
Jun 18 08:40:00 flatrack PIMEDIA_PLUGLEAK[15018]: Found that **** Pot. VM notice in the log, killing services
Jun 18 08:40:00 flatrack com.apple.launchd[1] (com.apple.servermgrd[80907]): Exited abnormally: Hangup
Jun 18 08:40:00 flatrack com.apple.launchd[1] (com.apple.DirectoryServices[80908]): Exited: Killed
Jun 18 08:40:00 flatrack DirectoryService[15022]: Launched version 5.3 (v514.19)
There are a few user login errors, but things sort themselves out in a minute or so when OpenDirectory is loaded. During the next large upload, the same thing happens, tons of connections, VM errors, then:
Jun 18 09:50:00 flatrack PIMEDIA_PLUGLEAK[17010]: Checking logs
Jun 18 09:50:00 flatrack PIMEDIA_PLUGLEAK[17011]: Found that **** Pot. VM notice in the log, killing services
Jun 18 09:50:00 flatrack com.apple.launchd[1] (com.apple.servermgrd[15021]): Exited abnormally: Hangup
Jun 18 09:50:00 flatrack com.apple.launchd[1] (com.apple.DirectoryServices[15022]): Exited: Killed
Jun 18 09:50:00 flatrack DirectoryService[17015]: Launched version 5.3 (v514.19)
I finished the job doing smaller batches. the VM error messages have not showed up since, and things seem to be running fine.