AppleFileServer process keeps hanging - stalls clients' AFP connections

Question

Since moving to SL Server (first used on 10.6.4, now 10.6.6), Mac clients have been complaining of spinning beachballs when accessing the server. This appears to have gradually got worse over time, as more of the clients have been upgraded from Tiger & Leopard to SL (we're now running 6 Tiger, 2 Leopard and 8 SL clients). It's driving us nuts, as it happens frequently to all users (all designers), several times daily.
There seems to be no common trigger from the client side for the symptoms, i.e. not restricted to any one client, any one app being used, nor any particular action on their part. It can happen while opening, saving, copying or moving files from/to the server, but very intermittently, so hard to reproduce on demand.
Indeed the same hang happens on the server when I'm the only one connected to it at night, from just about any of the client Macs, whether they are SL, Leopard or Tiger. If I do a test file copy from/to the server, the AFS process will often stick at some point, though not always, even when using the same test files/folders each time. Have tried various different folder/file combinations during the tesing, with pretty much no difference in behaviour, nor any consistency.
Having searched the web, I can't turn up any hard explanation for what's happening, nor a fix. Others out there are having all sorts of hassles with AFP, SMB, etc, but all in different configs & flavours it seems.
Anyway, I did finally manage to find a correlation on the server... when any of the clients hangs during open, save or copy, the AppleFileServer process on the server shows as "stuck" in the output of the 'top' command (top -o state). There are a couple of other processes that stick sometimes, but they have no obvious hit on performance or stability (these are usually launchd, DirectoryServices & msd, and no correlation to the AFS stick as far as I've seen).
Whilst 'stuck' like this, the AFS process CPU load mostly shows as 0.0%, and eventually frees itself up, with AFP then working normally again for all users. So, there are no server or client crashes, but it can lock up for anything from a few seconds to several minutes at a time, and perhaps 30-50 times a day, during which time no-one can load, save or browse files on the server
Am now very close to doing a full re-install of SL, though I doubt it will help, as the problem's been there since I upgraded the server to a new XServe with SL 10.6.4. But first I was wondering if anyone here might have the same/similar problem, and throw any more light on it?
The Macs are used on a corporate WAN, with Active Directory logins (but the problem still happens when connecting with the server Admin a/c). Networked Home folders are not being used - all users have managed user Homes locally. Users are located in both London and Amsterdam, but the problem occurs for all.
Interestingly, am seeing similar client hangs when testing with SMB as an alternative to AFP (but didn't try that with the top command yet), so I'm suspecting this might be related to the networking layer of things, rather than AFP specifically? Not exactly a safe assumption I know, especially as thus far I can't even tell whether it's originating from the clients or the server side of things. I'd put my money on a bugged AppleFileServer app though. We didn't get this nonsense when everything was Tiger, on the same network as now.
Short of running with Tiger on old hardware (simply not an option for us), we're stuck with SL. Countless other (mostly) recurring errors showing up in the logs (as per usual with any Mac OS), but none with so vicious a payload as this. Now desperate for a solution, but little spare time to find one.
Any suggestions, or requests for more info, please let me know. Have already tried numerous avenues, including AFP access/error logs, tcpdump, Google, etc. etc., but no clear indicator I could find.
I'm no Mac geek btw, so please be gentle!

Answer 1

Hi,
You are not alone with this.
We have an OD master, separate file server (AFP with Raid attached with 2000+ users, 250 mac clients and on average 100 clients connected to network homes at a time.
All caches are redirected locally, time machine and spotlight searching disabled on the servers.
Running on 10.6.4, and working with no updates or changes nicely until...
Since December (after a clean install in July) we have really noticed the well documented (but not fixed) AFP crashes. It seems to occur for us when we have a few users using InDesign CS3 ( i have no proof of this but there is a pattern emerging). Up until december we had the most stable apple server setup i have known in 5 years, whereas afp used to just lock up completely, now it just restarts itself upon error.
Apple Premium support recommended updating to 10.6.6 before they would support us on this.
10.6.6 made it worse (slower logon times and even more crashes). Eventually they recommended to stop using Adobe programmes (in an art and design college).
We are currently testing with mobile home directories on this setup.
We have been using a similar setup (smaller though) on our music tech campus but with mobile homes, with great success for over a year now, although there are only about 50 concurrent logons on average there.
Well worth looking into as i have no faith in apple fixing this as most of the new server users are using very small setups with 10 clients max (home users with mac minis??).
I will keep you posted on how i get on because this really is a show stopper for us several times a week and it is really embarrassing for me to say to the customers i dont know what to do.
Regards,
Kev