Tiger 10.4.4 Slow Afp access when AppleFileServer app takes up 100% CPU

Question

I'm hoping maybe someone here will have a clue on this because so far everyone else I've talked to doesnt!
G4 Xserve Specs:
G4 1Ghz Proc
1.5 Gig Ram
x2 60Gig drives (Software Stripe)
LAN: 1Gig Asante 1000SX Fiber Card (w/ tiger drivers from Manufacturer)
Previously running 10.3.9 then upgraded to 10.4.4
Supports: Mac Manger (90 clients doing MMLocalPrefs)
Reading Counts (about 60 concurrent clients)
General Shares (130-160 clients) basically just mounts were people backup their local documents.
Any given time about 250 users connected
Users all in NetInfo
Problem:
Over Christmas break we upgraded our existing Xserve G4 Slot loader server from 10.3.9 to 10.4.3. When we returned from the break and all the teachers were back on we noticed that AFP file transfers were very slow. Mac Manger login, Reading Counts, Grade Quick, and even general shares were not responding as quickly as they did prior to the upgrade. The day after everyone returned the 10.4.4 update became available. We upgraded and still had the same problem. I called Apple support who ran me through plugging a machine directly into the server and more or less that was very fast. They blamed our switch gear and that was more or less the end of the conversation.
Through the next 3 weeks I probed our switch guy for a resolution. He made modifications which seemed to get us a little fast, but still not back to where we were. So at the end of the 3rd week we paid a consulting company to come in and turn the new G5 server we recently purchased into the new main server. 10.3.9 Was installed then upgraded to 10.4.4. After all patches were applied we migrated the Netinfo users into Open Directory, moved over their home directories, Reading Counts databases, and Grade Quick databases. Also all Mac Manger stuff was recreated from scratch. Here is what I ended up with:
G5 Xserve Specs:
G5 2Ghz Proc
3 Gig Ram
x1 80 Gig drive (for System)
x2 400 Gig drives (Software Stripe for storage)
LAN: 1 Gig onboard NIC Copper
10.3.9 Installed for MM usage then upped to 10.4.4
Supports: Mac Manger (90 clients doing MMLocal Prefs)
Reading Counts (about 60 concurrent clients)
General Shares (130-160 clients)
Users all in Open Directory
The next morning when all the users came in things were better. Processor was no longer going to 100% early in the morning and staying there all day like it previously had. However it would do so on occasion still. Often staying at 100% for 30 mins and then dropping for no reason. When we continued to have the problem I decided to drop some clients. We previously had all student machines in one of our Elementary buildings doing MM logins as well. It was felt that these student (1 acct for each grade level) accounts could have corrupt files (which Disk Utility doesn’t pick up) because we were allowing them Multiple Logins with the same account. MM was disabled on all these machines and so were the accounts. This dropped us down to approx. 50-60 MM clients and again we seen an improvement. CPU was still sticking but not nearly as long.
So that more or less brings us up to current. Here are things I’ve tried which don’t really fit too well into the above timeline, but were all done after the new server came online:
- Went to all MM machines and deleted all the prefs from them. Also deleted the Classic folder out of all users’ profiles forcing new prefs to be created.
- We had Mac Manger set to copy all prefs to the machines. Changed it to just internet and Admin prefs.
- Have tested with different buildings disabled to see if one building was more so an issue then another.
As it stands now when my CPU hits 100% the AppleFileServer app according to Top and activity monitor is using all of it. The hard drives are not blinking and the network graph is only showing approx. 400Kbs of traffic average. All I’m hearing from Apple is that I’m serving too many people which if my HDs were blinking like crazy and the server was putting out some decent network through put I would understand. Others have told me its all Mac Manger’s fault. Again though prior to the upgrade I never had any problem with this stuff! Even when I had high CPU usage in 10.3.9 it was barely noticeable by my network users both MM and regular shares.
Any one have any ideas?
-Jesse C. Smillie
Gateway School District
Monroeville, PA
Xserve G4 and Xserve G5   Mac OS X (10.4.4)   Lots of Imacs, G4s, Ibooks, and now some Mac Mini's and G5s

Answer 1

Hi folks.
I've upgraded my servers to OS X 10.4.5 server...
We're still experiencing slowness with clients (about 35), with network home folders. Especially noticable when more users are on-board.
But even with just a few users logged in. And logging in seems to take longer (30 secs, instead of say, 10 secs).
And file transfers are painfully slow.
Logging in as a "local" admin user on the machine works great -- file transfers are faster too.
Previously, we had a whole boat-load of other services on this particular server box. I've been progressively moving them off to other/new boxes... Aiming for the "one server - one service" principle.
We also have a file-sharing volume, and Mail on the same box as the LDAP authentication/home folders, so I'm migrating these services to new boxes too (easier said than done in a production environment!).
I usually run this command in a terminal window on a given machine or server, to see what's running at the time...
run top -us 10 5
Check how many page in-outs you have as well (listed in the info at the top of the 'top' command when it's being run).
Other changes...
I've recently put in 10/100/1000 HP switches to see if that helped (was 10/100 hubs and switches); that helped a lot, but the problems are still bad. I also upgraded my OS X Server boxes to 5GB of RAM.
We're not a design studio... We're just an office with a "standard" load of documents being opened, editted and saved. Occasional PDFs, that type of thing. But mostly Word docs, web browsing and Mail.
Hopefully migrating the large shared folder and Mail services will help too.
I suspect any directories with a large number of files will be contributing (500+ files in a directory). But I don't have many/any like this.
Add to that, using apps like Office/Word or Quark across a network, and that will definitely contribute.... Hence my reasoning to separate the stand-alone file-share to a different server box.
I've also got client machines to only check mail every 15 minutes (especially until I get migrate them to another box).
I'm also getting my network cabling checked out, just in case ... it is a few years old after all !!! I've replaced suspect patch leads with new CAT5 or CAT6 leads, and destroyed the old ones.
Also, try checking through all your Log files on both the server, and a 'typical' client machine or two. I've noticed some 'interesting' entries in the Directory Services client log file, for example, talking about "Network transition occurred" and so on.
My HP managed switches also report some CRC errors, and the occasional duplex-mismatch error... and a few traffic "errors", which I didn't know about until the managed switches were installed. (Might be solved by setting the speed/duplex of the port / Mac manually, or could be cabling issue).
My client machines are all iMac G4's and iMac G5's. Oh, and occasional recent Powerbook G4's, with gigabit networking.
Let us know if anyone has a break-through!
Cheers,
Derek!
  Mac OS X (10.4.5)