Leopard Server 10.5.4 + SMB + Windows XP Clients

Question

Hello!
I got quite some interesting problems with my Xserv under Leopard Server 10.5.4 with Windows XP Clients.
The server runs as an open directory master and a standalone server for smb.
Shares are setup for AFP and SMB clients, ACLs are set up with read+write access for the user's group. So far everything works, users can connect from their Macs (OS X 10.3.x - 10.5.4) without problems.
My problem are the windows clients.
Connecting works fine, and apart from one permission bug (more on that later) they can access all files and do everything the macs can do.
Problem 1:
Users cannot rename files if they don't have write permission at the POSIX-level as either owner or group on the enclosing folder. ACLs do not matter at this point, if the user connects through SMB he cannot rename files or folders.
No big problem, I just put all users into one group and set this group as the POSIX-group and do a chmod -R g+w on all shares.
Inheritate this settings for SMB-connections works.
But not for AFP-connections. Every new folder a mac-client creates comes with permissions 755. So I changed UMasks on the clients to 002, and at least folders created by the finder now have the right permissions of 775.
Folders created by "new folder" on most applications load/save dialogs still come as 755 which screws up renaming for windows-clients.
Folders created when expanding a zip-file also behave this way.
The only solution I found for this was to make a cronjob that does a chmod -R g+w on the shares. Not that great.
Another "fun" feature with POSIX-permissions and SMB:
Files inside a folder. Files+Folder have an ACL set up that specifically denies deleting files+folders for a user. This users group has POSIX-permissions with write access to the enclosing folder. Window's explorer will let that user delete files and folders without any complaint about missing permissions to do so. They even disappear from the explorer-window, but are not actually deleted.
If you refresh the view, the files+folders are there again.
If the user instead has no POSIX-permission that grants him write-access, but an ACL that grants him delete, explorer happily deletes the files the user wishes to. And these files actually get deleted.
If the user has no write POSIX-permission and no ACL-permission to delete files, explorer complains about missing rights to delete files/folders.
I tried adding "acl check permissions = no" to smb.conf, but that didn't change this behaviour.
Problem 2:
Sometimes, when copying "large" files to a server share with windows explorer (from 50 MB upto 3 GB), the user immidiately gets an error message: "Cannot copy. The specified network name is no longer available."
The copied file shows up at once in the destination directory and has the correct file size, but only contains "garbage". Half a second later the same copy works and the file is actually copied successfully.
Sometimes this error shows up while a large file is copied and if you browse through other shared folders while the copy takes place.
Most often it will show up when you copy&paste with explorer.
I am stomped and have no idea where to search for a solution to this one.
I tried changing ethernet cables (premade and selfmade), the network switch, ethernet port of the Xserv ... connected a workstation directly to the Xserv - no change.
DNS works, all clients+Xserv have static IPs and the same settings for DNS-Server. Name resolution works, as does reverse resolution.
I even tried using a windows server 2003 as a WINS-Server without success for this problem.
There are no event log entries on the clients when this happens, and smbd.log on the server shows
read failure for 4 bytes to client 192.168.1.137. Error Connection reset by peer
I used wireshark to monitor what exactly happens when this shows up, but it just happens out of nowhere ... the clients just sends a TCP_RST. No unusual things before or after the TCP_RST.
I ran some tests with samba 3.0.32 and samba 3.2.x under linux 2.6.x - no such errors at all.
I tried installing Leopard Server on my Mac Pro, same behaviour.
I ran some batch-scripts that would copy large and small files in excess from/to the server from 4 different windows xp clients for a whole weekend 24 hours - not a single error. As soon as I use windows explorer to copy a file this error has a chance of showing itself.
Problem 3, more of an application bug:
Adobe InDesign CS2 (Windows again ;)) cannot export a PDF to a shared folder if the filename is longer than 8 characters. It starts exporting and after reaching 100% it just stops with the error "PDF cannot be exported." If the filename is shorter than 8 characters, all works fine. Exporting locally or to other samba servers (linux) or windows servers (2003) works fine. Copying this files to the share works, too.
Exporting any other filetype from IDCS2 (eps, inx, jpeg ...) works regardless of filename length. So does saving InDesign-files.
InDesign CS1 and CS3 do not show this, so I guess it's safe to assume this one is Adobe's problem
Did anyone experience similar problems or even better has a solution to some of these?
The real showstopper is problem #2, at least for us.
Thanks in advance and excuse my poor english skills!

Answer 1

I tried several new approaches, but nothing worked so far.
The update to 10.5.5 made things worse. Connections are dropping all over the place, when saving from Adobe programs in particular.
Printing from clients to a Xerox Phaser 5500 works, but choosing a different paper size than the default chosen in the cups admin interface goes all wrong and either prints on the default paper size, but down/upscaled to the chosen paper size or prints on the chosen paper size and down/upscaled as if you were printing to the default paper size. (e.g. default paper size is A4, you choose A3 with an A3 document. Document prints on A4 and is downscaled to A4, or prints on A3 and is downscaled to A4.)
There is no way I can keep this thing in production much longer. I tried reinstalling again, but that did not change any behaviour. Another nice "feature": InDesign disables "edit original" when the path to the linked file contains any directoryname with more than 8 characters in length.
Coupled with a sometimes disappearing "dirserv" (it just stops working until I reboot the whole computer) the decision stands to move away from the Xserve.
Leopard Server 10.5.x just does not seem to work well with windows clients.
We will migrate to a windows server 2003 running ExtremZ-IP for the macintosh clients this weekend. Too bad that Apple does not allow Bootcamp to work on Xserves, at least the hardware would have been useful then.
(As a nice finishing touch, the harddisk partition where all data resides did become corrupt over the weekend. "The volume Daten could not be repaired."
Disk Utility and first aid did show nothing on friday ... saturday after no work being done on the partition this error shows up. The only solution I could find was to reformat the partition and restore from a previous backup. Another weekend down the drain. Sorry for the rant, but 6 weekends in a row is enough.)