Any changes in workgroup manager I make are not working!

Question

First off I know this isn't the place to vent, but I am so sick of all the bugs loaded in the directory system of mac. I think Apple has produced a horrible product and it is always one problem after another. I am experienced with all the other directory systems and I firmly believe Open Directory is the worst!
Ok, now on to my problem. Our set up includes 6 Mac OS X 10.6 servers. 1 is the Master OD and the others are all replicas. We recently upgrades the servers from 10.5 and ever since I have been having problems with OD. During the upgrades I took an archive of the system, decommissioned the Master and all replicas, then after the upgrade I restored the archive and replicated all the others. Things worked fine for a couple of days. Everyone can login and I could create new users and make changes in workgroup manager. After a couple of days I could no longer make any changes in workgroup manager. I can physically make the changes and create new users and save with no errors, but when I try to log into the file server with the new user or use the changed password, it doesn't work. No errors in the logs that I looked in.
Here is something weird: When I log into workgroup manager using the local administrator account, I do not see the changes I made in workgroup manager. When I click the unlock button and log in as the diradmin I can see the changes.
Any, ANY, help would be greatly appreciated. I've been working on this for weeks now. Please don't advise to decommission the Master and rebuild the directory. I already tried that about 10 times with the same result.
Thanks!

Answer 1

5 days should be plenty, unless you have a really slow link. Here are a few diagnostics that come to mind to figure out more about what's going on:
To test to figure out of I'm right about replication being the source of the problem, and if so which replicas aren't up to date, use the "ldapsearch" command to compare the data in the master vs. the various replicas. For instance, to check whether a new user named "fred" has shown up in the server at serverIP, run the command: ldapsearch –LLL –x –h serverIP –b cn=users,dc=masterserver,dc=example,dc=com “(uid=fred)”
(replace serverIP with your server's address, "dc=masterserver,dc=example,dc=com" with your search base, and fred with the actual account name). Try this on the master and the various replicas, and see which (if any) of the replicas are out of sync.
If some/all of the replicas are out of sync, check /var/log/slapd.log on the problematic replica(s). It's visible in Server Admin -> servername > Open Directory in sidebar -> Logs in toolbar -> LDAP Log in the View pop-up at the bottom of the window.
Also, check the replica status in Server Admin -> Open Directory -> Settings -> General.
Also, on the problematic replica(s), check the replication connections: "sudo lsof -i | grep ldap" should show an entry starting with "slapd" and ending "->masterserver.example.com:ldap (ESTABLISHED)". If it doesn't, you may have some sort of network connectivity/firewall issue.
If you want to get even more detailed, running "sudo tcpdump host masterserver.example.com and port ldap" on the replica will let you spy on the content of the replication connection.