Authentication Delays / Slow Authentication for Open Directory Users

Question

I'm experiencing delays when authenticating Open Directory users and it absolutely has me at my wit's end.
The problem is quite simple: any time an Open Directory user authenticates his password there is a delay of at least 5-10 seconds. This goes for clients that are bound to the directory server and also authenticating locally on the server. Here are some examples:
* On the server, there is a several second delay on the Login Window screen when trying to log in using an Open Directory account. Logging in as a local user is instantaneous.
* In Workgroup manager, authenticating as the Directory Administrator takes several seconds.
* On a remote computer, sharing the screen using an Open Directory user take several seconds and again, a local user is instantaneous. Screen sharing takes particularly long and often temporarily shows a sheet saying it has lost the connection with the server while authenticating.
* Connecting with AFP takes several seconds when using an Open Directory login
* On a client computer, unlocking the screen after sleep or screen saver takes several seconds for Open Directory users
* Connecting with SSH does NOT exhibit the behavior
In addition to all of this, I've seen periodic random unexplainable freezes for several seconds on client computers that are bound to the directory even when logged in as a local user account (and with no other users logged in.) For example, launching applications often results in a freeze. After unbinding the computer from the directory the problem goes away entirely.
The history of the problem:
Used Tiger Server for over a year = no problems
Clean install of Leopard Server 10.5.0 back in October = no problems
Update to Leopard Server 10.5.1 = no problems
Then, all of the sudden one day several weeks back I started having problems. The server had been up for a few weeks. I didn't install any updates. I didn't change any configuration. Literally the only thing that I had done recently was unplug the Apple Cinema Display and keyboard+mouse that was connected to the server. Then I started having problems so I plugged the display, keyboard and mouse back in to troubleshoot it. I cleared the directory services caches on my server and clients and rebooted the Airport Base Station that's serving as my router and eventually the problem went away. I wish I could tell you which of those things resolved the problem but I have no idea. It was fine for a couple more weeks (and incidentally I once again unplugged the display, keyboard and mouse from the server). Then last week I started having problems again and this time no amount of rebooting, cache clearing, rebinding, troubleshooting using information in these forums or anything else will fix the problem. I only mention the display/keyboard/mouse thing because it's literally the only thing I changed around the time the problems started happening. I truly don't think it has anything to do with it.
So in desperation I backed up and did a clean install today. Here's the process I used:
0. Erase the disk
1. Install Leopard Server 10.5.0 from the install DVD
2. In the setup assistant, use the Advanced Configuration option but I didn't enable any services. Set up network settings and host name of myserver.mydomain.private.
3. Reboot
4. Use Software Update to update to 10.5.1 and Security Update 2007-009 v1.1
5. Reboot
6. Configure DNS (see below for detailed configuration)
7. Reboot
8. Change role to Open Directory Master
9. Reboot
... and the problem is still there. Simply logging into the server GUI with the Directory Administrator account has the delay. Authenticating in Workgroup Manager has the delay. I haven't even bothered to set up AFP or any other users yet. I'm truly at my wit's end and I'm ready to chuck the server out the window.
I've done a lot of googling and searching of these forums looking for answers. All of the responses seem to point to a problem with DNS or with the Kerberos realm. I believe all of my setup is correct. Here it is:
== Basic Configuration ==
OS: Mac OS X Server 10.5.1 (9B18) with Security Update 2007-009 v.1.1
Services Enabled:
DNS
Open Directory
(All other services are not yet enabled)
== DNS Setup ==
Primary Zone: mydomain.private.
Allows zone transfer: no
Nameservers: ns.mydomain.private.
myserver (Machine) 10.0.22.201
ns (Alias) myserver.mydomain.private.
Reverse Zone: 22.0.10.in-addr.arpa.
10.0.22.201 (Reverse Mapping) myserver.mydomain.private.
Accept recursive queries from the following networks:
localnets
Forwarder IP Addresses:
208.67.222.222
208.67.220.220
== Open Directory Setup ==
Role: Open Directory Master
LDAP Search Base: dc=myserver,dc=mydomain,dc=private
Kerberos Realm: myserver.mydomain.private
== Network Configuration ==
Configure: Manually
IP Address: 10.0.22.201
Subnet Mask: 255.255.255.0
Router: 10.0.22.1
DNS Server: 127.0.0.1
Search Domains: mydomain.private
== Other Stuff ==
Using 'changeip -checkhostname' verifies that the hostname and DNS hostname are both myserver.mydomain.private.
I set the realm to myserver.mydomain.private (though the default was myserver.local) based on the advice of another poster to this forum. Kerberos.app reveals something interesting: the kdc and admin servers are both myserver.local and the domains are .local and local. I tried changing all instances of 'local' to 'mydomain.private' to see if that would solve the problem. No luck.
I verified on a client that 'host myserver' and 'host 10.0.22.201' return proper DNS and reverse DNS resolutions.
Hopefully one of the gurus out there will be able to help me out.
Thanks,
jeff

Answer 1

I gathered together some log information for when I try to authenticate user 'diradmin' in Workgroup Manager. You can see from the log messages that this authentication took 4 seconds. There's an interesting error message in slapd.log (see below) but it doesn't say what it's looking for in the keytab that it's not finding. Grr! I've provided a listing of the principles in my keytab. I haven't monkeyed around with it at all -- this is just what resulted from promoting the server to an Open Directory Master.
== kdc.log ==
Dec 30 18:21:48 myserver.mydomain.private krb5kdc[79](debug): handling authdata
Dec 30 18:21:48 myserver.mydomain.private krb5kdc[79](debug): handling authdata
Dec 30 18:21:48 myserver.mydomain.private krb5kdc[79](debug): .. .. ok
Dec 30 18:21:48 myserver.mydomain.private krb5kdc[79](debug): .. .. ok
Dec 30 18:21:48 myserver.mydomain.private krb5kdc[79](info): AS_REQ (7 etypes {18 17 16 23 1 3 2}) fe80::216:cbff:fea5:f3ce: ISSUE: authtime 1199060508, etypes {rep=16 tkt=16 ses=16}, [email protected] for krbtgt/[email protected]
Dec 30 18:21:48 myserver.mydomain.private krb5kdc[79](info): AS_REQ (7 etypes {18 17 16 23 1 3 2}) fe80::216:cbff:fea5:f3ce: ISSUE: authtime 1199060508, etypes {rep=16 tkt=16 ses=16}, [email protected] for krbtgt/[email protected]
Dec 30 18:21:52 myserver.mydomain.private krb5kdc[79](info): TGS_REQ (7 etypes {18 17 16 23 1 3 2}) fe80::216:cbff:fea5:f3ce: ISSUE: authtime 1199060508, etypes {rep=16 tkt=16 ses=16}, [email protected] for ldap/[email protected]
Dec 30 18:21:52 myserver.mydomain.private krb5kdc[79](info): TGS_REQ (7 etypes {18 17 16 23 1 3 2}) fe80::216:cbff:fea5:f3ce: ISSUE: authtime 1199060508, etypes {rep=16 tkt=16 ses=16}, [email protected] for ldap/[email protected]
== slapd.log ==
Dec 30 18:21:48 myserver slapd[36]: <= bdbsubstringcandidates: (authAuthority) index_param failed (18)
Dec 30 18:21:52 myserver slapd[36]: SASL [conn=20] Failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No principal in keytab matches desired name)
== sudo klist -k ==
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
3 afpserver/LKDC:SHA1.D711BEA4D0DDB570D64ED88C5D06A78A34B7167C@LKDC:SHA1.D711BEA4 D0DDB570D64ED88C5D06A78A34B7167C
3 afpserver/LKDC:SHA1.D711BEA4D0DDB570D64ED88C5D06A78A34B7167C@LKDC:SHA1.D711BEA4 D0DDB570D64ED88C5D06A78A34B7167C
3 afpserver/LKDC:SHA1.D711BEA4D0DDB570D64ED88C5D06A78A34B7167C@LKDC:SHA1.D711BEA4 D0DDB570D64ED88C5D06A78A34B7167C
3 cifs/LKDC:SHA1.D711BEA4D0DDB570D64ED88C5D06A78A34B7167C@LKDC:SHA1.D711BEA4D0DDB 570D64ED88C5D06A78A34B7167C
3 cifs/LKDC:SHA1.D711BEA4D0DDB570D64ED88C5D06A78A34B7167C@LKDC:SHA1.D711BEA4D0DDB 570D64ED88C5D06A78A34B7167C
3 cifs/LKDC:SHA1.D711BEA4D0DDB570D64ED88C5D06A78A34B7167C@LKDC:SHA1.D711BEA4D0DDB 570D64ED88C5D06A78A34B7167C
3 vnc/LKDC:SHA1.D711BEA4D0DDB570D64ED88C5D06A78A34B7167C@LKDC:SHA1.D711BEA4D0DDB5 70D64ED88C5D06A78A34B7167C
3 vnc/LKDC:SHA1.D711BEA4D0DDB570D64ED88C5D06A78A34B7167C@LKDC:SHA1.D711BEA4D0DDB5 70D64ED88C5D06A78A34B7167C
3 vnc/LKDC:SHA1.D711BEA4D0DDB570D64ED88C5D06A78A34B7167C@LKDC:SHA1.D711BEA4D0DDB5 70D64ED88C5D06A78A34B7167C
3 cifs/[email protected]
3 cifs/[email protected]
3 cifs/[email protected]
3 ldap/[email protected]
3 ldap/[email protected]
3 ldap/[email protected]
3 xgrid/[email protected]
3 xgrid/[email protected]
3 xgrid/[email protected]
3 vpn/[email protected]
3 vpn/[email protected]
3 vpn/[email protected]
3 ipp/[email protected]
3 ipp/[email protected]
3 ipp/[email protected]
3 xmpp/[email protected]
3 xmpp/[email protected]
3 xmpp/[email protected]
3 XMPP/[email protected]
3 XMPP/[email protected]
3 XMPP/[email protected]
3 host/[email protected]
3 host/[email protected]
3 host/[email protected]
3 smtp/[email protected]
3 smtp/[email protected]
3 smtp/[email protected]
3 nfs/[email protected]
3 nfs/[email protected]
3 nfs/[email protected]
3 http/[email protected]
3 http/[email protected]
3 http/[email protected]
3 HTTP/[email protected]
3 HTTP/[email protected]
3 HTTP/[email protected]
3 pop/[email protected]
3 pop/[email protected]
3 pop/[email protected]
3 imap/[email protected]
3 imap/[email protected]
3 imap/[email protected]
3 ftp/[email protected]
3 ftp/[email protected]
3 ftp/[email protected]
3 afpserver/[email protected]
3 afpserver/[email protected]
3 afpserver/[email protected]