Solaris 10 LDAP Clients Intermittently Fail

Question

I'm working on a rather puzzling issue with some of our Solaris 10 systems authenticating against DSEE 6.3. These clients previously worked without issue but starting last week SSH connections would hang for a few minutes and then start working again. This never happened on more than one system at a time.
I found the following messages in /var/adm/messages during the time we have these problems:
Apr 27 08:04:57 hostname nscd[20634]: [ID 293258 user.warning] libsldap: Status: 7 Mesg: LDAP ERROR (85): Timed out.
Apr 27 08:05:47 hostname nscd[20634]: [ID 293258 user.warning] libsldap: Status: 7 Mesg: LDAP ERROR (85): Timed out.
... many of these
Apr 27 08:10:07 hostname nscd[20634]: [ID 293258 user.warning] libsldap: Status: 7 Mesg: LDAP ERROR (85): Timed out.
Apr 27 08:10:17 hostname nscd[20634]: [ID 293258 user.warning] libsldap: Status: 7 Mesg: LDAP ERROR (85): Timed out.
Apr 27 08:10:31 hostname nscd[20634]: [ID 293258 user.warning] libsldap: Status: 7 Mesg: LDAP ERROR (81): Can't contact LDAP server.
To test connectivity to the LDAP server I have a ldapsearch running every 15 seconds an logging the time it took and checking for correct results. during the time that I see the libsldap messages and ssh connections are hanging, the ldapsearch command continues to run fine without slowing down.
A final note is that all three of the problem systems are on the same subnet and systems outside of this subnet aren't having any problems with the same configuration. My first thought was the firewall but ldapsearch continues to work.
Does anyone know if nscd tries to keep the LDAP connection open. Looking at the logged messages it appears as though it gives up after 5 minutes or so, throws the LDAP ERROR (81) and then starts to work again.
Any ideas would be appreciated. This one is making me crazy (crazier).
Thanks.

Answer 1

rukbat wrote:
Has anything changed in that time frame?
Any physical changes such as office-moves? new hires? lay-offs?
Could there have been any modifications to the networking hardware such as lengthening the cabling? Is it possible to re-route the subnet to different switches or to different posts on the switches? You might consider snooping the traffic to watch how it traverses the paths to the LDAP server.
If there are other systems on the subnet, do they experience any sort of timeouts ( even if it is to unrelated tasks such as database access or surfing to the Intranet/Internet ) ?
... just random thoughts from a hardware perspective.Given that this started after a maintenance night I'm sure you are correct and something changed. However there are no changes in the maintenance plan that could cause this and nobody will own up to any additional changes. This leaves it to me to try to find what is causing the failure so I can get it corrected.
These are the only three Unix systems on that subnet and they are all experiencing the problem so I don't have anything that is working to compare them to except for the other systems that aren't on that subnet. The other systems are working fine with the same configuration. That's why I'm thinking that it is something external to the problem systems.
Given that all other services on these systems are working, I'm not currently exploring a hardware type failure.
I've been running pfiles on nscd and it appears that it is indeed holding a connection to the LDAP server open (if I'm reading it correctly). The inode assocated with #8 hasn't changed. So my current theory is that maybe the firewall is killing off long connections after a while. This appears to be consistent with the log entries where I get many ERROR (85) and then a final (81). I'm thinking that after the ERROR 81, it re-opens the connection. Just guesses though.
8: S_IFSOCK mode:0666 dev:329,0 ino:3753 uid:0 gid:0 size:0
O_RDWR|O_NONBLOCK
SOCK_STREAM
SO_SNDBUF(49152),SO_RCVBUF(49680),IP_NEXTHOP(0.0.194.16)
sockname: AF_INET6 ::ffff:10.1.50.50 port: 42758
peername: AF_INET6 ::ffff:10.1.52.25 port: *636*