Having LDAPS intermittent connection delays of 30sec - 3 min on DSEE 6.3.1

Question

After a lengthy evaluation, we've been running 6.3.1 for a couple of months now in production. Just recently we've been experiencing issues with delayed connections to the server. The delays seem to happen in increments of 30 seconds - sometimes as long as 3 minutes (e.g. :30, 1:00, 1:30, 2:00, 2:30) - although many seem to be 2:30 min. Access logs show that no search or modify operations occur during this period - just connection closures (either B4 or T1 tags). It's as if the DSEE software thinks the network has disappeared. Once the period of time is up, operations resume normally. Does not happen at fixed intervals - can be working fine for 2-3 hours and then hit a snag. Needless to say, some of the users hitting the directory have shorter time-outs than the 2-3 minutes this can take to resolve itself, so there are complaints. Error log does not provide any information except that there are no entires in it for the duration of the service disruption. Note that all connections to this server are SSL connections(LDAPS). I think I ruled out the network as this problem happens even when ldapsearches are issued from the localhost box. The system does not seem to be stressed that I can tell - iostat shows minimal disk usage, prstat shows under 8% CPU utilization.
Particulars:
Platform: Sparc Solaris 10
Hardware: T2000 16GB RAM
Software: DSEE 6.3.1
A couple of theories - Java garbage collection? Thread count issues? Solaris 10 listener issue?
Has this been seen before? Please help!
Thanks.

Answer 1

DPS tends to be the answer to the kind of rate limiting you are talking about. You can define a specific connection handler for a particular pattern of bind dn, send the connections to a specific view that uses a data source pool with whatever limits you want.
On the subject of open connections, I can tell you that in one of my load tests on 6.3, I was able to get a throughput of ~30K ops/sec using rsearch with 10 threads (i.e., 10 open connections). The etimes on those operations were in the 20 - 40 ms range, very acceptable performance. When I increased the number of threads to 100, I got the exact same throughput, but etimes increased substantially. When I raised the number of threads to 1000, I again got ~30K ops/sec but average etimes were over one second!
There is a polling thread configuration that can increase the number of threads available to handle incoming requests, but raising it did not change the results appreciably. So I would tend to agree that one key to your performance is limiting the number of open connections to the Directory in the first place. DPS is an excellent way to accomplish this. DPS can also offload SSL computing from your DS if you have the network security to terminate SSL at the proxy and use LDAP only to your DS systems.
Two additional thoughts regarding your OP.
- The 30 second interval is suspicious. Have you explained it? Does your connection flurry happen on a 30 second interval? What is your db checkpoint interval?
- BINDs aren't neccesarily all that expensive, but they can be if they induce writes. Do you have any password policies turned on?