IPlanet 5.1 sp 1 hung

Question

Hi,
I've read some posting regarding how iDS 5.1 hung. Some of the cause will be "backup" task.
I'm experiencing the same hung process (the ldapsearch doesn't return anything, just stay like that). I tried to redo the "backup binary and ldif", but it doesn't make the instance hung.
I think the connection is being made, but never been served.
Here some example of access log :
[01/May/2006:12:16:26 -0400] conn=1488823 op=-1 fd=48 closed - B1
[01/May/2006:12:16:26 -0400] conn=1488834 op=-1 fd=57 closed - B1
[01/May/2006:12:20:30 -0400] conn=1488871 fd=62 slot=62 connection from x.x.x.x to y.y.y.y
[01/May/2006:12:20:30 -0400] conn=1488871 op=0 BIND dn="cn=Directory Manager" method=128 version=2
[01/May/2006:12:20:30 -0400] conn=1488871 op=0 RESULT err=0 tag=97 nentries=0 etime=0 dn="cn=directory manager"
[01/May/2006:12:20:30 -0400] conn=1488871 op=1 MOD dn="uid=myuid,ou=people,o=extranet,dc=brownies,dc=org"
[01/May/2006:12:21:37 -0400] conn=1488872 fd=71 slot=71 connection from x.x.x.x to y.y.y.y
[01/May/2006:12:20:30 -0400] conn=1488871 op=1 MOD dn="uid=myuid1,ou=people,o=extranet,dc=brownies,dc=org"
[01/May/2006:12:21:37 -0400] conn=1488872 fd=71 slot=71 connection from x.x.x.x to y.y.y.y
[01/May/2006:12:21:37 -0400] conn=1488872 op=0 BIND dn="cn=Directory Manager" method=128 version=2
[01/May/2006:12:21:37 -0400] conn=1488872 op=0 RESULT err=0 tag=97 nentries=0 etime=0 dn="cn=directory manager"
[01/May/2006:12:21:37 -0400] conn=1488872 op=1 MOD dn="uid=myuid2,ou=people,o=extranet,dc=brownies,dc=org"
[01/May/2006:12:21:42 -0400] conn=1488873 fd=72 slot=72 connection from x.x.x.x to y.y.y.y
[01/May/2006:12:23:20 -0400] conn=1488874 fd=73 slot=73 connection from z.z.z.z to y.y.y.y
[01/May/2006:12:21:42 -0400] conn=1488873 fd=72 slot=72 connection from x.x.x.x to y.y.y.y
[01/May/2006:12:21:42 -0400] conn=1488873 fd=72 slot=72 connection from w.w.w.w to y.y.y.y
There are a bunch of open connection and never closed, until I killed the process (kill -9) and restarted the instance (Took sometimes to restart, in errors file I see the DS try to gather up the database from inorder shutdown. I believe that because of my kill -9)
If some1 can help where I start to investigate ? Is this a patch issue ? according to the owner this instance never hung before for a long time (1+ years). Can "B1" cause memory leak ?
thanks in advance,
joey

Answer 1

Hi Wajih,
thanks for the reply. Yeah, I suspect the culprit would be "bug". It the box owner doesn't want to apply patches without proof... well, finding proof for "bug" and recreate the situation is nightmare... ^_^
I believe the situation is not entirely, iDS 5.1 fault. This single master topology has to go through firewall. The master needs to replicate consumer which is on the other side of the island.
I see a bunch of "NSMMReplicationPlugin" errors (like couldn't send replica, bla..., failed to connect). Also, I run "cronjob" to do "ldapsearch" every 10 mins. Some of them failed, randomly.
It might be when the master tries to send to consumer, and the process (BER element) got joepardized somewhere, by the power of firewall stopage, the replication got stuck ! (like MOD process).
Then when, other MOD process comes, it just pile up and make bottleneck/eat up resource/used thread/etc, until the "ns-slapd" finally hung.
Yeah, if some colleague has experience in finding "bug" and guidelines, and can share it with me, that would be wonderful :)
thanks....