Frequent c2s failures

Question

Hi,
BM3.9sp2ir1
NW6.5sp8(with post sp8 patches through April 17,2010)
eDir8.8sp4
We frequently are having problems with clients connecting to our system.
Most using the bm3xvpn12 client with NMAS.
Often have to try multiple times to get connected, sometimes restart the computer, sometimes it just doesn't want to work, but sometimes it will work just fine.
No rhyme or reason that I can find, same user and system will work and then not work. Though those with problems frequently have the problem.
When some have problems, others will be connected just fine, so it isn't just that the server is refusing to accept connections.
the last item shown before it times out on the IKE screen is:
IKE : Nmas user check authentication and traffic rule.
Client sits at the negotiating and authentication message for several minutes then fails with:
may be Invalid VPN Server or IKE not loaded.
No problems at all if I unload the filters, consistently get a VPN connection in under 15seconds.
Is this a possible filter issue?
I have run brdcfg to see that they are all applied. I only do filter work in Filtcfg, never in iManager.
I've run wireshark on the client side and have good and bad captures. everything seems to match up ok until it is in the port 500/4500 sections.
good:
2 cycles of port 500 communications, then switches to 4500 Wireshark Info column says: "Identity Protection (main mode)"
3 sends on 4500, then 3 receives on 4500 Wireshark Info column says: "Identity Protection (main mode)"
and then a send/receive/send on 4500 Wireshark info column says: "Quick Mode"
and then switches to UDP 353 ndsauth
and connected on the VPN.
bad:
2 cycles of port 500 communications, then switches to 4500 Wireshark Info column says: "Identity Protection (main mode)"
3 sends on 4500 Wireshark Info column says: "Identity Protection (main mode)"
NO reply from VPN server
3 sends on 500 Wireshark Info column says: "Identity Protection (main mode)"
No reply from VPN server
1 send on 4500 protocal UDPENCAP wireshare info column says NAT-keepalive
1 send on 500 Wireshark Info column says: "Identity Protection (main mode)"
4 replies on 4500 Wireshark Info column says: "Identity Protection (main mode)"
1 reply on 500 Wireshark Info column says: "Identity Protection (main mode)"
Then client sends on 500 to 4500 Wireshark Info column says: Informational
then back and forth port 500 sends show Informational, replies Identity Protection.
we also have similar problems connecting to another BM server in the same tree (different location)
BM3.8sp5
NW6.5sp5
eDir8.7.3.9

Answer 1

Ok, spoke too soon. just happened to work a couple times.
Server has read-only replicas of everything, plus is master of its own partition.
Turned on NMAS in DSTrace, any other options I should turn on when tracing? I don't see any SLP or NCP addresses.
In a failure, the trace is identical to a success, except that it appears the client doesn't recognize the success message.
Though there is a error in the traces: (both success and failure)
11:16:44 942A3060 NMAS: Accessing local replica of CN=PW Policy.CN=Password Policies.CN=Security
11:16:44 942A3060 NMAS: ERROR: -631 Failed set password for CN=user.OU=ou.O=o
11:16:44 942A3060 NMAS: 34: ERROR: -631 Server Module 0x00000007 Set Password
11:16:44 942A3060 NMAS: 34: ERROR: -631 MAF_SetPassword
11:16:44 942A3060 NMAS: 34: Server Module 0x00000007 Write
11:16:44 942A3060 NMAS: 34: Server Module 0x00000007 Read
11:16:46 942A3060 NMAS: 34: Server Module 0x00000007 Successful
11:16:46 942A3060 NMAS: 34: NDS Login Method Successful
11:16:46 942A3060 NMAS: 34: WhatNext
11:16:46 942A3060 NMAS: 34: Successful login
11:16:48 94201140 NMAS: 34: NMAS session succeeded
11:16:48 94201140 NMAS: 34: Client Session Destroy Request
11:16:48 94201140 NMAS: 34: Local Session Cleared (Not Destroyed)
11:16:48 942A3060 NMAS: 34: ERROR: -1645 Server timed out waiting for data
11:16:48 942A3060 NMAS: 34: Server thread exited
11:16:48 942A3060 NMAS: 34: Pool thread 0x8d43a600 work complete