Session Bean Client Hangs when one Server in Cluster Fails

Question

We are testing several failure scenarios and one has come up that concerns us.
Some background: Were running a WLS6.1 cluster on two separate machines. We
start a test client consisting of 50 active threads and let them start calling
into a session bean. After a couple minutes we pull the network plug out of one
of the machines to simulate an uncontrolled crash of the machine. Once the plug
is pulled the clients hang and of more concern any new clients that we startup
also hang. Has anyone successfully solved this problem?

Answer 1

When we kill one of Weblogic instances in the cluster none of the clients fail.
All of our clients fail-over to the remaining servers. It's pulling the network
plug to our of the server that causes everything to hang. Not just or test client,
but the other servers in the cluster hang. The control panel doesn't respond
at all either. We currently have a support case open with BEA #348184 about this.
We've gotten a prompt response in which we were asked to modify our configuration
by deploying our beans to each individual server rather than the cluster. We
did this, but the results so far have not changed.
Thanks for the feedback,
Howard.
"Ade Barkah" <[email protected]> wrote:
We haven't encountered something like that, so it could be a setup problem.
Can you verify that the t3 url hostname the client threads use resolves
to the
ip addresses of each machine in the cluster? Are all machines in the
cluster
listening at the same port number? Also, does it matter if you kill one
of the
weblogic processes instead of pulling the plug? (i.e., if you leave the
network
layer up?)
Check also that your threads aren't simply blocking each other when the
server
goes down? E.g. start multiple test client processes with one thread
each just
to test.
What we notice is (with round-robin cluster policy), as we bring down
one of
the servers, the clients will continue to work on the second server,
but will slow
down between method invocations as they still attempt to connect to the
downed
server.
After a short period of time (~30 seconds) the clients will fully switch
to the
second machine and processing continues at full speed again, until the
downed
machine is brought back up, at which point work is distributed evenly
again.
Also, when the first server is brought down, some of the clients may
terminate
with a PeerGoneException (or something similar to that.) So unless your
threads
are catching exceptions, they might terminate as well.
regards,
-Ade
"howard spector" <[email protected]> wrote in message news:[email protected]...
We are testing several failure scenarios and one has come up that concernsus.
Some background: Were running a WLS6.1 cluster on two separate machines.We
start a test client consisting of 50 active threads and let them startcalling
into a session bean. After a couple minutes we pull the network plugout of one
of the machines to simulate an uncontrolled crash of the machine. Once the plug
is pulled the clients hang and of more concern any new clients thatwe startup
also hang. Has anyone successfully solved this problem?