Abrupt shutdown of master node causes problem

Question

A service in usmbcom1 (LMID ) makes tpacall to a service which is
present in
both usmbapp1 ( master node LMID ) and usmbapp2 ( slave node LMID
) ( HERE
usmbcom1 , usmbapp1 and usmbapp2 are LMIDs whereas the
corresponding physical m/cs unames are usmbd5 , usmbd3 and usmbd4
They are separate sun boxes) . LDBAL = Y and tpacall is done at
25 / sec.
Now there are 2 scenarios.
1. While tpacall is in progress we kill the servers in usmbapp1
using
kill command ( not kill -9) . Then clean up the ipcs. Only
few ( 3-5 out of a
total of 5000 ) messages were lost . This is understandable since
messages
which were already in the queue got lost). The rest of the messages
were
processes by usmbapp2.
2. In this case we switched off the sun box usmbapp1 ( m/c name
usmbd3)
while tpacall was in progress. This time we lost approx 50% of
the
messages. However if we go to the slave m/c i.e usmbapp2 and manually
make it master ( tmadmin ... master) ., then from that point of
time we stopped
losing messages.
Does that mean manual intervention is necessary if DBBL goes
down? Is there anything which I am missing out while configuring
the system?

Answer 1

Hi Scott,
You did understand the scenario and the problems.
The answers are quite convincing.
Actually the QA team here are doing failover testing
and they have both these ( kill and m/c shutdown)
as their test cases.
However I would like to know about what you meant
by High Availability Solutions.
Do you also mean that if I shutdown my master m/c
then no event would be written in the ULOG of slave,
which can be monitored and used to convert the slave
into master programatically ( I mean thru tpadmcall)
Thanks
Somashish
Scott Orshan <[email protected]> wrote:
Hi,
I'm not sure if I completely comprehend your situation,
but let me take
a guess.
When you killed the processes (including the Bridge),
which by the way
is a bad thing to do to TUXEDO, TCP notified the other
connected nodes
that the connection had dropped. This happens fairly quickly.
But if you just turn off a machine, TCP may not detect
it until it times
out, which can take several minutes. Since TUXEDO was
doing Round Robin
load balancing, half the requests were sent to the Bridge,
with a
destination of the dead machine.
To answer your final question, the DBBL has to be migrated
manually,
unless you are using one of our High Availability solutions
that uses an
external monitor.
The reason is that it is very hard to distinguish between
a network
failure or slowdown, and a real failure of the Master
node. And it would
be very bad to have two machines in the domain acting
as the Master.
     Scott Orshan
     BEA Systems
Somashish Gupta wrote:
A service in usmbcom1 (LMID ) makes tpacall to a servicewhich is
present in
both usmbapp1 ( master node LMID ) and usmbapp2 ( slavenode LMID
) ( HERE
usmbcom1 , usmbapp1 and usmbapp2 are LMIDs whereas the
corresponding physical m/cs unames are usmbd5 , usmbd3and usmbd4
They are separate sun boxes) . LDBAL = Y and tpacallis done at
25 / sec.
Now there are 2 scenarios.
1. While tpacall is in progress we kill the serversin usmbapp1
using
kill command ( not kill -9) . Then clean up theipcs. Only
few ( 3-5 out of a
total of 5000 ) messages were lost . This is understandablesince
messages
which were already in the queue got lost). The restof the messages
were
processes by usmbapp2.
2. In this case we switched off the sun box usmbapp1( m/c name
usmbd3)
while tpacall was in progress. This time we lost approx50% of
the
messages. However if we go to the slave m/c i.e usmbapp2and manually
make it master ( tmadmin ... master) ., then from thatpoint of
time we stopped
losing messages.
Does that mean manual intervention is necessary if DBBLgoes
down? Is there anything which I am missing out whileconfiguring
the system?