Reconnexion of two isolated master environments: election fails

Question

First some details about my configuration:
* Berkeley DB 4.7.25, no patches applied
* Replication enabled, two sites on two different servers
* Using the base replication API, not the replication manager
Initial state is: two servers up and communicating (let's say S1 and S2), one BDB environment on each server, S1 is master and S2 slave.
Then the Ethernet interface is stopped on S2.
Loss of connectivity detection occurs on S1 and S2, S2 triggers an election (not needed on S1), determines it's alone, becomes master.
Then the Ethernet interface is restored on S2.
Connectivity between the 2 servers is detected at both sides, the two isolated "partitions" have to be merged. Election triggered on both sides: nsites=2, nvotes=0 (thus nvotes=2).
I never succeed in making the election process successful. It appears that the two election "gen" numbers differ between the two servers (here, gen(S2) > gen(S1)), and that makes S1 always raise "DUPMASTER", triggering again an election, while S2 may miss one vote in its election process, and also fails electing a master.
Of course, the sequence of events may differ from time to time, but even in case of one environment winning the election, the other environment never succeed in synchronizing with the master.
The only work-around I found so far is to dynamically close both environments when a merge has to be done, remove the files "__db.rep.egen" and "__db.rep.gen", and re-open them. It's always successful.
Thanks for your help.

Answer 1

I have several questions to ask. Do you ever call rep_start with DB_CLIENT after
the isolation is complete and communication is restored? If you call rep_elect
while a master, it will send out a message asserting its mastership. From your
description that is what S2 is doing, and S1 raises the DB_REP_DUPMASTER error.
Even at that point, S1 should both return the DB_REP_DUPMASTER error as well as
broadcast out a DUPMASTER message, and therefore S2 should receive that
and return the DB_REP_DUPMASTER error. At that point, the application at
both sites should call rep_start(DB_CLIENT) and then call rep_elect again.
Is your application downgrading and then calling an election in this case?
You should not ever remove the __db.rep files yourself.
Sue LoVerso