Adding Replication to an existing App

Question

We're looking to add replication to our Bdb JE enabled application. We use the Direct Persistence Layer. So far, I've swapped out our use of the Environment object with that of a ReplicatedEnvironment object and wrapped our transactions with code similar to je.rep.quote.RunTransaction example. So basically, the application assumes that the local environment is master and replicates out to the group.
This works, however, when I bring the app down, the other nodes elect a new master and when I bring the app back up and try to write, the write fails because the local environment is now a Replica. This is disconcerting to me as I don't see a manual way of forcing the local environment to be the master when it comes up (after having sync'ed with the other nodes).
An alternative would be to move all persistence out of my application into remote nodes. I'm looking at the example code under je/rep. The example shows an HARouter class that can forward write requests to the master. However it looks rather low level, with the client application needing to marshall the requests to the server and the DPL code presumably would reside on the server nodes. My client code would need to change significantly to be able to write to an out of process node and I would have to keep the server DPL in sync with the client (my data's lifetime is one day, so schema changes are irrelevant).
Am I correct in this assessment, or are there APIs to more seamlessly forward writes while still offering the client the DPL API? If this is the case, are there APIs to control which node is master?
Thanks,
Adam
Edited by: 833022 on Feb 1, 2011 11:13 AM

Answer 1

Adam,
You're right that BDB JE does not currently provide APIs to forward writes seamlessly. There's two ways to think about this.
If high availability is a priority, then you'd like your application to be able to flexibly handle the fact that mastership may have moved to any node in your group. If it can do that, you''ll be able to support writes with the least amount of disruption and delay in the event of a failover in the system. Alternatively, an application may find it preferable to have the writes serviced from a specific node. Replication is serving more like a backup, and the application is willing to incur some overhead to have mastership moved to a specific node.
In the HA-is-a-priority case, there are two options for directing write operations. You've looked at one of them, the HA aware router based on com.sleepycat.je.rep.Monitor, which directs write requests to the master. That's the HARouter example. A second option is to set up an HA unaware router, which uses the com.sleepycat.je.rep.StateChangeListener, to track the node that is the master, and redirect operations to that node. That's the SimpleRouter example.
With both options, your application would still have to do the work of forwarding. When using StateChangeListener, the code in the server nodes does it directly. The StockQuotesRMIForwarding example illustrates how to uses RMI to add forwarding to your application.
If you'd like to take the approach where you mandate who is the master, you gain some application ease, but do reduce how available your application is, in the case where the targeted master node goes down. In BDBJE 4.2, which is not yet available, we have both an API and command line utility that transfers mastership to the specified node. I've appended the description. Unfortunately, while I think it would serve your needs, it's not yet in a released version. In BDBJE 4.1, your main option would be to use ReplicationMutableConfig.NODE_PRIORITY to skew the mastership toward the desired node. If you do that, and specify the initialElectionPolicy parameter in the ReplicatedEnvironment constructor to require that all nodes in the group participate in the first election, you could ensure that your desired node is always the master. But be careful -- if that node were down, your application would stall, unable to elect a new master!
Hope this helps,
Linda
In JE 4.2, available in the future:
ReplicationGroupAdmin.transferMastership
public void transferMastership(String nodeName, int retries, int retryInterval, TimeUnit timeUnit)
throws EnvironmentFailureException,
InterruptedException,
MasterStateException,
MasterTransferFailureException,
MemberNotFoundException,
UnknownMasterException
Transfer mastership of the replication group from the currentl master to a specified replica.
This command is forwarded to the original, current master. The original master will request that the specified replica perform a syncup so its data is up to date with the original master. The period allowed for this syncup is specified by je.rep.catchupMasterTimeout, defined in ReplicationMutableConfig.CATCHUP_MASTER_TIMEOUT. When the replica is sufficiently caught up, the original master will send out messages to notify all nodes that the specified replica will be the new master.
The original master shuts down after the mastership transfer is successfully done. A MasterReplicaTransitionExcepiton will be thrown when users try to do further operations on the original master. At that point, the original master will have to be closed and reopened to resume life as a replica.
In the event of a network partition, it's possible that the original master is isolated from other nodes, and its transfer notification message is not seen by those nodes. In that case, the new master may not be the node specified in this API. To detect such unexpected transactions, this method will check to see if the specified replica is elected as a master
Edited by: Linda Lee on Feb 1, 2011 3:42 PM
Edited by: Linda Lee on Feb 1, 2011 3:53 PM