Chatty replication protocol (2.6 messages per transaction): bad settings?

Question

Hi,
I'm trying to determine why for a two-site replication setup (Linux, BDB 4.6.21 with the latest patch) it seems like a single transaction can take up to 100 ms to be replicated under heavy load when the non-replicated transaction takes 5 ms.
The transaction in question is a fully-synchronized update (ie: a commit at site R1 should not return until R2 has also committed and flushed the transaction to disk), and there are only two sites in the replication group, the master and the replica.
Here are the statistics I gathered from my application, at the replica after it was promoted to the master (the master having been killed):
Replication: txnsApplied=3584, repMsgsProcessed=9390, repMsgsSent=17, repMsgsSendFailures=4, nextLSN=1/1162804, nextPages=0, pagesRecords=6, pagesRequested=0, status=3, waitingLSN=0/0, waitingPages=0
Replication Manager: connectionAttemptsFailed=0, connectionsDropped=1, repMessagesDropped=0, repMessagesQueued=0, permFailed=4
It seems like the replication protocol is quite chatty (2.6 messages per transaction), which may explain some of the latency.
I use 2048-byte pages, and the typical transaction size is no more than 1024 bytes; I experimented with relaxing the transactional guarantees (setTxnNoSync et al) but had a similar experience.
Does anyone have any suggestions for this environment configuration to increase the throughput of this kind of replication setup? Your help would be kindly appreciated.

Answer 1

Replication: txnsApplied=3584, repMsgsProcessed=9390,
repMsgsSent=17, repMsgsSendFailures=4,
nextLSN=1/1162804, nextPages=0, pagesRecords=6,
pagesRequested=0, status=3, waitingLSN=0/0,
waitingPages=0
Replication Manager: connectionAttemptsFailed=0,
connectionsDropped=1, repMessagesDropped=0,
repMessagesQueued=0, permFailed=4
It seems like the replication protocol is quite
chatty (2.6 messages per transaction), which may
explain some of the latency.The replication protocol is that each log record written to the
log is also transmitted to the client. Therefore, if your transaction
has a lot of operations, or if even a single update results in several
log records written (splitting a page, for example), each of those
log records will be transmitted to the client.
Does anyone have any suggestions for this environment
configuration to increase the throughput of this kind
of replication setup? Your help would be kindly
appreciated.You can try the bulk message configuration option. Look at
the dbenv->rep_set_config method with the DB_REP_CONF_BULK
flag. That option collects the log records locally in memory and
then sends them all either on a txn_commit operation or if the
buffer fills up. That should result in fewer messages.
However, it is also interesting that there were 4 message send failures
as well as 4 permFailed counts in your statistics. It is likely whatever
caused those resulted in some messages to rerequest the missing
records.
Sue LoVerso
Oracle