Question about behavior of txn_checkpoint()

Question

I'm just trying to determine if the behaviour I'm seeing is expected or whether I have coded something wrong.
I am using version 4.7.25 in a replicated (1 master 1 slave) environment. I have a main thread that reads network requests and does db->get() or db->put() as appropriate. I have a second thread that calls txn_checkpoint every minute. During a checkpoint, my put() performance falls right away, with some puts taking over 16 seconds to complete (the checkpoint runs for about 50 seconds).
Is this expected? Does checkpointing lock the environment, database or many pages at once or is all caused by heavy disk IO?
The environment has the following flags set: DB_AUTO_COMMIT | DB_TXN_NOSYNC and is opened with DB_CREATE | DB_INIT_LOCK | DB_INIT_LOG | DB_INIT_MPOOL | DB_INIT_TXN | DB_RECOVER | DB_THREAD|DB_INIT_REP;
My primary database has no flags set, my secondary has DB_DUPSORT. My cache is large compared to the data in the database.
One thing worth mentioning is that the machine only has 1 disk, so my databases and log records are both sharing it.
Would setting DB_MULTIVERSION and using snapshot isolation improve throughput during checkpoints at all?
TIA for any help or pointers anyone can give.
Ashley

Answer 1

I am not very familiar with the internals of checkpoints or how they do their locking. But I am aware of some known performance considerations when using checkpoints with replication.
Are you using Replication Manager? If so, what is your ack_policy and ack_timeout? I'll assume you are using Replication Manager in my answer below.
I am using version 4.7.25 in a replicated (1 master 1 slave) environment. I have a main thread that reads network requests and does db->get() or db->put() as appropriate. I have a second thread that calls txn_checkpoint every minute. During a checkpoint, my put() performance falls right away, with some puts taking over 16 seconds to complete (the checkpoint runs for about 50 seconds).
Is this expected? Does checkpointing lock the environment, database or many pages at once or is all caused by heavy disk IO?When you do a checkpoint on the master, it causes a checkpoint to happen on the client as well. For most ack_policies, the master needs to wait for the client's checkpoint to consider its own checkpoint durable.
It is possible that due to things like hardware differences or workload differences, the client's checkpoint may take longer than the master's. We provide the DB_REP_CHECKPOINT_DELAY timeout to account for this difference. You set it with DB_ENV->rep_set_timeout() and its default is 30 seconds. If you are sure your client checkpoints are going to be shorter than that, you can decrease this timeout value.
A client definitely cannot apply new master transactions while it is doing its checkpoint. This means any new master transactions must wait until the client checkpoint is finished before they can be considered durable. This may explain some of your performance issue.
>
The environment has the following flags set: DB_AUTO_COMMIT | DB_TXN_NOSYNC and is opened with DB_CREATE | DB_INIT_LOCK | DB_INIT_LOG | DB_INIT_MPOOL | DB_INIT_TXN | DB_RECOVER | DB_THREAD|DB_INIT_REP;
My primary database has no flags set, my secondary has DB_DUPSORT. My cache is large compared to the data in the database.
One thing worth mentioning is that the machine only has 1 disk, so my databases and log records are both sharing it.You are using DB_TXN_NOSYNC, which means transactions are not flushed to disk when the are committed. Depending on the size of your workload and your log buffers, you could be building up considerable activity that needs to be done during your checkpoint.
>
Would setting DB_MULTIVERSION and using snapshot isolation improve throughput during checkpoints at all?We do not support replication with transaction snapshots. This is on our list of future enhancements to consider.
Paula Bingham
Oracle