Starting new client from scratch with large db takes forever

Question

I found a similar thread on the forum but I think my situation is a little different. I'm using the pure C API instead of Java for example.
What I currently have running is a berkeley DB database with 7 million records. There is also a secondary database used as an index on a specific key. New records are inserted about one record per second on average. Records and keys are small, 20 and 120 bytes. The primary db is 750MB in size and the index about 500.
The master runs a 'master-db' process that owns the database and a 'server' process that opens the database read-only to expose it's data to a simple network service. I currently have about 1000 queries per second on this. It's implemented io event driven (using boost.asio) and uses about 10% CPU time on a single thread.
The slave runs a 'slave-db' process that is configured as a slave replica node. It also runs the same server process and deals with about the same amount of hits.
I'm really happy with this setup, it works well.
Except in one case: when I need to sync a new slave with an empty database. This barely ever works ok and requires many manual restarts.
What I see is that the slave tries to transfer two large sets of pages. One set is 180.000 and the other 120.000. I assume that these two are for the two databases that I have.
What happens is that the slave gets about 140.000 pages in less than a couple of seconds (gigabit ethernet) but then stalls either completely or gets the remaining 40.000 pages at a 1 page/second rate. I don't see anything that looks like an error in the logs. Also no queue limit warnings.
I've noticed two things that help: stopping the master server process (so that no updates are done to the database) or just manually restarting the slave a couple of times so that eventually gets all pages. Both are not ideal of course.
Two questions:
Does this sound familiar to anyone?
Are there alternatives to the initial sync? Would it be possible to for example rsync the main db home to a slave and use that as a starting point instead?
S.

Answer 1

Hi Stefan,
I recall that we talked previously about the "queue
limit exceeded" problem. But you say that is not
happening this time. I take it you installed the bug
fix patch?No I did not install the patch actually. Do you think it could be relevant for issues besides the 'queue limit exceeded' problem or was it a very specific fix for just that?
Are you still using Replication Manager in a 4.6
version of Berkeley DB?Yes. 4.6.21.
>
What you describe is indeed strange, particularly the
part where stopping updates on the master seems to
clear up the problem. I'll make a test setup and just let it run for a much longer time. So far I've been waiting and looking at database files to change. Maybe I've been too impatient. (I did look at iostat output to see whether there was disk activity though)
Usually it's the other way
around: receiving an update from the master "kicks"
the replica into requesting any missing log records.
Can you post a representative sample of what the
verbose output at the replica looks like during the
time it's getting only about a page per second (and
especially what it looks like after stopping the
master updates)?I'll try, but the log files are HUGE and I will have to find time to analyze them really properly. So far I've only grepped for possible errors.
How do I identify the 'boundaries' in the log for one replication task?
>
You can indeed do a "manual" hot backup if you want
to avoid the automatic internal init. See the
Reference Guide page entitled "Initializing a new
site."Hm. This looks good. I had missed that page. I'm actually more fond of this approach for my specific use-case. It looks easy to automate. If I can't figure out the real problem with the auto-sync then I'll probably switch to (automated) manual.
S.