Slow merge replication over a WAN link - only downloads

Question

Hi all,
We've been using SQL Server merge replication for a few years to synchronise data between our data centres, but we are now suffering with a big performance issue. This may be because the amount of data we are synchronising has increased a lot this year.
Our publisher is an always-on data centre in the UK. Our subscriber is a mobile data centre that travels around the world and is on for periods of up to a week at a time, approx. 25 times a year. However, it also spends the same amount of time (if not more)
switched off whilst on its travels - it is a well travelled data centre!
We have 5 database that we synchronise on these servers. However, one of our databases has high numbers of data changes between periods of subscriber downtime and our issue is that it take days to catch up when the server is powered up - the other databases
are fine.
Downloads from publisher to subscriber run at about 1.5 rows a second (which is annoying when we have hundreds of thousands of rows) but strangely uploads from subscriber to publisher run about ten times faster.
Things I have checked / tried:
all tables have non-clustered primary keys on guid columns that have the rowguid property set
changing the generation levelling threshold doesn't help
setting the agent profile to high volume doesn't help
running a trace at the publisher and subscriber shows the queries are all running very fast (less than 20 m/s generally, but there are gaps of 200 m/s or so between some batches of queries)
analysis on our WAN link shows we have huge amounts of bandwidth spare
analysis on our servers show we have huge amounts of Ram and CPU spare
Some of the places the subscriber is at do suffer from high latency but this doesn't seem to have an impact - 300 m/s or 100m/s and we still get the same poor performance.
One things I did wonder about - does the replication confirm to the publisher every time it has successfully processed a row at the subscriber? If we have thousands of rows and there is a latency on the line will this compound the issue if it confirms each
item? If this does happen, is there a way to batch up messages between publisher and subscriber?
Any help that you can offer will be gladly received!
Thanks
Mark

Answer 1

1) merge replication processes the uploads/downloads in batches. So no, there is no confirmation for each row processes, but there is per batch. If there is an error/conflict a batch is retried as singletons (a single row processed at a time), so you need
to minimize conflicts/errors.
2) No, but you should use pull and wan accelerators for max performance and change your profile to use 2000 for the following:
MaxDownloadChanges 
MaxUploadChanges
UploadGenerationsPerBatch  
DownloadGenerationsPerBatch 
UploadReadChangesPerBatch 
DownloadReadChangesPerBatch 
UploadWriteChangesPerBatch 
DownloadWriteChangesPerBatch 
set network packet size to a large value - you will need to work with your wan engineer to find out what works best. 32k will likely be best.
Also you may want to copy a file of a known size from your publisher to your subscriber and then from your subscriber to your publisher to see if there is a significant difference in time. For drives will yield significant differences.
I have found that you may have a trigger which is generating a large volume of unnessary changes. You will need to see what table/article is generating all the changes and evaluate whether the changes are legit.
Precomputed partitions should speed up the enumeration times. Ensure you have supporting indexes - the missing indexes DMV helps here.
Lastly you may find that a reinitialization is faster than processing the changes.
looking for a book on SQL Server 2008 Administration?
http://www.amazon.com/Microsoft-Server-2008-Management-Administration/dp/067233044X looking for a book on SQL Server 2008 Full-Text Search?
http://www.amazon.com/Pro-Full-Text-Search-Server-2008/dp/1430215941