SQL Server 2008 R2 Replication - not applying snapshot and not updating all repliacted columns

Question

We are using transactional replicating on SQL Server 2008 R2 (SP1) using a remote distributor. We are replicating from BaanLN, which is an ERP application to up to 5 subscribers, all using push publications. 
Tables can range from a couple million rows to 12 million rows and 100's of GBs in size. 
And it's due to the size of the tables that it was designed with a one publisher to one table architecture.  
Until recently it has been working very smooth (last four years)) but we have come across two issues I have never encountered.
While this has happen a half dozen times before, it last occurred a couple weeks ago when I was adding three new publications, again a one table per publication architecture.
We use standard SS repl proc calls to create the publications, which have been successful for years. 
On this occasion replication created the three publications, assigned the subscribers and even generated the new snapshot for all three new publications. 
However,  while it appeared that replication had created all the publications correctly from end to end, it actually only applied one of the three snapshot and created the new table on both of the new subscribers (two on each of the
publications).  It only applied the snapshot to one of the two subscribers for the second publications, and did not apply to any on the third.  
I let it run for three hours to see if it was a back log issue. 
Replication was showing commands coming across when looking at the sync verification at the publisher and 
it would even successfully pass a tracer token through each of the three new publications, despite there not being tables on either subscriber on one of the publishers and missing on one of the subscribers on another.  
I ended up attempting to reinitialize roughly a dozen times, spanning a day, and one of the two remaining publications was correctly reinitialized and the snapshot applied, but the second of the two (failed) again had the same mysterious result, and
again looked like it was successful based on all the monitoring. 
So I kept reinitializing the last and after multiple attempts spanning a day, it too finally was built correctly.  
Now the story only get a little stranger.  We just found out yesterday that on Friday the 17th 
at 7:45, the approximate time started the aforementioned deployment of the three new publications, 
we also had three transaction from a stable and vetted publication send over all changes except for a single status column. 
This publication has 12 million rows and is very active, with thousands of changes daily. 
, The three rows did not replicate a status change from a 5 to a 6. 
We verified that the status was in fact 6 on the publisher, and 
5 on both subscribers, yet no messages or errors.  All the other rows successfully updated.  
We fixed it by updating the publication from 6 back to 5 then back to 6 again on those specific rows and it worked.
The CPU is low and overall latency is minimal on the distributor. 
From all accounts the replication is stable and smooth, but very busy. 
The issues above have only recently started.  I am not sure where to look for a problem, and to that end, a solution.

Answer 1

I suspect the problem with the new publication/subscriptions not initializing may have been a result of timeouts but it is hard to say for sure.  The fact that it eventually succeeded after multiple attempts leads me to believe this.  If this happens
again, enable verbose agent logging for the Distribution Agent to see if you are getting query timeouts.  Add the parameters
-OutputVerboseLevel 2 -Output C:\TEMP\DistributionAgent.log to the Distribution Agent Run Agent job step, rerun the agent, and collect the log.
If you are getting query timeouts, try increasing the Distribution Agent -QueryTimeOut parameter.  The default is 1800 seconds.  Try bumping this up to 3600 seconds.
Regarding the three transactions not replicating, inspect MSrepl_errors in the distribution database for the time these transactions occurred and see if any errors occurred.
Brandon Williams (blog |
linkedin)