Why updates on some tables with always on synchronize mode take * 100 than when asynchronize mode

Question

I have Always-On synchronization on some databases, We want to change from asynchronize mode to synchronize mode, I tested to see what the  change in performance between the two, and found that in general the select statements keep the same
duration with and without synchronize mode, most of the update\insert usually take around * 4 times from the asynchronize mode, BUT some updates take much longer. I would like to understand why things take MUCH longer (I understand that working with synchronize
mode is a 2 phase commit and should take * 4 times, but I can't understand why some of them take 50 - 100 times more).
1. One query is an update made on a table with few records (up to 40 records), this update run 250K times a day, and a very simple update on this table takes * 98 than without synchronize mode
1. An update on a big table with 2 million records, using a query that specify the unique column in the primary cluster key, takes 50 times more.
What are  the factors that have dramatic influence on performance when 2 phase commit (asynchronize mode) is used?

Answer 1

I've never even looked at the details for SQL Server, but on any kind of a system doing synchronous updates, you have to figure there is a fixed cost, a variable cost, and a queuing cost.  The fixed cost always hits whether its a big transaction or
a small transaction, let's guess it's about 0.1 seconds for a round trip.  Variable cost depends on how much is updated, let's say it's linear, and is maybe 0.05 seconds for a couple of small rows.  The queuing cost varies from 0 to huge.
So if you have a small transaction on a local system that takes 0.05 seconds (50 milliseconds), then just one will incur the local 0.05, plus 0.10, plus 0.05, and zero queuing cost, so 0.05 -> 0.20, about 4x.
But if you have something that runs 250k times a day I hope it's faster, maybe it's only 0.01 seconds locally, and that turns to 0.01 + 0.10 + 0.01 + q, so 0.01 -> 0.12+q, so it's at *least* 12x slower, and maybe if you get 100 of them
in the same second you start incurring queuing delays also, in fact you may have similar queuing delays on the local system and remote system besides any communications queuing, but if just the synchronization system has some queuing limits and it even gets
to 0.30 seconds then 0.01 -> 0.42, or 42x, and you start seeing what can happen.  If your local transaction is only 0.001 seconds when not synchronized, then you'd have 100x slowdown just on the fixed overhead!
IOW, synchronized systems and tons of small, fast transactions just don't fit together very well.
HTH,
Josh