SQL Server 2012 Undetected Deadlock in a table with only one row

Question

  We have migrated our SQL 2000 Enterprise Database to SQL 2012 Enterprise few days ago.
  This is our main database, so most of the applications access it.
  The day after the migration, when users started to run tasks, the database access started to experiment a total failure.
  That is, all processes in the SQL 2k12 database were in lock with each other. This is a commom case of deadlock, but the Database Engine was unable to detect it.
  After some research, we found that the applications were trying to access a very simple table with only one row. This table has a number that is restarted every day and is used to number all the transactions made against the system.   So, client
applications start a new transaction, get the current number, increment it by one and commit the transaction.
  The only solution we found was to kill all user processes in SQL Server every time this situation occurs (no more than 5 minutes when all clients are accessing the database).
  No client application was changed in this migration and this process was working very well for the last 10 years.
  The problem is that SQL 2k12 is unable to handle this situation compared to SQL 2k.
  It seems to occurs with other tables too, but as this is an "entry table" the problem occurs with it first.
  I have searched internet and some suggest some workarounds like using table hints to completely lock the table at the begining of the transaction, but it can't be used to other tables.
  Does anyone have heard this to be a problem with SQL 2k12? Is there any fixes to make SQL 2k12 as good as SQL 2k?

Answer 1

First off re: "Unfortunatelly, this can't be used in production environment as exclusive table lock would serialize the accesses to tables and there will be other tables that will suffer with this problem."
This is incorrect. 
Using a table to generate sequence numbers like this is a bad idea exactly because the access must be serialized.  Since you can't switch to a SEQUENCE object, which is the correct solution, the _entire goal_ of this exercise to find a way to properly
serialize access to this table.  Using exclusive locking will not be necessary for all the tables; just for the single-row table used for generating sequence values with a cursor.
I converted the sample program to VB.NET:
Public Class Form1
Private mbCancel As Boolean = False
Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
Dim soConn As ADODB.Connection
Dim soRst As ADODB.Recordset
Dim sdData As Date
Dim slValue As Long
Dim slDelay As Long
'create database vbtest
'go
' CREATE TABLE [dbo].[ControlNumTest](
' [UltData] [datetime] NOT NULL,
' [UltNum] [int] NOT NULL,
' CONSTRAINT [PK_CorreioNumTeste] PRIMARY KEY CLUSTERED
' [UltData] Asc
' )WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON, FILLFACTOR = 80) ON [PRIMARY]
' ) ON [PRIMARY]
mbCancel = False
Do
' Configure the Connection object
soConn = New ADODB.Connection
With soConn
.ConnectionString = "Provider=SQLNCLI11;Initial Catalog=vbtest;Data Source=localhost;trusted_connection=yes"
.IsolationLevel = ADODB.IsolationLevelEnum.adXactCursorStability
.Mode = ADODB.ConnectModeEnum.adModeReadWrite
.CursorLocation = ADODB.CursorLocationEnum.adUseServer
.Open()
End With
' Start a new transaction
Call soConn.BeginTrans()
' Configure the RecordSet object
soRst = New ADODB.Recordset
With soRst
.ActiveConnection = soConn
.CursorLocation = ADODB.CursorLocationEnum.adUseServer
.CursorType = ADODB.CursorTypeEnum.adOpenForwardOnly
.LockType = ADODB.LockTypeEnum.adLockPessimistic
.Open("SELECT * FROM dbo.ControlNumTest")
End With
With soRst
sdData = .Fields!UltData.Value ' Read the last Date (LOCK INFO 1: See comments bello
slValue = .Fields!UltNum.Value ' Read the last Number
If sdData <> Date.Now.Date Then ' Date has changed?
sdData = Date.Now.Date
slValue = 1 ' Restart number
End If
.Fields!UltData.Value = sdData ' Update data
.Fields!UltNum.Value = slValue + 1 ' Next number
End With
Call soRst.Update()
Call soRst.Close()
' Ends the transaction
Call soConn.CommitTrans()
Call soConn.Close()
soRst = Nothing
soConn = Nothing
txtUltNum.Text = slValue + 1 ' Display the last number
Application.DoEvents()
slDelay = Int(((Rnd * 250) + 100) / 100) * 100
System.Threading.Thread.Sleep(slDelay)
Loop While mbCancel = False
If mbCancel = True Then
Call MsgBox("The test was canceled")
End If
Exit Sub
End Sub
Private Sub Button2_Click(sender As Object, e As EventArgs) Handles Button2.Click
mbCancel = True
End Sub
End Class
And created the table
CREATE TABLE [dbo].[ControlNumTest](
[UltData] [datetime] NOT NULL,
[UltNum] [int] NOT NULL,
CONSTRAINT [PK_CorreioNumTeste] PRIMARY KEY CLUSTERED
[UltData] Asc
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = on, FILLFACTOR = 80) ON [PRIMARY]
) ON [PRIMARY]
go insert into ControlNumTest values (cast(getdate()as date),1)
Then ran 3 copies of the program and generated the deadlock:
<deadlock>
<victim-list>
<victimProcess id="processf27b1498" />
</victim-list>
<process-list>
<process id="processf27b1498" taskpriority="0" logused="0" waitresource="KEY: 35:72057594039042048 (a01df6b954ad)" waittime="1970" ownerId="3181" transactionname="implicit_transaction" lasttranstarted="2014-02-14T15:49:31.263" XDES="0xf04da3a8" lockMode="X" schedulerid="4" kpid="9700" status="suspended" spid="51" sbid="0" ecid="0" priority="0" trancount="2" lastbatchstarted="2014-02-14T15:49:31.267" lastbatchcompleted="2014-02-14T15:49:31.267" lastattention="1900-01-01T00:00:00.267" clientapp="vbt" hostname="DBROWNE2" hostpid="21152" loginname="NORTHAMERICA\dbrowne" isolationlevel="read committed (2)" xactid="3181" currentdb="35" lockTimeout="4294967295" clientoption1="671088672" clientoption2="128058">
<executionStack>
<frame procname="adhoc" line="1" stmtstart="80" sqlhandle="0x020000008376181f3ad0ea908fe9d8593f2e3ced9882f5c90000000000000000000000000000000000000000">
UPDATE [dbo].[ControlNumTest] SET [UltData]=@Param000004,[UltNum]=@Param000005 </frame>
<frame procname="unknown" line="1" sqlhandle="0x0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000">
unknown </frame>
</executionStack>
<inputbuf>
(@Param000004 datetime,@Param000005 int)UPDATE [dbo].[ControlNumTest] SET [UltData]=@Param000004,[UltNum]=@Param000005 </inputbuf>
</process>
<process id="processf6ac9498" taskpriority="0" logused="10000" waitresource="KEY: 35:72057594039042048 (a01df6b954ad)" waittime="1971" schedulerid="5" kpid="30516" status="suspended" spid="55" sbid="0" ecid="0" priority="0" trancount="1" lastbatchstarted="2014-02-14T15:49:31.267" lastbatchcompleted="2014-02-14T15:49:31.267" lastattention="1900-01-01T00:00:00.267" clientapp="vbt" hostname="DBROWNE2" hostpid="27852" loginname="NORTHAMERICA\dbrowne" isolationlevel="read committed (2)" xactid="3182" currentdb="35" lockTimeout="4294967295" clientoption1="671156256" clientoption2="128058">
<executionStack>
<frame procname="adhoc" line="1" sqlhandle="0x020000003c6309232ab0edbe0a7790a816a09c4c5ac6f43c0000000000000000000000000000000000000000">
FETCH API_CURSOR0000000000000001 </frame>
<frame procname="unknown" line="1" sqlhandle="0x0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000">
unknown </frame>
</executionStack>
<inputbuf>
FETCH API_CURSOR0000000000000001 </inputbuf>
</process>
</process-list>
<resource-list>
<keylock hobtid="72057594039042048" dbid="35" objectname="vbtest.dbo.ControlNumTest" indexname="PK_CorreioNumTeste" id="lockff6e6c80" mode="U" associatedObjectId="72057594039042048">
<owner-list>
<owner id="processf6ac9498" mode="S" />
<owner id="processf6ac9498" mode="U" requestType="wait" />
</owner-list>
<waiter-list>
<waiter id="processf27b1498" mode="X" requestType="convert" />
</waiter-list>
</keylock>
<keylock hobtid="72057594039042048" dbid="35" objectname="vbtest.dbo.ControlNumTest" indexname="PK_CorreioNumTeste" id="lockff6e6c80" mode="U" associatedObjectId="72057594039042048">
<owner-list>
<owner id="processf27b1498" mode="U" />
<owner id="processf27b1498" mode="U" />
<owner id="processf27b1498" mode="X" requestType="convert" />
</owner-list>
<waiter-list>
<waiter id="processf6ac9498" mode="U" requestType="wait" />
</waiter-list>
</keylock>
</resource-list>
</deadlock>
It's the S lock that comes from the cursor read that's the villian here.  U locks are compatible with S locks, so one session gets a U lock and another gets an S lock.  But then the session with an S needs a U, and the session with a U needs an
X.  Deadlock. 
I'm not sure what kind of locks were taken by this cursor code on SQL 2000, but on SQL 2012, this code is absolutely broken and should deadlock.
The right way to fix this code is to add (UPDLOCK,SERIALIZABLE) to the cursor
.Open("SELECT * FROM dbo.ControlNumTest with (updlock,serializable)")
So each session reads the table with a restrictive lock, and you don't mix S, U and X locks in this transaction.  This resolves the deadlock, but requires a code change.
I tried several things that didn't require a code, which did not resolve the deadlock;
1) setting ALLOW_ROW_LOCKS=OFF ALLOW_PAGE_LOCKS=OFF
2) SERIALIZABLE isolation level
3) Switching OleDB providers from SQLOLEDB to SQLNCLI11
Then I replaced the table with a view containing a lock hint:
CREATE TABLE [dbo].[ControlNumTest_t](
[UltData] [datetime] NOT NULL,
[UltNum] [int] NOT NULL,
CONSTRAINT [PK_CorreioNumTeste] PRIMARY KEY CLUSTERED
[UltData] Asc
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = on, FILLFACTOR = 80) ON [PRIMARY]
) ON [PRIMARY]
go
create view ControlNumTest as
select * from ControlNumTest_t with (tablockx)
Which, at least in my limited testing, resovlved the deadlock without any client code change.
David
David http://blogs.msdn.com/b/dbrowne/