SQL stored procedure Staging.GroomDwStagingData stuck in infinite loop, consuming excessive CPU

Question

Hello
I'm hoping that someone here might be able to help or point me in the right direction. Apologies for the long post.
Just to set the scene, I am a SQL Server DBA and have very limited experience with System Centre so please go easy on me.
At the company I am currently working they are complaining about very poor performance when running reports (any).
Quick look at the database server and CPU utilisation being a constant 90-95%, meant that you dont have to be Sherlock Holmes to realise there is a problem. The instance consuming the majority of the CPU is the instance hosting the datawarehouse and in particular
a stored procedure in the DWStagingAndConfig database called Staging.GroomDwStagingData.
This stored procedure executes continually for 2 hours performing 500,000,000 reads per execution before "timing out". It is then executed again for another 2 hours etc etc.
After a bit of diagnosis it seems that the issue is either a bug or that there is something wrong with our data in that a stored procedure is stuck in an infinite loop
System Center 2012 SP1 CU2 (5.0.7804.1300)
Diagnosis details
SQL connection details
program name = SC DAL--GroomingWriteModule
set quoted_identifier on
set arithabort off
set numeric_roundabort off
set ansi_warnings on
set ansi_padding on
set ansi_nulls on
set concat_null_yields_null on
set cursor_close_on_commit off
set implicit_transactions off
set language us_english
set dateformat mdy
set datefirst 7
set transaction isolation level read committed
Store procedures executed
1. dbo.p_GetDwStagingGroomingConfig (executes immediately)
2. Staging.GroomDwStagingData (this is the procedure that executes in 2 hours before being cancelled)
The 1st stored procedure seems to return a table with the "xml" / required parameters to execute Staging.GroomDwStagingData
Sample xml below (cut right down)
<Config>
<Target>
<ModuleName>TransformActivityDim</ModuleName>
<WarehouseEntityName>ActivityDim</WarehouseEntityName>
<RequiredWarehouseEntityName>MTV_System$WorkItem$Activity</RequiredWarehouseEntityName>
<Watermark>2015-01-30T08:59:14.397</Watermark>
</Target>
<Target>
<ModuleName>TransformActivityDim</ModuleName>
<WarehouseEntityName>ActivityDim</WarehouseEntityName>
<RequiredWarehouseEntityName>MTV_System$WorkItem$Activity</RequiredWarehouseEntityName>
<ManagedTypeViewName>MTV_Microsoft$SystemCenter$Orchestrator$RunbookAutomationActivity</ManagedTypeViewName>
<Watermark>2015-01-30T08:59:14.397</Watermark>
</Target>
</Config>
If you look carefully you will see that the 1st <target> is missing the ManagedTypeViewName, which when "shredded" by the Staging.GroomDwStagingData returns the following result set
Example
DECLARE @Config xml
DECLARE @GroomingCriteria NVARCHAR(MAX)
SET @GroomingCriteria = '<Config><Target><ModuleName>TransformActivityDim</ModuleName><WarehouseEntityName>ActivityDim</WarehouseEntityName><RequiredWarehouseEntityName>MTV_System$WorkItem$Activity</RequiredWarehouseEntityName><Watermark>2015-01-30T08:59:14.397</Watermark></Target><Target><ModuleName>TransformActivityDim</ModuleName><WarehouseEntityName>ActivityDim</WarehouseEntityName><RequiredWarehouseEntityName>MTV_System$WorkItem$Activity</RequiredWarehouseEntityName><ManagedTypeViewName>MTV_Microsoft$SystemCenter$Orchestrator$RunbookAutomationActivity</ManagedTypeViewName><Watermark>2015-01-30T08:59:14.397</Watermark></Target></Config>'
SET @Config = CONVERT(xml, @GroomingCriteria)
SELECT
ModuleName = p.value(N'child::ModuleName[1]', N'nvarchar(255)')
,WarehouseEntityName = p.value(N'child::WarehouseEntityName[1]', N'nvarchar(255)')
,RequiredWarehouseEntityName =p.value(N'child::RequiredWarehouseEntityName[1]', N'nvarchar(255)')
,ManagedTypeViewName = p.value(N'child::ManagedTypeViewName[1]', N'nvarchar(255)')
,Watermark = p.value(N'child::Watermark[1]', N'datetime')
FROM @Config.nodes(N'/Config/*') Elem(p)
/* RESULTS - NOTE THE NULL VALUE FOR ManagedTypeViewName
ModuleName WarehouseEntityName RequiredWarehouseEntityName ManagedTypeViewName Watermark
TransformActivityDim ActivityDim MTV_System$WorkItem$Activity NULL 2015-01-30 08:59:14.397
TransformActivityDim ActivityDim MTV_System$WorkItem$Activity MTV_Microsoft$SystemCenter$Orchestrator$RunbookAutomationActivity 2015-01-30 08:59:14.397
When the procedure enters the loop to build its dynamic SQL to delete relevant rows from the inbound schema tables it concatenates various options / variables into an executable string. However when adding a NULL value to a string the entire string becomes
NULL which then gets executed.
Whilst executing "EXEC(NULL)" would cause SQL to throw an error and be caught, executing the following doesnt
DECLARE @null_string VARCHAR(100)
SET @null_string = 'hello world ' + NULL
EXEC(@null_string)
SELECT @null_string
So as it hasnt caused an error the next part of the procedure is to move to the next record and this is why its caught in an infinite loop
DELETE @items WHERE ManagedTypeViewName = @View
The value for the variable @View is the ManagedTypeViewName which is NULL, as ANSI_NULLS are set to ON in the connection and not overridded in the procedure then the above statement wont delete anything as it needs to handle NULL values differently (IS NULL),
so we are now stuck in an infinite loop executing NULL for 2 hours until cancelled.
I amended the stored procedure and added the following line before the loop statement which had the desired effect and "fixed" the performance issue for the time being
DELETE @items WHERE ManagedTypeViewName IS NULL
I also noticed that the following line in dbo.p_GetDwStagingGroomingConfig is commented out (no idea why as no notes in the procedure)
--AND COALESCE(i.ManagedTypeViewName, j.RelationshipTypeViewName) IS NOT NULL
There are obviously other ways to mitigate the dynamic SQL string being NULL, there's more than one way to skin a cat and thats not why I am asking this question, but what I am concerned about is that is there a reason that the xml / @GroomingCriteria is incomplete
and / or that the procedures dont handle potential NULL values.
I cant find any documentation, KBs, forum posts of anyone else having this issue which somewhat surprises me.
Would be grateful of any help / advice that anyone can provide or if someone can look at their 2 stored procedures on a later version to see if it has already been fixed. Or is it simply that we have orphaned data, this is the bit that concerns most as I dont
really want to be deleting / updating data when I have no idea what the knock on effect might be
Many many thanks
Andy

Answer 1

First thing I would do is upgrade to 2012 R2 UR5. If you are running non-US dates you need the UR5 hotfix also.
Rob Ford scsmnz.net
Cireson www.cireson.com
For a free SCSM 2012 Notify Analyst app click
here