How to find bottleneck in pivot query

Question

I have the following table and query I am running against that table.
CREATE TABLE StagingTable(
DateKey int NOT NULL,
VersionNumber smallint NOT NULL,
SetID varchar(10) NULL,
ClassID char(5) NULL,
VariableName varchar(50) NULL,
VariableDescription varchar(255) NULL,
PeriodNumber int NULL,
PeriodData decimal(18, 6) NULL,
Column1 varchar(50) NULL,
Column2 varchar(50) NULL,
Column3 varchar(50) NULL,
Column4 varchar(50) NULL,
Column5 int NULL,
Column6 varchar(25) NULL,
Column7 int NULL,
Column8 int NULL,
Column9 varchar(50) NULL,
Column10 varchar(50) NULL,
Column11 varchar(50) NULL,
Column12 int NULL,
RowNumber int NULL
GO
WITH cte as (
SELECT
DateKey ,
VersionNumber ,
SetID ,
ClassID ,
VariableName ,
PeriodNumber ,
PeriodData
from StagingTable
SELECT
DateKey ,
VersionNumber ,
SetID ,
ClassID ,
VariableName,
[0], [1], [2], [3], ,,,, [360]
from cte
pivot (SUM(PeriodData)
FOR PeriodNumber IN (
[0], [1], [2], [3], ..., [360]
) as pvt;
CREATE UNIQUE NONCLUSTERED INDEX IX_StagingTable ON dbo.StagingTable
DateKey,
VersionNumber,
SetId,
ClassID,
VariableName,
PeriodNumber
INCLUDE ([PeriodData])
I have about 5 million rows in StageTable and the pivot query returns 32,000 rows.  It takes about 3 and a half minutes to run this query, whether I am outputting the results to the results window in SSMS or selecting into a new table.
I checked the execution plan for this query (outputting to results window) and the only operation that takes any time is the non-clustered index scan (84% with Stream aggregate showing 14%).
Here are the things I compared this to.  I ran a select from this table using just the columns in the index. The results returned in less than one minute and as expected the most expensive operation (100%) was the non-clustered index scan. 
By the way, a select into returned almost immediately.
If I take the pivot results (after a select into) and create a new table by select into, it takes almost no time for the query to finish.
I did an estimated execution plan on these three queries (where PivotTable is a dump of the pivot query into a heap):
WITH cte as (
SELECT
DateKey ,
VersionNumber ,
SetID ,
ClassID ,
VariableName ,
PeriodNumber ,
PeriodData
from StagingTable
SELECT
DateKey ,
VersionNumber ,
SetID ,
ClassID ,
VariableName,
[0], [1], [2], [3], ,,,, [360]
from cte
pivot (SUM(PeriodData)
FOR PeriodNumber IN (
[0], [1], [2], [3], ..., [360]
) as pvt;
SELECT * from PivotTable;
SELECT
DateKey ,
VersionNumber ,
SetID ,
ClassID ,
VariableName ,
PeriodNumber ,
PeriodData
from StagingTable;
The first query (pivot query) had a cost of 41%, the second 25% (select pivottable) and the third 34% (select base table).  These are nowhere close to the ratios of the amount of time that it takes to run these queries (pivot query 3.5 minutes,
select  PivotTable 5 seconds and select base table almost one minute).
For the following:
WITH cte as (
SELECT
DateKey ,
VersionNumber ,
SetID ,
ClassID ,
VariableName ,
PeriodNumber ,
PeriodData
from StagingTable
SELECT
DateKey ,
VersionNumber ,
SetID ,
ClassID ,
VariableName,
[0], [1], [2], [3], ,,,, [360]
into newTable1
from cte
pivot (SUM(PeriodData)
FOR PeriodNumber IN (
[0], [1], [2], [3], ..., [360]
) as pvt;
SELECT *
into newTable2
from PivotTable;
SELECT
DateKey ,
VersionNumber ,
SetID ,
ClassID ,
VariableName ,
PeriodNumber ,
PeriodData
into newTable3
from StagingTable;
The distribution of query cost is strikingly different.  The pivot query has a cost of 98% and the other two queries have costs of 1%.  The big difference between these queries and the versions that just output the results (in the execution plan)
is in the table insert.  The pivot query shows a cost of 26,000 while the select from pivottable shows cost of 160 and select from StagingTable shows cost of 280.
What else can I look at?  Also, what else can I try to optimize the pivot query.
For now, I am going to see if I get better performance letting SSRS do the pivoting.
Russel Loski, MCT, MCSE Data Platform/Business Intelligence. Twitter: @sqlmovers; blog: www.sqlmovers.com

Answer 1

I have finally nailed down what is causing the bottleneck.
The Compute Scalar step for some reason shows almost no cost to the execution plan.  But I noticed that the actual number of rows is 5 million (the same as the scan).  It appears that SQL Server expands every row to 360+ columns, all 5 million
of them.
So I did an experiment.  I took the table that I was populating and did a cross join to a number table.  This table has 30 K rows and 360 columns.  I figured that if I select 150 rows from the number table it will generate around 5 million
rows.  The results were almost identical to using the MAX(CASE WHEN ...) construct:  1 minute and some change.
I then cross joined with only 75 rows.  It took 30 seconds.  37 rows took about 15 seconds.
So the Compute Scalar costs a lot more then the Execution plan analyzer says.
SSIS appears to gather one row at a time.  Once it finishes one row, one group by set, it sends the row on its way.  So, in my case, the 5 million rows is read in with 7 columns and 30 K rows are output with 365 columns or so.
SQL Server creates 5 million rows with 365 columns, then aggregates them down to 30 K rows.
Thank you, Tom for you discussion with me on this.  I think that I can present my thoughts to my lead.
Russel Loski, MCT, MCSE Data Platform/Business Intelligence. Twitter: @sqlmovers; blog: www.sqlmovers.com