Very Slow Query with CTE inner join

Question

I have 2 tables (heavily simplified here to show relevant columns):
CREATE TABLE tblCharge
(ChargeID int NOT NULL,
ParentChargeID int NULL,
ChargeName varchar(200) NULL)
CREATE TABLE tblChargeShare
(ChargeShareID int NOT NULL,
ChargeID int NOT NULL,
TotalAmount money NOT NULL,
TaxAmount money NULL,
DiscountAmount money NULL,
CustomerID int NOT NULL,
ChargeShareStatusID int NOT NULL)
I have a very basic View to Join them:
CREATE VIEW vwBASEChargeShareRelation as
Select c.ChargeID, ParentChargeID, s.CustomerID, s.TotalAmount, isnull(s.TaxAmount, 0) as TaxAmount, isnull(s.DiscountAmount, 0) as DiscountAmount
from tblCharge c inner join tblChargeShare s
on c.ChargeID = s.ChargeID Where s.ChargeShareStatusID < 3
GO
I then have a view containing a CTE to get the children of the Parent Charge:
ALTER VIEW [vwChargeShareSubCharges] AS
WITH RCTE AS
SELECT ParentChargeId, ChargeID, 1 AS Lvl, ISNULL(TotalAmount, 0) as TotalAmount, ISNULL(TaxAmount, 0) as TaxAmount,
ISNULL(DiscountAmount, 0) as DiscountAmount, CustomerID, ChargeID as MasterChargeID
FROM vwBASEChargeShareRelation Where ParentChargeID is NULL
UNION ALL
SELECT rh.ParentChargeID, rh.ChargeID, Lvl+1 AS Lvl, ISNULL(rh.TotalAmount, 0), ISNULL(rh.TaxAmount, 0), ISNULL(rh.DiscountAmount, 0) , rh.CustomerID
, rc.MasterChargeID
FROM vwBASEChargeShareRelation rh
INNER JOIN RCTE rc ON rh.PArentChargeID = rc.ChargeID and rh.CustomerID = rc.CustomerID
Select MasterChargeID as ChargeID, CustomerID, Sum(TotalAmount) as TotalCharged, Sum(TaxAmount) as TotalTax, Sum(DiscountAmount) as TotalDiscount
from RCTE
Group by MasterChargeID, CustomerID
GO
So far so good, I can query this view and get the total cost for a line item including all children.
The problem occurs when I join this table. The query:
Select t.* from vwChargeShareSubCharges t
inner join
tblChargeShare s
on t.CustomerID = s.CustomerID
and t.MasterChargeID = s.ChargeID
Where s.ChargeID = 1291094
Takes around 30 ms to return a result (tblCharge and Charge Share have around 3.5 million records).
But the query:
Select t.* from vwChargeShareSubCharges t
inner join
tblChargeShare s
on t.CustomerID = s.CustomerID
and t.MasterChargeID = s.ChargeID
Where InvoiceID = 1045854
Takes around 2 minutes to return a result - even though the only charge with that InvoiceID is the same charge as the one used in the previous query.
The same thing occurs if I do the join in the same query that the CTE is defined in.
I ran the execution plan for each query. The first (fast) query looks like this:
The second(slow) query looks like this:
I am at a loss, and my skills at decoding execution plans to resolve this are lacking.
I have separate indexes on tblCharge.ChargeID, tblCharge.ParentChargeID, tblChargeShare.ChargeID, tblChargeShare.InvoiceID, tblChargeShare.ChargeShareStatusID
Any ideas? Tested on SQL 2008R2 and SQL 2012

Answer 1

>> The database is linked [sic] to an established app and the column and table names can't be changed. <<
Link? That is a term from pointer chains and network databases, not SQL. I will guess that means the app came back in the old pre-RDBMS days and you are screwed. 
>> I am not too worried about the money field [sic], this is used for money and money based calculations so the precision and rounding are acceptable at this level. <<
Field is a COBOL concept; columns are totally different. MONEY is how Sybase mimics the PICTURE clause that puts currency signs, commas, period, etc in a COBOL money field. 
Using more than one operation (multiplication or division) on money columns will produce severe rounding errors. A simple way to visualize money arithmetic is to place a ROUND() function calls after 
every operation. For example,
Amount = (Portion / total_amt) * gross_amt
can be rewritten using money arithmetic as:
Amount = ROUND(ROUND(Portion/total_amt, 4) * 
gross_amt, 4)
Rounding to four decimal places might not seem an 
issue, until the numbers you are using are greater 
than 10,000. 
BEGIN
DECLARE @gross_amt MONEY,
 @total_amt MONEY,
 @my_part MONEY,
 @money_result MONEY,
 @float_result FLOAT,
 @all_floats FLOAT;
 SET @gross_amt = 55294.72;
 SET @total_amt = 7328.75;
 SET @my_part = 1793.33;
 SET @money_result = (@my_part / @total_amt) * 
@gross_amt;
 SET @float_result = (@my_part / @total_amt) * 
@gross_amt;
 SET @Retult3 = (CAST(@my_part AS FLOAT)
 / CAST( @total_amt AS FLOAT))
 * CAST(FLOAT, @gross_amt AS FLOAT);
 SELECT @money_result, @float_result, @all_floats;
END;
@money_result = 13525.09 -- incorrect
@float_result = 13525.0885 -- incorrect
@all_floats = 13530.5038673171 -- correct, with a -
5.42 error 
>> The keys are ChargeID(int, identity) and ChargeShareID(int, identity). <<
Sorry, but IDENTITY is not relational and cannot be a key by definition. But it sure works just like a record number in your old COBOL file system. 
>> .. these need to be int so that they are assigned by the database and unique. <<
No, the data type of a key is not determined by physical storage, but by logical design. IDENTITY is the number of a parking space in a garage; a VIN is how you identify the automobile. 
>> What would you recommend I use as keys? <<
I do not know. I have no specs and without that, I cannot pull a Kabbalah number from the hardware. Your magic numbers can identify Squids, Automobile or Lady Gaga! I would ask the accounting department how they identify a charge. 
>> Charge_Share_Status_ID links [sic] to another table which contains the name, formatting [sic] and other information [sic] or a charge share's status, so it is both an Id and a status. <<
More pointer chains! Formatting? Unh? In RDBMS, we use a tiered architecture. That means display formatting is in a presentation layer. A properly created table has cohesion – it does one and only one data element. A status is a state of being that applies
to an entity over a period time (think employment, marriage, etc. status if that is too abstract). 
An identifier is based on the Law of Identity from formal logic “To be is to be something in particular” or “A is A” informally. There is no entity here! The Charge_Share_Status table should have the encoded values for a status and perhaps a description if
they are unclear. If the list of values is clear, short and static, then use a CHECK() constraint. 
On a scale from 1 to 10, what color is your favorite letter of the alphabet? Yes, this is literally that silly and wrong. 
>> I understand what a CTE is; is there a better way to sum all children for a parent hierarchy? <<
There are many ways to represent a tree or hierarchy in SQL.  This is called an adjacency list model and it looks like this:
CREATE TABLE OrgChart 
(emp_name CHAR(10) NOT NULL PRIMARY KEY, 
 boss_emp_name CHAR(10) REFERENCES OrgChart(emp_name), 
 salary_amt DECIMAL(6,2) DEFAULT 100.00 NOT NULL,
 << horrible cycle constraints >>);
OrgChart 
emp_name  boss_emp_name  salary_amt 
==============================
'Albert'    NULL    1000.00
'Bert'    'Albert'   900.00
'Chuck'   'Albert'   900.00
'Donna'   'Chuck'    800.00
'Eddie'   'Chuck'    700.00
'Fred'    'Chuck'    600.00
This approach will wind up with really ugly code -- CTEs hiding recursive procedures, horrible cycle prevention code, etc.  The root of your problem is not knowing that rows are not records, that SQL uses sets and trying to fake pointer chains with some
vague, magical non-relational "id".  
This matches the way we did it in old file systems with pointer chains.  Non-RDBMS programmers are comfortable with it because it looks familiar -- it looks like records and not rows.  
Another way of representing trees is to show them as nested sets. 
Since SQL is a set oriented language, this is a better model than the usual adjacency list approach you see in most text books. Let us define a simple OrgChart table like this.
CREATE TABLE OrgChart 
(emp_name CHAR(10) NOT NULL PRIMARY KEY, 
 lft INTEGER NOT NULL UNIQUE CHECK (lft > 0), 
 rgt INTEGER NOT NULL UNIQUE CHECK (rgt > 1),
  CONSTRAINT order_okay CHECK (lft < rgt));
OrgChart 
emp_name         lft rgt 
======================
'Albert'      1   12 
'Bert'        2    3 
'Chuck'       4   11 
'Donna'       5    6 
'Eddie'       7    8 
'Fred'        9   10 
The (lft, rgt) pairs are like tags in a mark-up language, or parens in algebra, BEGIN-END blocks in Algol-family programming languages, etc. -- they bracket a sub-set.  This is a set-oriented approach to trees in a set-oriented language. 
The organizational chart would look like this as a directed graph:
            Albert (1, 12)
    Bert (2, 3)    Chuck (4, 11)
                   /    |   \
                 /      |     \
               /        |       \
             /          |         \
        Donna (5, 6) Eddie (7, 8) Fred (9, 10)
The adjacency list table is denormalized in several ways. We are modeling both the Personnel and the Organizational chart in one table. But for the sake of saving space, pretend that the names are job titles and that we have another table which describes the
Personnel that hold those positions.
Another problem with the adjacency list model is that the boss_emp_name and employee columns are the same kind of thing (i.e. identifiers of personnel), and therefore should be shown in only one column in a normalized table.  To prove that this is not
normalized, assume that "Chuck" changes his name to "Charles"; you have to change his name in both columns and several places. The defining characteristic of a normalized table is that you have one fact, one place, one time.
The final problem is that the adjacency list model does not model subordination. Authority flows downhill in a hierarchy, but If I fire Chuck, I disconnect all of his subordinates from Albert. There are situations (i.e. water pipes) where this is true, but
that is not the expected situation in this case.
To show a tree as nested sets, replace the nodes with ovals, and then nest subordinate ovals inside each other. The root will be the largest oval and will contain every other node.  The leaf nodes will be the innermost ovals with nothing else inside them
and the nesting will show the hierarchical relationship. The (lft, rgt) columns (I cannot use the reserved words LEFT and RIGHT in SQL) are what show the nesting. This is like XML, HTML or parentheses. 
At this point, the boss_emp_name column is both redundant and denormalized, so it can be dropped. Also, note that the tree structure can be kept in one table and all the information about a node can be put in a second table and they can be joined on employee
number for queries.
To convert the graph into a nested sets model think of a little worm crawling along the tree. The worm starts at the top, the root, makes a complete trip around the tree. When he comes to a node, he puts a number in the cell on the side that he is visiting
and increments his counter.  Each node will get two numbers, one of the right side and one for the left. Computer Science majors will recognize this as a modified preorder tree traversal algorithm. Finally, drop the unneeded OrgChart.boss_emp_name column
which used to represent the edges of a graph.
This has some predictable results that we can use for building queries.  The root is always (left = 1, right = 2 * (SELECT COUNT(*) FROM TreeTable)); leaf nodes always have (left + 1 = right); subtrees are defined by the BETWEEN predicate; etc. Here are
two common queries which can be used to build others:
1. An employee and all their Supervisors, no matter how deep the tree.
 SELECT O2.*
   FROM OrgChart AS O1, OrgChart AS O2
  WHERE O1.lft BETWEEN O2.lft AND O2.rgt
    AND O1.emp_name = :in_emp_name;
2. The employee and all their subordinates. There is a nice symmetry here.
 SELECT O1.*
   FROM OrgChart AS O1, OrgChart AS O2
  WHERE O1.lft BETWEEN O2.lft AND O2.rgt
    AND O2.emp_name = :in_emp_name;
3. Add a GROUP BY and aggregate functions to these basic queries and you have hierarchical reports. For example, the total salaries which each employee controls:
 SELECT O2.emp_name, SUM(S1.salary_amt)
   FROM OrgChart AS O1, OrgChart AS O2,
        Salaries AS S1
  WHERE O1.lft BETWEEN O2.lft AND O2.rgt
    AND S1.emp_name = O2.emp_name 
   GROUP BY O2.emp_name;
4. To find the level and the size of the subtree rooted at each emp_name, so you can print the tree as an indented listing. 
SELECT O1.emp_name, 
   SUM(CASE WHEN O2.lft BETWEEN O1.lft AND O1.rgt 
   THEN O2.sale_amt ELSE 0.00 END) AS sale_amt_tot,
   SUM(CASE WHEN O2.lft BETWEEN O1.lft AND O1.rgt 
   THEN 1 ELSE 0 END) AS subtree_size,
   SUM(CASE WHEN O1.lft BETWEEN O2.lft AND O2.rgt
   THEN 1 ELSE 0 END) AS lvl
  FROM OrgChart AS O1, OrgChart AS O2
 GROUP BY O1.emp_name;
5. The nested set model has an implied ordering of siblings which the adjacency list model does not. To insert a new node, G1, under part G.  We can insert one node at a time like this:
BEGIN ATOMIC
DECLARE rightmost_spread INTEGER;
SET rightmost_spread 
    = (SELECT rgt 
         FROM Frammis 
        WHERE part = 'G');
UPDATE Frammis
   SET lft = CASE WHEN lft > rightmost_spread
                  THEN lft + 2
                  ELSE lft END,
       rgt = CASE WHEN rgt >= rightmost_spread
                  THEN rgt + 2
                  ELSE rgt END
 WHERE rgt >= rightmost_spread;
 INSERT INTO Frammis (part, lft, rgt)
 VALUES ('G1', rightmost_spread, (rightmost_spread + 1));
 COMMIT WORK;
END;
The idea is to spread the (lft, rgt) numbers after the youngest child of the parent, G in this case, over by two to make room for the new addition, G1.  This procedure will add the new node to the rightmost child position, which helps to preserve the idea
of an age order among the siblings.
6. To convert a nested sets model into an adjacency list model:
SELECT B.emp_name AS boss_emp_name, E.emp_name
  FROM OrgChart AS E
       LEFT OUTER JOIN
       OrgChart AS B
       ON B.lft
          = (SELECT MAX(lft)
               FROM OrgChart AS S
              WHERE E.lft > S.lft
                AND E.lft < S.rgt);
7. To find the immediate parent of a node: 
SELECT MAX(P2.lft), MIN(P2.rgt)
  FROM Personnel AS P1, Personnel AS P2
 WHERE P1.lft BETWEEN P2.lft AND P2.rgt 
   AND P1.emp_name = @my_emp_name;
I have a book on TREES & HIERARCHIES IN SQL which you can get at Amazon.com right now. It has a lot of other programming idioms for nested sets, like levels, structural comparisons, re-arrangement procedures, etc. 
--CELKO-- Books in Celko Series for Morgan-Kaufmann Publishing: Analytics and OLAP in SQL / Data and Databases: Concepts in Practice Data / Measurements and Standards in SQL SQL for Smarties / SQL Programming Style / SQL Puzzles and Answers / Thinking
in Sets / Trees and Hierarchies in SQL