Estimated and actual number of rows at filtered statistics

Question

I have query like this:
SELECT productID FROM dbo.table WHERE nar_closed=1 AND nar_status=0
I have filtered index:
CREATE NONCLUSTERED INDEX [x_status] ON [dbo].[Table]
(nar_status ASC)
INCLUDE(productID)
WHERE nar_closed=1
In execution plan I can see that there is index seek - so optimal plan. But, estimated number of rows is 1716930 and actual number of rows is only 63260. How is that possible? I have rebuild index and statistic is up to date. Statistic is also filtered(like
index). Is there some problem with filtered statistics?
What should i do, to get actual and estimated number of rows similar?

Answer 1

I'm usually busy Monday night, but it turned that I got this one off.
SELECT @productID=i.item_id
FROM dbo.order_item i INNER JOIN dbo.order_header h ON i.header_id = h.header_id
WHERE i.last_item=1 AND i.item_finished=0 AND h.order_status IN(1,2,3) OPTION (MAXDOP 1, RECOMPILE)
So this is a different query from which I looked at during the weekend. Yes, you get incorrect estimates, but there is a simple reason for this. Look this query:
SELECT h.order_status, COUNT(*)
FROM order_header h
JOIN order_item i ON i.header_id = h.header_id
GROUP BY h.order_status
ORDER BY h.order_status
The output is:
order_status
0 434
1 4
2 357231
4 2311649
But look at:
SELECT h.order_status, COUNT(*)
FROM order_header h
JOIN order_item i ON i.header_id = h.header_id
WHERE i.last_item = 1
AND i.item_finished = 0
GROUP BY h.order_status
ORDER BY h.order_status
This produces:
order_status
0 76
1 4
2 63179
4 1
You have a skew. There is big correlation between having a row with item_last = 1 and item_finished = 0 and order_status = 2. Which you probably knew already. But SQL Server has no clue!
I was trying to figure out exactly how it arrived at that number, but SQL Server may be smarter than me, because my computations landed at even lower numbers.
Dealing with skews is often difficult, and you tend to end up with tricks, some uglier than others. I tried adding this filtering statistics on order_item:
CREATE STATISTICS order_item.header_filter_stat ON
order_item(header_id) WHERE last_item = 1 AND item_finished = 0
WITH FULLSCAN
But that lowered the estimate even more! If I drop the filter index ix_last, I get a Merge Join, wich only has a 50% misestimate, but I suspect that it does not give better performance. (The data volume is not significant to tell.)
I'm not concered about memory grant but with number of estimated rows.
But your concern was hash spill, and hash spill occurs because the memory grant is too low, and it is low due to the low row estimate.
Erland, I'm assigning to the varibale just for the sake of example - to get rid of table result set.
But since the query is logically different, the optimizer could produce a different plan. It does not seem to make that shortcut, but you never know. When I don't want to see the result set, I usually use SELECT INTO.
Erland Sommarskog, SQL Server MVP, [email protected]