Query In SP runs 10x slower than straight query - NOT Parameter Sniffing Issue

Question

Hi Everyone,
I have a real mystery on my hands that appears to be a parameter sniffing issue with my SP but I think is not. I have tried every parameter sniffing workaround trick in the book ( local variables, option recompile, with recompile, optimize for variables,UNKNOWN,
table hints, query hints, MAXDOP, etc.) I have dropped indexes/recreated indexes, updated statistics, etc. I have restored a copy of the DB to 2 different servers ( one with identical HW specs as the box having the issue, one totally different ) and the SP
runs fine on those 2, no workarounds needed. Execution plans are identical on all boxes. When I run a profiler on the 2 different boxes however, I see that on the server having issues, the Reads are around 8087859 while the other server with identical HW,
the reads are 10608. Not quite sure how to interpret those results or where to look for answers. When the sql server service is restarted on the server having issues and the sp is run, it runs fine for a time ( not sure how long ) and then goes back to its
snail pace speed. Here is the profile trace:
Here is the stored procedure. The only modifications I made were the local variables to eliminate the obvious parameter sniffing issues and I added the WITH NOLOCK hints:
/****** Object:  StoredProcedure [dbo].[EC_EMP_APPR_SIGNATURE_SEL_TEST]    Script Date: 12/03/2014 08:06:01 ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
/****** Object:  StoredProcedure [dbo].[EC_EMP_APPR_SIGNATURE_SEL_TEST]    Script Date: 12/02/2014 22:24:45 ******/
ALTER PROCEDURE [dbo].[EC_EMP_APPR_SIGNATURE_SEL_TEST]
@EMPLOYEE_ID varchar(9) ,
@OVERRIDE_JOB_CODE varchar(8),
@OVERRIDE_LOCATION_CODE varchar(8),
@OVERRIDE_EST_CODE varchar(8),
@OVERRIDE_EMP_GROUP_CODE varchar(8)
AS
set NOCOUNT ON
declare
@EMPLOYEE_ID_LOCAL varchar(9),
@OVERRIDE_JOB_CODE_LOCAL varchar(8),
@OVERRIDE_LOCATION_CODE_LOCAL varchar(8),
@OVERRIDE_EST_CODE_LOCAL varchar(8),
@OVERRIDE_EMP_GROUP_CODE_LOCAL varchar(8)
set @EMPLOYEE_ID_LOCAL = @EMPLOYEE_ID
set @OVERRIDE_JOB_CODE_LOCAL = @OVERRIDE_JOB_CODE
set @OVERRIDE_LOCATION_CODE_LOCAL = @OVERRIDE_LOCATION_CODE
set @OVERRIDE_EST_CODE_LOCAL = @OVERRIDE_EST_CODE
set @OVERRIDE_EMP_GROUP_CODE_LOCAL = @OVERRIDE_EMP_GROUP_CODE
select 3 as SIGNATURE_ID
from EC_EMPLOYEE_POSITIONS p
where p.EMPLOYEE_ID = @EMPLOYEE_ID_LOCAL
 and DATEADD(dd, 0, DATEDIFF(dd, 0, GETDATE())) between p.POSITION_START_DATE and ISNULL(p.POSITION_END_DATE, convert(datetime, '47121231', 112))
 and (dbo.lpad(isnull(p.job_code,''), 8, ' ') + dbo.lpad(isnull(p.location_code,''), 8, ' ') + dbo.lpad(isnull(p.ESTABLISHMENT_CODE,'<N/A>'), 8, ' ') + dbo.lpad(isnull(p.EMP_GROUP_CODE,''), 8, ' '))
 in
 (select (dbo.lpad(isnull(ep.REPORTING_JOB_CODE,''), 8, ' ') + dbo.lpad(isnull(ep.REPORTING_LOCATION_CODE,''), 8, ' ') + dbo.lpad(isnull(ep.REPORTING_ESTABLISHMENT_CODE,'<N/A>'), 8, ' ') + dbo.lpad(isnull(ep.REPORTING_EMP_GROUP_CODE,''),
8, ' '))
  from EC_POSITIONS  ep
  where DATEADD(dd, 0, DATEDIFF(dd, 0, GETDATE())) between ep.POSITION_EFFECT_DATE
   and ISNULL(ep.POSITION_TERM_DATE, convert(datetime, '47121231', 112))
   and (dbo.lpad(isnull(ep.job_code,''), 8, ' ') + dbo.lpad(isnull(ep.location_code,''), 8, ' ') + dbo.lpad(isnull(ep.ESTABLISHMENT_CODE,'<N/A>'), 8, ' ') + dbo.lpad(isnull(ep.EMP_GROUP_CODE,''), 8, ' '))
   = (dbo.lpad(isnull(@OVERRIDE_JOB_CODE_LOCAL,''), 8, ' ') + dbo.lpad(isnull(@OVERRIDE_LOCATION_CODE_LOCAL,''), 8, ' ') + dbo.lpad(isnull(@OVERRIDE_EST_CODE_LOCAL,'<N/A>'), 8, ' ') + dbo.lpad(isnull(@OVERRIDE_EMP_GROUP_CODE_LOCAL,''),
8, ' ')))
union
select 4 as SIGNATURE_ID
from EC_EMPLOYEE_POSITIONS p
where p.EMPLOYEE_ID = @EMPLOYEE_ID_LOCAL
 and DATEADD(dd, 0, DATEDIFF(dd, 0, GETDATE())) between p.POSITION_START_DATE and ISNULL(p.POSITION_END_DATE, convert(datetime, '47121231', 112))
 and (dbo.lpad(isnull(p.job_code,''), 8, ' ') + dbo.lpad(isnull(p.location_code,''), 8, ' ') + dbo.lpad(isnull(p.ESTABLISHMENT_CODE,'<N/A>'), 8, ' ') + dbo.lpad(isnull(p.EMP_GROUP_CODE,''), 8, ' '))
 in
 (select (dbo.lpad(isnull(ep.REPORTING_JOB_CODE,''), 8, ' ') + dbo.lpad(isnull(ep.REPORTING_LOCATION_CODE,''), 8, ' ') + dbo.lpad(isnull(ep.REPORTING_ESTABLISHMENT_CODE,'<N/A>'), 8, ' ') + dbo.lpad(isnull(ep.REPORTING_EMP_GROUP_CODE,''),
8, ' '))
  from EC_POSITIONS  ep with (NOLOCK)
  where DATEADD(dd, 0, DATEDIFF(dd, 0, GETDATE())) between ep.POSITION_EFFECT_DATE
   and ISNULL(ep.POSITION_TERM_DATE, convert(datetime, '47121231', 112))
   and (dbo.lpad(isnull(ep.job_code,''), 8, ' ') + dbo.lpad(isnull(ep.location_code,''), 8, ' ') + dbo.lpad(isnull(ep.ESTABLISHMENT_CODE,'<N/A>'), 8, ' ') + dbo.lpad(isnull(ep.EMP_GROUP_CODE,''), 8, ' '))
   in   
 (select (dbo.lpad(isnull(ep.REPORTING_JOB_CODE,''), 8, ' ') + dbo.lpad(isnull(ep.REPORTING_LOCATION_CODE,''), 8, ' ') + dbo.lpad(isnull(ep.REPORTING_ESTABLISHMENT_CODE,'<N/A>'), 8, ' ') + dbo.lpad(isnull(ep.REPORTING_EMP_GROUP_CODE,''),
8, ' '))
  from EC_POSITIONS ep  with (NOLOCK)
  where DATEADD(dd, 0, DATEDIFF(dd, 0, GETDATE())) between ep.POSITION_EFFECT_DATE
   and ISNULL(ep.POSITION_TERM_DATE, convert(datetime, '47121231', 112))
   and (dbo.lpad(isnull(ep.job_code,''), 8, ' ') + dbo.lpad(isnull(ep.location_code,''), 8, ' ') + dbo.lpad(isnull(ep.ESTABLISHMENT_CODE,'<N/A>'), 8, ' ') + dbo.lpad(isnull(ep.EMP_GROUP_CODE,''), 8, ' '))
   = (dbo.lpad(isnull(@OVERRIDE_JOB_CODE_LOCAL,''), 8, ' ') + dbo.lpad(isnull(@OVERRIDE_LOCATION_CODE_LOCAL,''), 8, ' ') + dbo.lpad(isnull(@OVERRIDE_EST_CODE_LOCAL,'<N/A>'), 8, ' ') + dbo.lpad(isnull(@OVERRIDE_EMP_GROUP_CODE_lOCAL,''),
8, ' '))))
order by SIGNATURE_ID
Any suggestions would be greatly appreciated by anyone. I have no more tricks left in my toolbox and am starting to think its either the hardware or the database engine itself that is at fault.
Max

Answer 1

fast server:
SQL Server parse and compile time:
   CPU time = 0 ms, elapsed time = 0 ms.
SQL Server Execution Times:
CPU time = 0 ms, elapsed time = 0 ms.
SQL Server Execution Times:
   CPU time = 0 ms,  elapsed time = 0 ms.
SQL Server Execution Times:
   CPU time = 0 ms,  elapsed time = 0 ms.
SQL Server Execution Times:
   CPU time = 0 ms,  elapsed time = 0 ms.
SQL Server Execution Times:
   CPU time = 0 ms,  elapsed time = 0 ms.
SQL Server Execution Times:
   CPU time = 0 ms,  elapsed time = 0 ms.
Table 'EC_POSITIONS'. Scan count 3, logical reads 10450, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'EC_EMPLOYEE_POSITIONS'. Scan count 2, logical reads 6, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
SQL Server Execution Times:
   CPU time = 3343 ms,  elapsed time = 3460 ms.
SQL Server Execution Times:
   CPU time = 3343 ms,  elapsed time = 3460 ms.
Slow server:
SQL Server parse and compile time:
   CPU time = 0 ms, elapsed time = 0 ms.
SQL Server Execution Times:
CPU time = 0 ms, elapsed time = 0 ms.
SQL Server Execution Times:
   CPU time = 0 ms,  elapsed time = 0 ms.
SQL Server Execution Times:
   CPU time = 0 ms,  elapsed time = 0 ms.
SQL Server Execution Times:
   CPU time = 0 ms,  elapsed time = 0 ms.
SQL Server Execution Times:
   CPU time = 0 ms,  elapsed time = 0 ms.
SQL Server Execution Times:
   CPU time = 0 ms,  elapsed time = 0 ms.
Table 'EC_POSITIONS'. Scan count 3, logical reads 10450, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'EC_EMPLOYEE_POSITIONS'. Scan count 2, logical reads 6, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
SQL Server Execution Times:
   CPU time = 37875 ms,  elapsed time = 38295 ms.
SQL Server Execution Times:
   CPU time = 37875 ms,  elapsed time = 38295 ms.
Big diff in server execution times....