Help diagnosing a sporadic long running proc

Question

We have a nightly ETLV job that our 30 clients run every night. Depending on the size of the client, it can run from 1 to 7 hours. We have one client, clientX, where the etlv job takes about 7 hours. But about 3 times a month, it takes 20 hours. Looking
at the logs, it is always the same proc, procx, that is the issue. Normally procx runs at clientx for 35 minutes. But these 3 times a month it runs 838, 839, 828, 835, or 825 minutes. And then procx finishes successfully! There does NOT seem to be a pattern
on day of week, time of day procx starts, time of day procx finishes, amount of data, etc. But it is so strange that if it does not finish in 35 or so minutes, it finishes in 832 +- 7 minutes.
 Yesterday when the etlv at clientx was running over by several hours, i could tell by our log that it was in procx. I did a sp_who2 and there was nothing in the BlkBy column for any SPID. Unfortunately I am not good with diagnosing performance problems
and do not remember exactly what sp_who2 said about procx. Know that it was not taking any CPU. Believe that the Status was Suspended. The Command said Update.
 What can I do so that in the future so that I can better diagnose the issue? I know that it is procx which has 3 insert and 10 update statements to the same db. I do not want to do something that will impact the entire etlv job, which is already pushing
its 7 hour limit.
I want some diagnosis steps that I can run if I am called while procx is running. And I would like some diagnosis steps that I can setup in advance because many times I am just told that "oh, yesterday we had that issue again.'
 Thanks

Answer 1

We have a nightly ETLV job that our 30 clients run every night. Depending on the size of the client, it can run from 1 to 7 hours. We have one client, clientX, where the etlv job takes about 7 hours. But about 3 times a month, it takes 20 hours. Looking
at the logs, it is always the same proc, procx, that is the issue. Normally procx runs at clientx for 35 minutes. But these 3 times a month it runs 838, 839, 828, 835, or 825 minutes. And then procx finishes successfully! There does NOT seem to be a pattern
on day of week, time of day procx starts, time of day procx finishes, amount of data, etc. But it is so strange that if it does not finish in 35 or so minutes, it finishes in 832 +- 7 minutes.
 Yesterday when the etlv at clientx was running over by several hours, i could tell by our log that it was in procx. I did a sp_who2 and there was nothing in the BlkBy column for any SPID. Unfortunately I am not good with diagnosing performance problems
and do not remember exactly what sp_who2 said about procx. Know that it was not taking any CPU. Believe that the Status was Suspended. The Command said Update.
 What can I do so that in the future so that I can better diagnose the issue?
I know that it is procx which has 3 insert and 10 update statements to the same db. I do not want to do something that will impact the entire etlv job, which is already pushing its 7 hour limit.
I want some diagnosis steps that I can run if I am called while procx is running. And I would like some diagnosis steps that I can setup in advance because many times I am just told that "oh, yesterday we had that issue again.'
 Thanks
>> I know that it is procx which has 3 insert and 10 update statements to the same db. I do not want to do something that will impact the entire etlv job, which is already
pushing its 7 hour limit.
Right now you are asking about monitoring, this is a great question, but remember that there is an option that we could help you in improving your proc as well, and maybe change this 7 hours to 7 minute (This is not just numbers, but improvements that I
have seen a lot, and even one time 1 hour execution improved to 4 second), on the same time maybe we can't :-)
This is a different question, but If you want us to take a look on your basic issue, then we need more information, which will let us reproduce your issue.
>> I want some diagnosis steps that I can run if I am called while procx is running. And I would like some diagnosis steps that I can setup in advance because many times I
am just told that "oh, yesterday we had that issue again.'
in addition to other great responses
Performance Dashboard Reports
* in SQL Server 2014 we have a new
Dynamic Management View to track what a query is actually doing at a physical operator level, like the sys.dm_exec_query_profiles, which is monitors real time query
progress while the query is in execution.
* Setting SET STATISTICS PROFILE or SET STATISTICS XML to ON into the Query
* Create an Extended Events Session and add sqlserver.query_post_execution_showplan as a target event.
*** Check those link as well:
http://www.sqlshack.com/performance-dashboard-reports-sql-server-2014/
http://www.mssqltips.com/sqlservertip/1949/sql-server-sysdmoswaitstats-dmv-queries/
  Ronen Ariely
 [Personal Site]    [Blog]    [Facebook]