Update Row into Run Table Task is not executing in correct sequence in DAC

Question

Update Row into Run Table Task is not executing in correct sequence in DAC.
The task phase for this task is "Post Lost" . The depth in the execution plan is 19 but this task is running some times in Depth 12, some times in 14 and some time in Depth 16. Would like to know is this sequence of execution is correct order or not? In the out of the Box this task is executed at the end of the entire load. No Errors were reported in DAC log.
Please let me know if any documents that would highlight this issue
rm

Answer 1

Update into Run table is a task thats required to update a table called W_ETL_RUN_S. The whole intention of this table is to keep the poor mans run history on the warehouse itself. The actual run history is stored in the DAC runtime tables, however the DAC repository could be on some other database/schema other than warehouse. Its mostly a legacy table, thats being carried around. If one were to pay close attention to this task, it has phase dependencies defined that dictate when this task should run.
Apologies in advance for a lengthy post.... But sure might help understanding how DAC behaves! And is going to be essential for you to find issues at hand.
The dependency generation in DAC follows the following rules of thumb!
- Considers the Source table target table definitions of the tasks. With this information the tasks that write to a table take precedence over the tasks that reads from a table.
- Considers the phase information. With this information, it will be able to resolve some of the conflicts. Should multiple tasks write to the same table, the phase is used to appropriately stagger them.
- Considers the truncate table option. Should there be multiple tasks that write to the same table with the same phase information, the task that truncates the table takes precedence.
- When more than one task that needs to write to the table that have similar properties, DAC would stagger them. However if one feels that either they can all go in parallel, or a common truncate is desired prior to any of the tasks execution, one could use a task group.
- Task group is also handy when you suspect the application logic dictates cyclical reads and writes. For example, Task 1 reads from A and writes to B. Task 2 reads from B and writes back to A. If these two tasks were to have different phases, DAC would be able to figure that out and order them accordingly. If not, for example those tasks need to be of the same phase for some reason, one could create a task group as well.
Now that I described the behavior of how the dependency generation works, there may be some tasks that have no relevance to other tasks either as source tables or target tables. The update into run history is a classic example. The purpose of this task is to update the run information in the W_ETL_RUN_S with status 'Completed' with an end time stamp. Because this needs to run at the end, it has phase dependency defined on it. With this information DAC will be able to stagger the position of execution either before (Block) or after (Wait) all the tasks belonging to a particular phase is completed.
Now a description about depth. While Depth gives an indication to the order of execution, its only an indication of how the tasks may be executed. Its a reflection of how the dependencies have been discovered. Let me explain with an example. The tasks that have no dependency will have a depth of 0. The tasks that depend on one or more or all of depth 0 get a depth of 1. The tasks that depend on one or more or all of depth 1 get a depth of 2. It also means implicitly a task of depth 2 will indirectly depend on a task of depth 0 through other tasks in depth 1. In essence the dependencies translate to an execution graph, and is different from the batch structures one usually thinks of when it comes to ETL execution.
Because DAC does runtime optimization in the order in which tasks are executed, it may pick a task thats of order 1 over something else with an order of 0. The factors considered for picking the next best task to run depend on
- The number of dependent tasks. For example, a task which has 10 dependents gets more priorty than the one whose dependents is 1.
- If all else equal, it considers the number of source tables. For example a task having 10 source tables gets more priority than the one that has only two source tables.
- If all else equal, it considers the average time taken by each of the tasks. The longer running ones will get more preference than the quick running ones
- and many other factors!
And of course the dependencies are honored through the execution. Unless all the predecessors of a task are in completed state a task does not get picked for execution.
Another way to think of this depth concept : If one were to execute one task at a time, probably this is the order in which the tasks will be executed.
The depth can change depending on the number of tasks identified for the execution plan.
The immediate predecessors and successor can be a very valuable information to look at and should be used to validate the design. All predecessors and successors provide information to corroborate it even further. This can be accessed through clicking on any task and choosing the detail button. You will see all these information over there. As an alternate method, you could also use the 'All/immediate Predecessors' and 'All/immediate Successor' tabs that provide a flat view of the dependencies. Note that these tabs may have to retrieve a large amount of data, and hence will open in a query mode.
SUMMARY: Irrespective of the depth, validate
- if this task has 'Phase dependencies' that span all the ETL phases and has a 'Wait' option.
- click on the particular task and verify if the task does not have any successors. And the predecessors include all the tasks from all the phases its supposed to wait for!
Once you have inspected the above two you should be good to go, no matter what the depth says!
Hope this helps!