SSIS Data Flow Question

Question

While working on my current project, a problem came to mind.
On past projects in SSIS, I was taking data from an Excel or csv source and converting for use in SSMS. Straightforward and fairly simple, as I became more and more acquainted with SSIS.
But, the stored procedure I'm working on now raised a question on how data flows through a project.
The procedure updates a couple of tables using SQL's UPDATE command. So, it does a direct modification of a table.
However, in duplicating that in SSIS - using Derived Column tools - I started to wonder what the data was doing as the flow progressed through the various steps in the Control and Data flows.
Here's my Control Flow so far;
In Step 1, the SQL table, sh1, has all records that meet the WHERE criteria of the embedded SQL code deleted, but leaving the rest of the table intact.
In Step 2, I define a variable for use in the project using an embedded SQL query.
In Step 3, I encounter the first update task;
This flow terminates in the lower right corner with Update sh1, Pass 4. Here is where I have my question.
When this flow terminates, does it return to the Control Flow with the table sh1 updated as required by the
Update sh1 Data Flow? In other words, does it enter Control Flow Step 4 with the updates, or does the next update task (one in which the stored procedure uses the modified table, sh1) simply reload sh1 from the server, without any updates?
If the former, then no worries, as that's how I was proceeding until a little bug in my ear said, "Whoa, buckwheat! Maybe the changes aren't stored."
If the latter, then I'm guessing that I need to have a Data Flow Destination tool dragged to the workspace and connected up to the output of the
Update sh1, Pass 4 tool.
Fine, I thought, I'll direct the output to the sh1 table.
But, that will put all of sh1 into the table, not just the updated rows. I'd have to TRUNCATE sh1 first. But, sh1 is active in the Data Flow. If I truncate it, there'll be nothing to update. Or, if I truncate it after the
Update sh1, Pass 4 tool, it'll lose all the updates - and everything else.
I figured that, in the latter case, I'd have to have some sort of interim table to hold the results of the
Update sh1 Data Flow, truncate sh1 and then put the contents of the interim table into the, now empty, sh1 before returning control back to the Control Flow.
I'd like to avoid an interim table, if at all possible, but if I have to have one, I have to.
Or, is there another way?
Thanx in advance for any help!

Answer 1

You can always use a Record-Set Destination to hold the results from your first Data Flow Transformation. The record set variable is retained in memory and available within the context of your package
Object Variables Recordsets
But then you need to modify  your package to read from the record set. you can either use a script task , or loop through each record in your record set and process the updates for each row
Shredding Record Set
While these are possible options, i do not think they are the best way to go about things.
I would like to know what you are doing in the Update Sh2 Step. You can keep adding data flows that follows Update sh1 Pass 4 and then finally update the table through an OLE DB Command transformation. then you do not have to store anything in memory. but
you might probably end up with a complex data flow transformation.
The design will depend on the volume of records, the number of transformations you want the data to go through.
You can avoid transformations by writing queries that retrieve results in the way you want. Example Sorting the results by adding an Order by in your Source query, as opposed to adding a sort transformation
Regards, Dinesh