Populate a dataset in a foreach loop

Hi,
I am trying to populate a dataset in foreach loop, but I have difficulties getting it working.
I have a set of plan_handles in a dataset. Now I want to go through them and execute a query to get the details about the plans. The result should get into a dataset (that I later will SqlBulkCopy into a database).
I have this code. I can see that the queries are executed against the server, but when I look at $Plans.Tables[0] it only contains one row.
foreach ($Plan in $PlanHandles.Tables[0])
$PlanHandle = $Plan["plan_handle"]
$SqlSourceConnection.Open()
$SqlCmdPlanGet = New-Object System.Data.SqlClient.SqlCommand("SELECT GETDATE() AS [timestamp], @plan_handle AS plan_handle, [dbid], objectid, number, encrypted, query_plan FROM sys.dm_exec_query_plan(@plan_handle)",$SqlSourceConnection)
[void]$SqlCmdPlanGet.Parameters.Add("@plan_handle",$PlanHandle)
$SqlCmdPlanGet.CommandTimeout = 30
$SqlAdapterPlanGet = New-Object System.Data.SqlClient.SqlDataAdapter($SqlCmdPlanGet)
$Plans = New-Object System.Data.DataSet
$SqlAdapterPlanGet.Fill($Plans)
$SqlSourceConnection.Close()
I have been reading about it and I understand, that it is something with that I should specify the datatable, but I am not sure how to get the code right.
Ola Hallengren
http://ola.hallengren.com

Hi,
What is the below code outpting:
$PlanHandles.Tables[0]
we may need to run each line and found out whether proper value was set to proper variable.
I also suggest we post in the official scripting guys forum for further analysis, as I am not familar with SQL:
https://social.technet.microsoft.com/Forums/scriptcenter/en-US/home?forum=ITCG
Regards,
Yan Li
Regards, Yan Li

Similar Messages

XML Node Selection in Foreach Loop

Good Morning,
I am just building up my first SSIS-package which uses a web service as data source. Now, that I managed the data to be processed using a Foreach Loop, I am wondering how the loop reacts when the sequence of teh nodes changes, or a certain node is not present.
In the "Foreach Loop Editor - Variable Mappings" I have mapped the variables (as they are at the moment) to the certain index (as they appear at the moment), but again: what if an additional nodes comes in? Isn't there a way to explicitly tell the
system to set variable ABC with node ABC, DEF with node DEF and so on...

Good Morning,
am copying in the reply, I receive from the web service which is then stored in the variable:
<?xml version=\"1.0\" encoding=\"utf-16\"?>\r\n
<DataSet>\r\n
<xs:schema id=\"NewDataSet\" xmlns=\"\" xmlns:xs=\"http://www.w3.org/2001/XMLSchema\" xmlns:msdata=\"urn:schemas-microsoft-com:xml-msdata\">\r\n
<xs:element name=\"NewDataSet\" msdata:IsDataSet=\"true\" msdata:UseCurrentLocale=\"true\">\r\n
<xs:complexType>\r\n
<xs:choice minOccurs=\"0\" maxOccurs=\"unbounded\">\r\n
<xs:element name=\"FLT\">\r\n
<xs:complexType>\r\n
<xs:sequence>\r\n
<xs:element name=\"CARRIER\" type=\"xs:string\" minOccurs=\"0\" />\r\n
<xs:element name=\"FLIGHT\" type=\"xs:string\" minOccurs=\"0\" />\r\n
<xs:element name=\"DEP\" type=\"xs:string\" minOccurs=\"0\" />\r\n
<xs:element name=\"DES\" type=\"xs:string\" minOccurs=\"0\" />\r\n
<xs:element name=\"DATE_LOC\" type=\"xs:string\" minOccurs=\"0\" />\r\n
<xs:element name=\"ETD_LOC\" type=\"xs:string\" minOccurs=\"0\" />\r\n
<xs:element name=\"ETA_LOC\" type=\"xs:string\" minOccurs=\"0\" />\r\n
<xs:element name=\"DATE_UTC\" type=\"xs:string\" minOccurs=\"0\" />\r\n
<xs:element name=\"ETD_UTC\" type=\"xs:string\" minOccurs=\"0\" />\r\n
<xs:element name=\"ETA_UTC\" type=\"xs:string\" minOccurs=\"0\" />\r\n
<xs:element name=\"PAX_C\" type=\"xs:short\" minOccurs=\"0\" />\r\n
<xs:element name=\"PAX_Y\" type=\"xs:short\" minOccurs=\"0\" />\r\n
<xs:element name=\"SEAT_C\" type=\"xs:short\" minOccurs=\"0\" />\r\n
<xs:element name=\"SEAT_Y\" type=\"xs:short\" minOccurs=\"0\" />\r\n
<xs:element name=\"PAX_EX\" type=\"xs:short\" default=\"0\" minOccurs=\"0\" />\r\n
<xs:element name=\"PAX_CHD_C\" type=\"xs:short\" default=\"0\" minOccurs=\"0\" />\r\n
<xs:element name=\"PAX_CHD_Y\" type=\"xs:short\" default=\"0\" minOccurs=\"0\" />\r\n
<xs:element name=\"PAX_INF\" type=\"xs:short\" default=\"0\" minOccurs=\"0\" />\r\n
</xs:sequence>\r\n
</xs:complexType>\r\n
</xs:element>\r\n
<xs:element name=\"FLTRem\">\r\n
<xs:complexType>\r\n
<xs:sequence>\r\n
<xs:element name=\"CARRIER\" type=\"xs:string\" minOccurs=\"0\" />\r\n
<xs:element name=\"FLIGHT\" type=\"xs:string\" minOccurs=\"0\" />\r\n
<xs:element name=\"DEP\" type=\"xs:string\" minOccurs=\"0\" />\r\n
<xs:element name=\"DES\" type=\"xs:string\" minOccurs=\"0\" />\r\n
<xs:element name=\"DATE_UTC\" type=\"xs:string\" minOccurs=\"0\" />\r\n
</xs:sequence>\r\n
</xs:complexType>\r\n
</xs:element>\r\n
</xs:choice>\r\n
</xs:complexType>\r\n
</xs:element>\r\n
</xs:schema>\r\n
<diffgr:diffgram xmlns:msdata=\"urn:schemas-microsoft-com:xml-msdata\" xmlns:diffgr=\"urn:schemas-microsoft-com:xml-diffgram-v1\">\r\n
<NewDataSet>\r\n
<FLT diffgr:id=\"FLT1\" msdata:rowOrder=\"0\">\r\n
<CARRIER>XY</CARRIER>\r\n
<FLIGHT>1234</FLIGHT>\r\n
<DEP>ABC</DEP>\r\n
<DES>DEF</DES>\r\n
<DATE_LOC>11.03.2014 00:00:00</DATE_LOC>\r\n
<ETD_LOC>30.12.1899 08:30:00</ETD_LOC>\r\n
<ETA_LOC>30.12.1899 12:15:00</ETA_LOC>\r\n
<DATE_UTC>11.03.2014 00:00:00</DATE_UTC>\r\n
<ETD_UTC>30.12.1899 07:30:00</ETD_UTC>\r\n
<ETA_UTC>30.12.1899 12:15:00</ETA_UTC>\r\n
<PAX_C>0</PAX_C>\r\n
<PAX_Y>120</PAX_Y>\r\n
<SEAT_C>0</SEAT_C>\r\n
<SEAT_Y>210</SEAT_Y>\r\n
<PAX_EX>0</PAX_EX>\r\n
<PAX_CHD_C>0</PAX_CHD_C>\r\n
<PAX_CHD_Y>0</PAX_CHD_Y>\r\n
<PAX_INF>0</PAX_INF>\r\n
</FLT>\r\n
<FLT diffgr:id=\"FLT2\" msdata:rowOrder=\"1\">\r\n
<CARRIER>YZ</CARRIER>\r\n
<FLIGHT>4567</FLIGHT>\r\n
<DEP>DEF</DEP>\r\n
<DES>GHI</DES>\r\n
<DATE_LOC>11.03.2014 00:00:00</DATE_LOC>\r\n
<ETD_LOC>30.12.1899 13:10:00</ETD_LOC>\r\n
<ETA_LOC>30.12.1899 18:35:00</ETA_LOC>\r\n
<DATE_UTC>11.03.2014 00:00:00</DATE_UTC>\r\n
<ETD_UTC>30.12.1899 13:10:00</ETD_UTC>\r\n
<ETA_UTC>30.12.1899 17:35:00</ETA_UTC>\r\n
<PAX_C>0</PAX_C>\r\n
<PAX_Y>203</PAX_Y>\r\n
<SEAT_C>0</SEAT_C>\r\n
<SEAT_Y>210</SEAT_Y>\r\n
<PAX_EX>0</PAX_EX>\r\n
<PAX_CHD_C>0</PAX_CHD_C>\r\n
<PAX_CHD_Y>0</PAX_CHD_Y>\r\n
<PAX_INF>0</PAX_INF>\r\n
</FLT>\r\n
</NewDataSet>\r\n
</diffgr:diffgram>\r\n
</DataSet>

Yet another Try Catch question. Iterating through a ForEach loop

Confused on error handling in a Powershell ForEach loop. I’m looping through a list of registry keys, attempting
to open each one. If it succeeds, I do a bunch of stuff. If it fails, I want to skip to the next iteration.
If I was doing It in VBScript I’d do this:
For Each Thing In colThings
Open Thing
If Err.Number <> 0 Then
“oops”
Else
Do stuff
Do stuff
Do stuff
End If
Next
This is what I came up with in PowerShell. It seems to work, but just doesn’t seem powershell-ish. There must be a better way to use the catch output than just creating a $return variable and assigning it success or fail?
ForEach ($subKeyName in $subKeyNames)
try{$subKey = $baseKey.OpenSubKey("$subKeyName")}
catch{$return = "error" }
If($return -eq "error" )
“Oops”
Else
Do stuff
Do stuff
Do Stuff

I totally get what you're saying about formatting. I don't' have any habits yet, since I've only been working in Powershell since... well, what time is it now?
Unfortunately, It Has Been Decreed that we are no longer to use VBScript for any engineering solutions at work, so my 15 years experience in it now needs to be transitioned over asap. I don't have the luxury of crawling before I run. I'm trying not to be
frustrated, but it's like an English major waking up one day and being told "You must now speak French exclusively. Here's a book."
The Do Stuff example of my ForEach loop is about 50 lines of code involving matching values in subkeys of this registry key with another and collecting output. I tried wrapping the whole thing in a try section based on some examples, but it seemed odd, that's
why I'm asking. I'm used to tightly focused error handling at the point where an error may occur.
In this example I'm only interested in whether or not I can open the subkey (it exists, but I may not have permission). If I can't, there's no point in continuing with this iteration of the loop, I want to skip to the next one. So why include all the "Do
Stuff" in the the try section? From a readability viewpoint, it doesn't seem helpful.
Also, there may be more error handling deeper in the code. If I then put that in a try/catch, and then something else inside that, now I have nested try/catches mixed in with nested if/elses, all wrapped in a For loop.
Again, I can see how it works logically, but for readability not so much, and having all these braces 50 lines apart to match up is giving me eye strain :).
It sounds like David is agreeing with jrv, that putting the entire ForEach loop code into a try/catch is the conventional way to do it. I guess it makes as much sense as putting it all in an If-else-Endif, and I just need to adjust my paradigm.
But if not, my specific question was more along the lines of, is there a built in way to tell that the catch section has been executed, rather than me using it to populate an arbitrary variable and then read it? In VBScript, you execute something, and the
next line, you check the Err.number. I wasn't sure if you could do that with a try/catch.

ForEach Loop Iteration Order

Hi,
I have a SSIS package that uses a Foreach Loop Container to iterate through a series of flat files. The order in which the files are iterated through is based on the name of the file. Is it possible to change the iteration order? More specifically, can the
iteration order be changed to the date when the file was placed in the iteration folder? i.e files that have been sitting in a folder for a longer amount of time get processed first?

you can change order but for that you need to use a control table based approach. Using file enumerator will always take order based on filename.
The approach would be like this. Have a for each loop based on file enumerator first. Take the filename and arrival date (this you can get inside a script task see
http://jamelcato.com/111/how-to-check-last-modified-date-using-ssis/) and populate the table with the information.
Have a object variable and set query as
SELECT fileName
FROM table
ORDER BY DateField
Then have a for each loop to iterate through object variable use ADO enumerator
http://www.select-sql.com/mssql/loop-through-ado-recordset-in-ssis.html
This would make sure it iterates based on file date value. Then have a variable inside to retrieve filename each time and use it for whatever processing you want.
Please Mark This As Answer if it helps to solve the issue Visakh ---------------------------- http://visakhm.blogspot.com/ https://www.facebook.com/VmBlogs

Foreach Loop only looping once

Hi,
Please be patient and bear with me, for I am new to SSIS packages.
I am using SQL Server 2008 R2 to host the SQL Agent job and the SSIS package.
I am using Visual Studio 2008 Professional Edition for designing the SSIS package. It is version 9.0.30729, Version 3.5 SP1.
I am on a Windows 8 machine.
My Foreach Loop Container seems to iterate only once when it should iterate numerous times, and this issue happens BOTH when I execute the SSIS package from within Visual Studio AND when I run a SQL Agent job to execute the SSIS package.
For the solution within Visual Studio, I have False for the property Run64BitRuntime.
In the SSIS pckg in Visual Studio, I have in the Variables pane a number of variables and these following variables pertain to my problem:
Name: objRs     Scope: the name of the SSIS package     DataType: Object    Value: System.Object
Name: txtVendorFamilyNumber   Scope: the name of the SSIS package DataType: String   Value: 111111
The value "111111" that I entered into the txtVendorFamilyNumber variable in the Variable Pane is NOT a realistic value as there are no vendor family numbers that have the value 111111. I will populate dynamically the real vendor family numbers
from within the SSIS package.
When I imported the SSIS package to SQL Server 2008 R2, via the ellipsis button for the Protection Level I chose the "Rely on server storage and roles for access control" choice.
Concerning the flow of the SSIS package, going down within Visual Studio from the beginning of the flow of the SSIS package, eventually I have an Execute SQL Task. In its General tab, it has "1252" for CodePage property, "Full result set"
for ResultSet property, "OLE DB" for ConnectionType property, "Direct input" for SQL Source Type property, "False" for BypassPrepare property, and the following for the SQLStatement property:
     select txtVendorFamilyNumber from tmpVendorFamily where VendorID = (select min(VendorID) from tmpVendorFamily)
In the tmpVendorFamily table on SQL Server 2008 R2, there are 379 rows of results for the SELECT statement above (starting with the values 100110, 100115, 100120, 100130, through the LAST row's value 106370.
In the Parameter Mapping tab of the Execute SQL Task Editor, I have nothing. In the Expressions tab of the Execute SQL Task Editor, I have nothing.
In the Result Set tab of the Execute SQL Task Editor, I have the value 0 for Result Name property and I have the value User::objRs as the value in the Variable Name property.
This Execute SQL Task has a green arrow (Constraint evaluation operation, Success in Value drop down) to the Foreach Loop Container.
In the Foreach Loop Editor, in the Collection tab, I have "Foreach ADO Enumerator" as the value in the Enumerator property. In the drop down for ADO Object Source Variable, I have chosen the User::objRs, and I have the radio button "Rows in
the first table" selected.
In the Foreach Loop Editor, in the Expressions tab, I have nothing.
In the Foreach Loop Editor, in the Variable Mappings tab, I have "User::txtVendorFamilyNumber" for the Variable property and I have the value "0" for the Index property.
So in the Foreach Loop, I first have a Script Task with the ReadOnly variable User::txtVendorFamilyNumber and with the ReadWrite variable User::TESTING_OnlyTesting that I created with a package scope, string datatype, and no default value. This Script Task
has the following code for testing purposes:
Dim strFullVerbiage As String
strFullVerbiage = "The value is: "
Dts.Variables("User::TESTING_OnlyTesting").Value = strFullVerbiage + Dts.Variables("User::txtVendorFamilyNumber").Value.ToString()
Dts.TaskResult = ScriptResults.Success
I have next in the Foreach Loop Container a Send Mail task. In the Expressions tab of the Send Mail Task Editor, I have for the Message Source property the following:
      @[User::TESTING_OnlyTesting]
The remaining tasks (including FTP Tasks) in the Foreach Loop Container are disabled for now as I am trying to figure out why the Foreach Loop only iterates the FIRST time, only iterates once.
When I run on SQL Server 2008 R2 the SELECT statement from the Execute SQL Task, I get 379 rows of results. The default value 111111 that I entered into the Variable Pane for the txtVendorFamilyNumber variable does NOT exist in any of these 379 rows of results.
The 1st row of results has for the txtVendorFamilyNumber column the value 100110. Then the subsequent rows of results have the values 100115, 100120, 100130, et cetera, ending with the value 106370 in the last row of results.
I imported the SSIS package into Integration Services on SQL Server 2008 R2. When I imported to Integration Services, in the Import Package editor for the Protection Level I chose the "Rely on server storage and roles for access control" choice
because there are FTP Tasks and we needed to use this "Rely on server storage and roles for access control" choice.
I created a SQL Agent job that executes this SSIS package.
When I run the SQL Agent job, no error happens and I get only ONE email with the value 100110 for the vendor family number in it. I should get 379 emails total, the 2nd email showing 100115, the 3rd email showing 100120, et cetera.
When I was testing while executing the SSIS package from within Visual Studio, I did NOT have the Send Mail task and I only had enabled one Script Task that had the ReadOnly variable User::txtVendorFamilyNumber and the following lines of code:
MsgBox(CType(Dts.Variables("User::txtVendorFamilyNumber").Value, String))
Dts.TaskResult = ScriptResults.Success
Executing the SSIS package with only this Script Task enabled within the Foreach Loop Container, I would get the ONE message box showing 100110 but I would not get any message box for a SECOND ITERATION of the loop.
Since my txtVendorFamilyNumber in the Variables Pane has the default hard-coded value 111111 that does NOT exist in the result set of the Execute SQL Task that feeds the Foreach Loop, and since the first and only email sent shows the value 100110 when I
run the SQL Agent job that runs the SSIS package that is set up to send an email from within the Foreach Loop Container, the FIRST LOOP seems to run correctly and shows correctly the 1st value 100110 in the 379 rows of results that populate the Foreach Loop
Container. Why does the Foreach Loop Container NOT loop more than once?
Could anybody give me any suggestions as to what the problem might be?
I thank you in advance for your consideration of my problem. I have been reading and reading on the Internet to try to find a solution but I have not solved yet my problem with the Foreach loop only iterating once.
Thanks much to you.

Gregarious greetings Arthur,
I very much appreciate your reply.
This morning I figured out my problem with the help of a fellow employee.
In my original post higher up, I had typed about the SELECT statement in the Execute SQL Task that is just before the Foreach Loop Container and that populates the User::objRs, and this SELECT statement was in the old DTS package in an old ActiveX script
task. The SELECT statement was in an old DTS package in an old ActiveX script task and I am creating a new SSIS package to replace the old DTS package so that we can move from the old version of SQL Server to a new SQL Server 2008 R2 server. Here is that
SELECT statement:
select txtVendorFamilyNumber from tmpVendorFamily where VendorID = (select min(VendorID) from tmpVendorFamily)
That SELECT statement above returns ONLY ONE result. I had not noticed it. That SELECT statement fills the objRs result set. The green arrow goes from the Execute SQL Task that has the SELECT statement down to the Foreach Loop Container that loops over the
result set. Since the SELECT statement that fills the result set only has one row of results, the Foreach Loop only looped once.
So for the Execute SQL Task that populates the result set User::objRs, I changed the SELECT statement to be the following:
--In the Utilities DB, this tmpVendorFamily table
--has only two columns named VendorID and txtVendorFamilyNumber.
SELECT DISTINCT txtVendorFamilyNumber
FROM Utilities.dbo.tmpVendorFamily
ORDER BY txtVendorFamilyNumber
Now the Foreach Loop will loop 379 times since the SELECT statement immediately above yields 379 rows of results.
So the Foreach Loop Container was fine. Its Collection tab has the "Foreach ADO Enumerator" for the Enumerator and has the "User::objRs" for the ADO Object Source Variable. Its Variable Mappings tab has "User::txtVendorFamilyNumber"
for the Variable and has "0" for the Index.
My problem was the SELECT statement in the Execute SQL Task that immediately precedes the Foreach Loop Container and that fills the User::objRs result set. The SELECT statement yielded only ONE row of results. When I corrected this morning that SELECT statement
so that it correctly yields 379 rows of results, the Foreach Loop Container loops 379 rows.
Thanks again for replying and trying to help. I very much appreciate it.
I consider this thread that I created to be closed since I figured out this morning how I needed to correct the SELECT statement in the Execute SQL Task that fills the result set on which the Foreach Loop loops.
Misko_Hviezdoslav   27-JAN-2015

How can I load a .xlsx File into a SQL Server Table using a Foreach Loop Container in SSIS?

I know I've REALLY struggled with this before. I just don't understand why this has to be soooooo difficult.
I can very easily do a straight Data Pump of a .xlsX File into a SQL Server Table using a normal Excel Connection and a normal Excel Source...simply converting Unicode to DT_STR and then using an OLE DB Destination of the SQL Server Table.
If I want to make the SSIS Package a little more flexible by allowing multiple .xlsX spreadsheets to be pumped in by using a Foreach Loop Container, the whole SSIS Package seems to go to hell in a hand basket. I simply do the following...
Put the Data Flow Task within the Foreach Loop Container
Add the Variable Mapping Variable User::FilePath that I defined as a Variable and a string within the FOreach Loop Container
I change the Excel Connection and its Expression to be ExcelFilePath ==> @[User::FilePath]
I then try and change the Excel Source and its Data Access Mode to Table Name or view name variable and provide the Variable Name User::FilePath
And that's when I run into trouble...
Exception from HRESULT: 0xC02020E8
Error at Data Flow Task [Excel Source [56]]:SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occured. Error code: 0x80004005.
Error at Data Flow Task [Excel Source [56]]: Opening a rowset for "...(the EXACT Path and .xlsx File Name)...". Check that the object exists in the database. (And I know it's there!!!)
I don't understand by adding a Foreach Loop Container to try and make this as efficient as possible has caused such an error unless I'm overlooking something. I have even tried delaying my validations and that doesn't seem to help.
I have looked hard in Google and even YouTube to try and find a solution for this but for the life of me I cannot seem to find anything on pumping a .xlsX file into SQL Server using a Foreach Loop Container.
Can ANYONE please help me out here? I'm at the end of my rope trying to get this to work. I think the last time I was in this quandry, trying to pump a .xlsX File into a SQL Server Table using a Foreach Loop Container in SSIS, I actually wrote a C# Script
to write the contents of the .xlsX File into a .csv File and then Actually used the .csv File to pump the data into a SQL Server Table.
Thanks for your review and am hoping and praying for a reply and solution.

Hi ITBobbyP,
If I understand correctly, you want to load data from multiple sheets in an .xlsx file into a SQL Server table.
If in this scenario, please refer to the following tips:
The Foreach Loop container should be configured as shown below:
Enumerator: Foreach ADO.NET Schema Rowset Enumerator
Connection String: The OLE DB Connection String for the excel file.
Schema: Tables.
In the Variable Mapping, map the variable to Sheet_Name, and change the Index from 0 to 2.
The connection string for Excel Connection Manager is the original one, we needn’t make any change.
Change Table Name or View name to the variable Sheet_Name.
If you want to load data from multiple sheets in multiple .xlsx files into a SQL Server table, please refer to following thread:
http://stackoverflow.com/questions/7411741/how-to-loop-through-excel-files-and-load-them-into-a-database-using-ssis-package
Thanks,
Katherine Xiong
Katherine Xiong
TechNet Community Support

SSIS BULK INSERT unsing UNC inside of ForEach Loop Container Failed could not be opened. Operating system error code 5(Access is denied.)

Hi,
I am trying to figure out how to fix my problem
Error: Could not be opened. Operating system error code 5(Access is denied.)
Process Description:
Target Database Server Reside on different Server in the Network
SSIS Package runs from a Remote Server
SSIS Package use a ForEachLoop Container to loop into a directory to do Bulk Insert
SSIS Package use variables to specified the share location of the files using UNC like this
\\server\files
Database Service accounts under the Database is runing it has full permission on the share drive were the files reside.
In the Execution Results tab shows the prepare SQL statement for the BULK insert and I can run the same exact the bulk insert in SSMS without errors, from the Database Server and from the server were SSIS package is executed.
I am on a dead end and I don’t want to re-write SSIS to use Data Flow Task because is not flexible to update when metadata of the table changed.
Below post it has almost the same situation:
https://social.msdn.microsoft.com/Forums/sqlserver/en-US/8de13e74-709a-43a5-8be2-034b764ca44f/problem-with-bulk-insert-task-in-foreach-loop?forum=sqlintegrationservices

Insteresting how I fixed the issue, Adding the Application Name into the SQL OLAP Connection String Fixed the issue. I am not sure why SQL Server wasn't able to open the file remotely without this.

[Forum FAQ] How to calculate the total count of insert rows within a Foreach Loop Container in SSIS?

Introduction
We need to loop through all the flat files that have the same structure in a folder and import all the data to a single SQL Server table. How can we obtain the total count of the rows inserted to the destination SQL Server table?
Solution
We can use Execute SQL Task or Script Task to aggregate the row count increment for each iteration of the Foreach Loop Container. The following steps are the preparations before we add the Execute SQL Task or Script Task:
Create a String type variable FilePath, two Int32 type variables InsertRowCnt and TotalRowCnt.
Drag a Foreach Loop Container to the Control Flow design surface, set the Enumerator to “Foreach File Enumerator”, specify the source folder and the files extension, and set the “Retrieve file name” option to “Fully qualified”.
On the “Variable Mappings” tab of the container, map the variable FilePath to the collection value.
Drag a Data Flow Task to the container, in the Data Flow Task, add a Flat File Source, a Row Count Transformation, and an OLE DB Destination, and join them. Create a Flat File Connection Manager to connect to one of the flat files, and then configure the
Flat File Source as well as the OLE DB Destination adapter. Set the variable for the Row Count Transformation to “User::InsertRowCnt”.
Open the Property Expressions Editor for the Flat File Connection Manager, and set the expression of “ConnectionString” property to
“@[User::FilePath]”.
(I) Execute SQL Task Method:
In the Control Flow, drag an Execute SQL Task under the Data Flow Task and join them.
Create one or using any one existing OLE DB Connection Manager for the Execute SQL Task, set the “ResultSet” option to “Single row”, and then set the “SQLStatement” property to:
DECLARE @InsertRowCnt INT,
               @TotalRowCnt INT
SET @InsertRowCnt=?
SET @TotalRowCnt=?
SET @TotalRowCnt=@InsertRowCnt+@TotalRowCnt
SELECT TotalRowCnt=@TotalRowCnt
On to parameter 1.
On the “Result Set” tab of the Execute SQL Task, map result 0 to variable “User::TotalRowCnt”.
(II) Script Task Method:
In the Control Flow, drag a Script Task under the Data Flow Task and join them.
In the Script Task, select variable InsertRowCnt for “ReadOnlyVariables” option, and select variable TotalRowCnt for “ReadWriteVariables”.
Edit the Main method as follows (C#):
public void Main()
// TODO: Add your code here
int InsertRowCnt = Convert.ToInt32(Dts.Variables["User::InsertRowCnt"].Value.ToString()
int TotalRowCnt = Convert.ToInt32(Dts.Variables["User::TotalRowCnt"].Value.ToString());
TotalRowCnt = TotalRowCnt + InsertRowCnt;
Dts.Variables["User::InsertRowCnt"].Value = TotalRowCnt;
Dts.TaskResult = (int)ScriptResults.Success;
          Or (VB)
          Public Sub Main()
        ' Add your code here
        Dim InsertRowCnt As Integer =
        Convert.ToInt32(Dts.Variables("User::InsertRowCnt").Value.ToString())
        Dim TotalRowCnt As Integer =
        Convert.ToInt32(Dts.Variables("User::TotalRowCnt").Value.ToString())
        TotalRowCnt = TotalRowCnt + InsertRowCnt
        Dts.Variables("User::TotalRowCnt").Value = TotalRowCnt
        Dts.TaskResult = ScriptResults.Success
       End Sub
Applies to
Microsoft SQL Server 2005
Microsoft SQL Server 2008
Microsoft SQL Server 2008 R2
Microsoft SQL Server 2012
Microsoft SQL Server 2014
Please click to vote if the post helps you. This can be beneficial to other community members reading the thread.

Hi ITBobbyP,
If I understand correctly, you want to load data from multiple sheets in an .xlsx file into a SQL Server table.
If in this scenario, please refer to the following tips:
The Foreach Loop container should be configured as shown below:
Enumerator: Foreach ADO.NET Schema Rowset Enumerator
Connection String: The OLE DB Connection String for the excel file.
Schema: Tables.
In the Variable Mapping, map the variable to Sheet_Name, and change the Index from 0 to 2.
The connection string for Excel Connection Manager is the original one, we needn’t make any change.
Change Table Name or View name to the variable Sheet_Name.
If you want to load data from multiple sheets in multiple .xlsx files into a SQL Server table, please refer to following thread:
http://stackoverflow.com/questions/7411741/how-to-loop-through-excel-files-and-load-them-into-a-database-using-ssis-package
Thanks,
Katherine Xiong
Katherine Xiong
TechNet Community Support

Can I append the file name from a Foreach Loop Container Enumerator into a SQL Server Statement?

I would like to pass back via an Email the name of the file that was successfully FTP'ed to a remote server. And I was hoping to do this via an Execute SQL Task with the following SQL Statement...but I just don't know if it will append the Foreach Loop Container
variable enumerator to the appended Email message that I am building in SQL Server or if I can and how I can...
I sort of get the feeling that I can try and do this via Dynamic SQL...I just don't know if by building dynamic SQL if my Foreach Loop Container enumerator variable will be appended to my Email Message via the SQL Server UPDATE
@[User::FTPFHFileName]
Or do I add a Data Flow Task and an OLE DB Command and pass the Foreach Loop Container enumerator variable to a Stored Procedure referenced in the OLE DB Command via a "?"
Thanks for your review and am hopeful for a reply.

Hello,
It seems that the issue had been solved and thanks for your sharing. It will be very beneficial for other community members who have similar questions.
I’d like to mark this issue as "Answered". Please also feel free to unmark the issue, with any new findings or concerns you may have.
Regards,
Katherine Xiong
Katherine Xiong
TechNet Community Support

Strange issue with rights of account, used by SSIS (Foreach Loop Container does not return file names without Admin rights on server)

Hello everyone.
Faced very strange issue with account, which is used to run SSIS package.
The specific package uses Foreach Loop Container to retrieve file names within the specified folder, and put them into Import file task.
The package is set up to run under specific service account. This service account is given all permissions (Full control) to the folder where source files reside.
So the issue is: SSIS package fails to execute this task (Foreach Loop Container and then Import), and shows that no files are found in the directory (although files ARE there).
Once we're adding the service account into local Administrators group on the SQL Server, it works! Removing - does not work again. We cannot leave the service account as SQL server's admin as it's prohibited by our IT policies, and is just a bad practice.
Any ideas, please?
MCP

Here's the real log output:
Date 16.04.2014 12:47:09
Log Job History (RU-BW: Update)
Step ID 1
Server Server
Job Name RU-BW: Update
Step Name bw_import_cust_master_data
Duration 00:00:02
Sql Severity 0
Sql Message ID 0
Operator Emailed
Operator Net sent
Operator Paged
Retries Attempted 0
Message
Executed as user: service_account Microsoft (R) SQL Server Execute Package Utility Version 10.50.4286.0 for 64-bit Copyright (C) Microsoft Corporation 2010. All rights reserved. Started: 12:47:09 Error: 2014-04-16 12:47:11.45
Code: 0xC0202070 Source: bw_import_cust_master_data Connection manager "Input" Description: The file name property is not valid. The file name is a device or contains invalid characters. End Error Error:
2014-04-16 12:47:11.47 Code: 0xC0202070 Source: bw_import_cust_master_data Connection manager "Input" Description: The file name property is not valid. The file name is a device or contains invalid characters. End
Error Error: 2014-04-16 12:47:11.48 Code: 0xC0202070 Source: bw_import_cust_master_data Connection manager "Input" Description: The file name property is not valid. The file name is a device or contains invalid
characters. End Error Error: 2014-04-16 12:47:11.48 Code: 0xC020207E Source: Import file Flat File Source [1] Description: The file name is not valid. The file name is a device or contains invalid characters.
End Error Error: 2014-04-16 12:47:11.48 Code: 0xC004701A Source: Import file SSIS.Pipeline Description: component "Flat File Source" (1) failed the pre-execute phase and returned error code 0xC020207E.
End Error DTExec: The package execution returned DTSER_FAILURE (1). Started: 12:47:09 Finished: 12:47:11 Elapsed: 2.324 seconds. The package execution failed. The step failed.
MCP

Foreach Loop Container with a Data Flow Task looking for file from Connection Manager

So I have a Data Flow Task within a Foreach Loop Container. The Foreach Loop Container has a Variable Mapping of User:FileName to pass to the Data Flow Task.
The Data Flow Task has a Flat File Source since we're looking to process .csv Files. And the Flat File Source has a Flat File Connection Manager where I specified the File name when I created it. I thought you needed to do this even though it won't really
use it since it should be getting its File name from the Foreach Loop Container. But when attempting to execute, it is blowing up because it seems to be looking for my test file name that I indicated in the Flat File Connection Manager rather than the file
it should be trying to process in User:FileName from the Foreach Loop Container.
What am I doing wrong here??? I thought you needed to indicate a File name within the Flat File Connection Manager even though it really won't be using it.
Thanks for your review...I hope I've been clear...and am hopeful for a reply.
PSULionRP

The Flat File Connection manager's Connection String needs to be set to reference the variable used in the ForEach Loop:
Arthur My Blog

[Solved] Foreach loop that loops over all files in a folder does not consider first file found

Hello,
I have a foreach loop in SSIS (as part of Visual Studio 10 and with SQL Server 2012).
It loops over all files in a folder (ForEach File Enumerator).
I set the folder: OK
Traverse Sub folders: OK
I have set up a Variable in Variable Mappings called FileName so it shows User::FileName: OK
Index of variable is 0: OK
It almost works ok. Except that the first file it finds is never considered.
I set a breakpoint then and the first file it finds, is shown in black in the watch variable window. All subsequent files found are shown in red font. When I change the names of the files so another file is the first file found then it skips the other one
which now is the first file.
What is going on here? Why does SSIS skip the first file it finds in a foreach file loop?
Any suggestions highly appreciated.
Thank you
Andi
Andreas

"red font - interesting, any example to show us (image)
It would be interesting to share with us whether you use any file masks or expressions.
Arthur
MyBlog
Twitter
It appears a variable value turns red when it changes between breakpoints. It started black for the first value (the first file, obtained BEFORE the execution reached the breakpoint), and then turned red. I was able to reproduce this behavior on a test environment.
However, it did not skip any files.
OP, please set up the test shown in the image below. Create a breakpoint in the sequence container for the "OnPreExecute" event.

ForEach Loop Container Mapping Variable From SSIS Package Not Working In SQL-Server StoredProcedure

I have an SSIS package that uses a ForEach Loop Container to enumerate Excel Files in a dir. I also have a Task Flow that inserts data from those Excel files into SQL-Server.
Im trying to insert the file names into a column into the same table in SQL-Server by using a mapping variable in my StoredProcedure.
Im having trouble with my MappingVariable at the end of the script with red squigglies. The following is my StoredProcedure script.
CREATE PROCEDURE [dbo].[Insert_F_STG_v2]
-- Add the parameters for the stored procedure here
@Hrs float,
@Type nvarchar(100),
@SN nvarchar(100),
@Op nvarchar(100),
@[USER::CurrentFileName]
AS
BEGIN
SET NOCOUNT ON;
INSERT INTO [CRM_RC].[dbo].[F_StgTbl]
[Hrs],
[Type],
[SN],
[Op],
[Report_Date]
VALUES
@Hrs ,
@Type,
@SN,
@Op,
@[USER::CurrentFileName]
END
The last @[USER::CurrentFileName] in the Values block at the bottom of the script is the one giving me issues.
The following is the error:
Msg 102, Level 15, State 1, Procedure Insert_F_STG_v2, Line 95
Incorrect syntax near 'USER::CurrentFileName'.

This seems to be the solution, but get the following exception:
[Derived Column [2]] Error: The "Derived Column" failed because truncation occurred, and the truncation row disposition on "Derived Column.Outputs[Derived Column Output].Columns[Derived Column 1]" specifies failure on truncation. A truncation error occurred
on the specified object of the specified component.
AND: [SSIS.Pipeline] Error: SSIS Error Code DTS_E_PROCESSINPUTFAILED. The ProcessInput method on component "Derived Column" (2) failed with error code 0xC020902A while processing input "Derived Column Input" (3). The identified component returned
an error from the ProcessInput method. The error is specific to the component, but the error is fatal and will cause the Data Flow task to stop running. There may be error messages posted before this with more information about the failure.

SSIS Variable Expression not Picking Up the value dynamically set from a Foreach Loop container

Hello, everyone,
I have three variables in my SSIS package:
1) ReportLoc that holds a file path (c:\test, not changed)
2) ReportFileName that holds a file name that get assigned in a Foreach Loop container (file1, file2, etc.)
3) OutputFilePath that is the full file pathname, set in the expression: @[User::LocalPath] + "\\" + @[User::ReportFileName]
All three variable have the package scope.
I expect OutputFilePath will be asigned values like c:\test\file1, c:\test\file2, etc.
However, it alwasy remains an empty string.
Anything I did wrong? Any help and information is greatly appreciated.
Regards

That's it -- I forgot to set Evaluate as Expression to true.
Thank you very much for your reply and solution! Really appreciate it!
Regards.

How to bypass errors in the control flow in a foreach loop in SSIS?

Hi All,
Can anyone please get me through the solution for the below scenario:
- I have multiple XML files existing in the source system from where I need to read the data from each XML, validate and pull to staging database.
- So, to read the XML files from source, I have a foreach loop which will get the XML files from the source folder one by one and executes.
- There are multiple data flow tasks, script tasks, SQL tasks etc in the foreach loop to validate the business requirements.
Question: If there is an error in any of the component in foreach loop, it should log the error, come out of the loop from that point and pick up the next file with out stopping the execution. How can I achieve this?
Thanks,
Sri

Hi Sri,
According to your description, when there is an error in any of the component in Foreach Loop Container, you want to continue processing rest of the loop after recording the error file details.
To achieve your requirement, the first thing you need to do is create an OnError event handler for the Data Flow (or any other child executable). Once the event handler is created show the system variables in the variables window, locate the Propagate variable
and set it to false.
For more details, please refer to the following two blogs:
http://www.codeproject.com/Articles/384690/In-SSIS-how-to-continue-a-for-each-loop-container
http://visakhm.blogspot.in/2013/03/error-handling-in-ssis-loops.html
Thanks,
Katherine Xiong
Katherine Xiong
TechNet Community Support

Populate a dataset in a foreach loop

Similar Messages

Maybe you are looking for