SSIS Foreach Loop Container Logic

Hi all,
I want to load files and delete them afterwards. Can anyone tell me what the logic behind this container is?
In case I put a Data Flow Task in the container (which loads the file into my database) and a File System Task which is going to delete the file immediately after loading. Does the container know
which file was loaded and passes this one through to the File System Task for deletion?
I definitely want to make sure that no data gets lost and that the container really deletes the exact file it was reading before.
Thanks for your help in advance!
Regards
Laumen

What we do is to add an intermediate step to capture the details of file along with data it loaded (rows added) to a control table. We do comparison whether any data was loaded before we actually remove the file. We also dont delete file as is but will
move it to a different archive folder. This will give us an opportunity to get back file and load it incase initial load didnt happen due to some reason. The files are deleted in bulk after an ageing period (say 2 weeks)
Please Mark This As Answer if it helps to solve the issue Visakh ---------------------------- http://visakhm.blogspot.com/ https://www.facebook.com/VmBlogs

Similar Messages

Strange issue with rights of account, used by SSIS (Foreach Loop Container does not return file names without Admin rights on server)

Hello everyone.
Faced very strange issue with account, which is used to run SSIS package.
The specific package uses Foreach Loop Container to retrieve file names within the specified folder, and put them into Import file task.
The package is set up to run under specific service account. This service account is given all permissions (Full control) to the folder where source files reside.
So the issue is: SSIS package fails to execute this task (Foreach Loop Container and then Import), and shows that no files are found in the directory (although files ARE there).
Once we're adding the service account into local Administrators group on the SQL Server, it works! Removing - does not work again. We cannot leave the service account as SQL server's admin as it's prohibited by our IT policies, and is just a bad practice.
Any ideas, please?
MCP

Here's the real log output:
Date 16.04.2014 12:47:09
Log Job History (RU-BW: Update)
Step ID 1
Server Server
Job Name RU-BW: Update
Step Name bw_import_cust_master_data
Duration 00:00:02
Sql Severity 0
Sql Message ID 0
Operator Emailed
Operator Net sent
Operator Paged
Retries Attempted 0
Message
Executed as user: service_account Microsoft (R) SQL Server Execute Package Utility Version 10.50.4286.0 for 64-bit Copyright (C) Microsoft Corporation 2010. All rights reserved. Started: 12:47:09 Error: 2014-04-16 12:47:11.45
Code: 0xC0202070 Source: bw_import_cust_master_data Connection manager "Input" Description: The file name property is not valid. The file name is a device or contains invalid characters. End Error Error:
2014-04-16 12:47:11.47 Code: 0xC0202070 Source: bw_import_cust_master_data Connection manager "Input" Description: The file name property is not valid. The file name is a device or contains invalid characters. End
Error Error: 2014-04-16 12:47:11.48 Code: 0xC0202070 Source: bw_import_cust_master_data Connection manager "Input" Description: The file name property is not valid. The file name is a device or contains invalid
characters. End Error Error: 2014-04-16 12:47:11.48 Code: 0xC020207E Source: Import file Flat File Source [1] Description: The file name is not valid. The file name is a device or contains invalid characters.
End Error Error: 2014-04-16 12:47:11.48 Code: 0xC004701A Source: Import file SSIS.Pipeline Description: component "Flat File Source" (1) failed the pre-execute phase and returned error code 0xC020207E.
End Error DTExec: The package execution returned DTSER_FAILURE (1). Started: 12:47:09 Finished: 12:47:11 Elapsed: 2.324 seconds. The package execution failed. The step failed.
MCP

SSIS: ForEach Loop Container Enumerates Same File

Trying to loop an Excel Directory. Have it in a ForEach Loop Container. It keeps looping the same file the amount of times the file pattern recognized; i.e Loop starts at report with 10 lines, there are 7 files that need looped...How
do I get it to loop all the files that need enumerated rather than just the default file path in the connection string?

I have both path and file in foreach loop container...How many variables should I be using in my case? I have one (ExcelInput) declared in the foreach loop container and passing to Excel Connection string expression.
Hi jsorrisone,
One SSIS variable is enough in your scenario. In the Foreach Loop Container, we set the Enumerator to “Foreach File Enumerator”, specify the path of the source folder, keep the “Retrieve file name” option as “Fully qualified”, and store the full path of
the Excel file to the string type variable ExcelInput.
Then, add a Data Flow Task to the Foreach Loop Container, drag an Excel Source, create an Excel Connection Manager. After that, set the expressions for the Excel Connection Manager, and set the expression of “ExcelFilePath” property to @[User::ExcelInput].
Reference:
http://mikedavissql.com/2013/09/16/loop-through-excel-files-in-ssis/
Regards,
Mike Yin
TechNet Community Support

SSIS Foreach Loop Container not processing all files

Morning
Currently there are 3 csv file in the directory. The for-each loop container only process the first file. Everything was working fine then all of a sudden only the first file get's processed.
Ty
W

Hi
Was a problem with the script task message box I was using. The message did not display on the screen. But now I am dealing with a precedence constraint problem. I have 3 precedence constraints that controls which part of the package executes based on the
file name received. The expression works for 1 file but not for the other 2 files. The expression is basically the same except the difference is that I specify a part of the file that is different.
FINDSTRING(@ADO_FileName_Logistics,"hd3",1) > 0 --this works :-)
FINDSTRING(@ADO_FileName_Logistics,"it3",1) > 0 -- not this
FINDSTRING(@ADO_FileName_Logistics,"pr3",1) > 0 -- and not this either
Could you perhaps guide me where the problem could be ?
W

How can I load a .xlsx File into a SQL Server Table using a Foreach Loop Container in SSIS?

I know I've REALLY struggled with this before. I just don't understand why this has to be soooooo difficult.
I can very easily do a straight Data Pump of a .xlsX File into a SQL Server Table using a normal Excel Connection and a normal Excel Source...simply converting Unicode to DT_STR and then using an OLE DB Destination of the SQL Server Table.
If I want to make the SSIS Package a little more flexible by allowing multiple .xlsX spreadsheets to be pumped in by using a Foreach Loop Container, the whole SSIS Package seems to go to hell in a hand basket. I simply do the following...
Put the Data Flow Task within the Foreach Loop Container
Add the Variable Mapping Variable User::FilePath that I defined as a Variable and a string within the FOreach Loop Container
I change the Excel Connection and its Expression to be ExcelFilePath ==> @[User::FilePath]
I then try and change the Excel Source and its Data Access Mode to Table Name or view name variable and provide the Variable Name User::FilePath
And that's when I run into trouble...
Exception from HRESULT: 0xC02020E8
Error at Data Flow Task [Excel Source [56]]:SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occured. Error code: 0x80004005.
Error at Data Flow Task [Excel Source [56]]: Opening a rowset for "...(the EXACT Path and .xlsx File Name)...". Check that the object exists in the database. (And I know it's there!!!)
I don't understand by adding a Foreach Loop Container to try and make this as efficient as possible has caused such an error unless I'm overlooking something. I have even tried delaying my validations and that doesn't seem to help.
I have looked hard in Google and even YouTube to try and find a solution for this but for the life of me I cannot seem to find anything on pumping a .xlsX file into SQL Server using a Foreach Loop Container.
Can ANYONE please help me out here? I'm at the end of my rope trying to get this to work. I think the last time I was in this quandry, trying to pump a .xlsX File into a SQL Server Table using a Foreach Loop Container in SSIS, I actually wrote a C# Script
to write the contents of the .xlsX File into a .csv File and then Actually used the .csv File to pump the data into a SQL Server Table.
Thanks for your review and am hoping and praying for a reply and solution.

Hi ITBobbyP,
If I understand correctly, you want to load data from multiple sheets in an .xlsx file into a SQL Server table.
If in this scenario, please refer to the following tips:
The Foreach Loop container should be configured as shown below:
Enumerator: Foreach ADO.NET Schema Rowset Enumerator
Connection String: The OLE DB Connection String for the excel file.
Schema: Tables.
In the Variable Mapping, map the variable to Sheet_Name, and change the Index from 0 to 2.
The connection string for Excel Connection Manager is the original one, we needn’t make any change.
Change Table Name or View name to the variable Sheet_Name.
If you want to load data from multiple sheets in multiple .xlsx files into a SQL Server table, please refer to following thread:
http://stackoverflow.com/questions/7411741/how-to-loop-through-excel-files-and-load-them-into-a-database-using-ssis-package
Thanks,
Katherine Xiong
Katherine Xiong
TechNet Community Support

[Forum FAQ] How to calculate the total count of insert rows within a Foreach Loop Container in SSIS?

Introduction
We need to loop through all the flat files that have the same structure in a folder and import all the data to a single SQL Server table. How can we obtain the total count of the rows inserted to the destination SQL Server table?
Solution
We can use Execute SQL Task or Script Task to aggregate the row count increment for each iteration of the Foreach Loop Container. The following steps are the preparations before we add the Execute SQL Task or Script Task:
Create a String type variable FilePath, two Int32 type variables InsertRowCnt and TotalRowCnt.
Drag a Foreach Loop Container to the Control Flow design surface, set the Enumerator to “Foreach File Enumerator”, specify the source folder and the files extension, and set the “Retrieve file name” option to “Fully qualified”.
On the “Variable Mappings” tab of the container, map the variable FilePath to the collection value.
Drag a Data Flow Task to the container, in the Data Flow Task, add a Flat File Source, a Row Count Transformation, and an OLE DB Destination, and join them. Create a Flat File Connection Manager to connect to one of the flat files, and then configure the
Flat File Source as well as the OLE DB Destination adapter. Set the variable for the Row Count Transformation to “User::InsertRowCnt”.
Open the Property Expressions Editor for the Flat File Connection Manager, and set the expression of “ConnectionString” property to
“@[User::FilePath]”.
(I) Execute SQL Task Method:
In the Control Flow, drag an Execute SQL Task under the Data Flow Task and join them.
Create one or using any one existing OLE DB Connection Manager for the Execute SQL Task, set the “ResultSet” option to “Single row”, and then set the “SQLStatement” property to:
DECLARE @InsertRowCnt INT,
               @TotalRowCnt INT
SET @InsertRowCnt=?
SET @TotalRowCnt=?
SET @TotalRowCnt=@InsertRowCnt+@TotalRowCnt
SELECT TotalRowCnt=@TotalRowCnt
On to parameter 1.
On the “Result Set” tab of the Execute SQL Task, map result 0 to variable “User::TotalRowCnt”.
(II) Script Task Method:
In the Control Flow, drag a Script Task under the Data Flow Task and join them.
In the Script Task, select variable InsertRowCnt for “ReadOnlyVariables” option, and select variable TotalRowCnt for “ReadWriteVariables”.
Edit the Main method as follows (C#):
public void Main()
// TODO: Add your code here
int InsertRowCnt = Convert.ToInt32(Dts.Variables["User::InsertRowCnt"].Value.ToString()
int TotalRowCnt = Convert.ToInt32(Dts.Variables["User::TotalRowCnt"].Value.ToString());
TotalRowCnt = TotalRowCnt + InsertRowCnt;
Dts.Variables["User::InsertRowCnt"].Value = TotalRowCnt;
Dts.TaskResult = (int)ScriptResults.Success;
          Or (VB)
          Public Sub Main()
        ' Add your code here
        Dim InsertRowCnt As Integer =
        Convert.ToInt32(Dts.Variables("User::InsertRowCnt").Value.ToString())
        Dim TotalRowCnt As Integer =
        Convert.ToInt32(Dts.Variables("User::TotalRowCnt").Value.ToString())
        TotalRowCnt = TotalRowCnt + InsertRowCnt
        Dts.Variables("User::TotalRowCnt").Value = TotalRowCnt
        Dts.TaskResult = ScriptResults.Success
       End Sub
Applies to
Microsoft SQL Server 2005
Microsoft SQL Server 2008
Microsoft SQL Server 2008 R2
Microsoft SQL Server 2012
Microsoft SQL Server 2014
Please click to vote if the post helps you. This can be beneficial to other community members reading the thread.

Hi ITBobbyP,
If I understand correctly, you want to load data from multiple sheets in an .xlsx file into a SQL Server table.
If in this scenario, please refer to the following tips:
The Foreach Loop container should be configured as shown below:
Enumerator: Foreach ADO.NET Schema Rowset Enumerator
Connection String: The OLE DB Connection String for the excel file.
Schema: Tables.
In the Variable Mapping, map the variable to Sheet_Name, and change the Index from 0 to 2.
The connection string for Excel Connection Manager is the original one, we needn’t make any change.
Change Table Name or View name to the variable Sheet_Name.
If you want to load data from multiple sheets in multiple .xlsx files into a SQL Server table, please refer to following thread:
http://stackoverflow.com/questions/7411741/how-to-loop-through-excel-files-and-load-them-into-a-database-using-ssis-package
Thanks,
Katherine Xiong
Katherine Xiong
TechNet Community Support

ForEach Loop Container Mapping Variable From SSIS Package Not Working In SQL-Server StoredProcedure

I have an SSIS package that uses a ForEach Loop Container to enumerate Excel Files in a dir. I also have a Task Flow that inserts data from those Excel files into SQL-Server.
Im trying to insert the file names into a column into the same table in SQL-Server by using a mapping variable in my StoredProcedure.
Im having trouble with my MappingVariable at the end of the script with red squigglies. The following is my StoredProcedure script.
CREATE PROCEDURE [dbo].[Insert_F_STG_v2]
-- Add the parameters for the stored procedure here
@Hrs float,
@Type nvarchar(100),
@SN nvarchar(100),
@Op nvarchar(100),
@[USER::CurrentFileName]
AS
BEGIN
SET NOCOUNT ON;
INSERT INTO [CRM_RC].[dbo].[F_StgTbl]
[Hrs],
[Type],
[SN],
[Op],
[Report_Date]
VALUES
@Hrs ,
@Type,
@SN,
@Op,
@[USER::CurrentFileName]
END
The last @[USER::CurrentFileName] in the Values block at the bottom of the script is the one giving me issues.
The following is the error:
Msg 102, Level 15, State 1, Procedure Insert_F_STG_v2, Line 95
Incorrect syntax near 'USER::CurrentFileName'.

This seems to be the solution, but get the following exception:
[Derived Column [2]] Error: The "Derived Column" failed because truncation occurred, and the truncation row disposition on "Derived Column.Outputs[Derived Column Output].Columns[Derived Column 1]" specifies failure on truncation. A truncation error occurred
on the specified object of the specified component.
AND: [SSIS.Pipeline] Error: SSIS Error Code DTS_E_PROCESSINPUTFAILED. The ProcessInput method on component "Derived Column" (2) failed with error code 0xC020902A while processing input "Derived Column Input" (3). The identified component returned
an error from the ProcessInput method. The error is specific to the component, but the error is fatal and will cause the Data Flow task to stop running. There may be error messages posted before this with more information about the failure.

SSIS Variable Expression not Picking Up the value dynamically set from a Foreach Loop container

Hello, everyone,
I have three variables in my SSIS package:
1) ReportLoc that holds a file path (c:\test, not changed)
2) ReportFileName that holds a file name that get assigned in a Foreach Loop container (file1, file2, etc.)
3) OutputFilePath that is the full file pathname, set in the expression: @[User::LocalPath] + "\\" + @[User::ReportFileName]
All three variable have the package scope.
I expect OutputFilePath will be asigned values like c:\test\file1, c:\test\file2, etc.
However, it alwasy remains an empty string.
Anything I did wrong? Any help and information is greatly appreciated.
Regards

That's it -- I forgot to set Evaluate as Expression to true.
Thank you very much for your reply and solution! Really appreciate it!
Regards.

Using an ssis variable object as a data source in a foreach loop container

hi we run 2012 std. I have an ssis var of type object that is hydrated from a dynamic query in an execute sql task. I can count on one thing in this object and that is that ID will always be the first "col" on each "row".
Otherwise, resultset can contain a variety of things based on params passed to this sub pkg.
I'd like to introduce a for each loop on this object and tap into index 0, ie the id column.
The first question is "will I be able to parse this object's id in a for each component?".
In the for each loop container properties, I c an item enumerator, ado enumerator, ado.net schema rowset and variable enumerator as enumerator choices. Which do I want if the answer to the 1st question is yes?

hoping 2 avoid a data flow. I wonder if what u r saying is that I cannot use an felc to do this. Or if u r showing one of many alternatives.
Sorry, bad title. I just changed that.
I just tried foreach from variable enumerator and aborted with a message that said my variable "coll" doesn't have an enumerator. Going 2 try some of the other choices.
I just tried ado enumerator and from what I can c it isn't actually iterating thru my resultset even once.
I just tried a foreach item enumerator giving "column 0" a data type that matches my ID but from what I can tell the component isn't iterating thru my collection.
ado.net schema rowset doesn't look inviting at all in that it looks like it wants to go back to the db. Not what I had in mind having already gotten this resultset into memory.
according to this article u r supposed to be able to do this using ado enumerator
social.technet.microsoft.com slash wiki slash articles slash sis-looping-over-object-variables-with-as-ado-enumeration-in-foreach-loop-container.aspx

SSIS 2008 R2 Foreach Loop Container on ADO object in Endless Loop

SSIS 2008 R2
I have an object Variable populated by a SQL Query. I then attach it to a Foreach Loop Container with "Foreach ADO Enumerator" and define the variable mappings. Inside the Foreach Loop is a series of tasks. It appears that the
tasks are processing each of the rows in the Object but it also appears that the Foreach loop never stops but runs until I kill it. I was under the impression that the Foreach loop would stop when after the last row in the Object was consumed.
As it is now, the Foreach Loop is useless as it runs forever.
Any suggestions would be appreciated.
Thanks,
Michael
P.S. I have determined that the ADO object has a size or count limitation which if exceeded causes the Foreach Loop task to malfunction in both Studio and as an Agent job. I could not find a limit in MS docs, but there sure seems to be
a limit somewhere which everyone should be aware of. 5,000 rows exceeds the limit, in my case.

A typical memory limit rather, especially noticeable in dev env where it is 32 bit
http://www.codeguru.com/csharp/.net/article.php/c16921/Understanding-NET-application-options-for-32-and-64bit-systems.htm
Arthur My Blog

SSIS Package to load multiple excel sheets using foreach loop container

I am creating a SSIS package which is accessing an excel file and then using a Foreach loop container to load each tab to sql server. However when I try to configure the Excel source, I get the following message
"The argument system_comobject cannot be converted into parameter type. Additional Information - Object must implement iconvertible (mscorlib)"
Any help to resolve would be much appreciated.
thanks
Scott

If the title is only using column D and the rest of the columns are empty on sheet1/row1
AND
if only sheet1 as headers,
THEN you can try something like the following
do NOT select row 1 as header or anything.
In your foreachloop > dataflow, you will start with EXCEL connection then Conditional Split inside which you will define one case for which expression should be like If column D has value like "Title expression" && Column A == "Column A Header
name" THEN yay or nay. You might want to look into
FINDSTRING function to assist with this (if you choose to use FINDSTRING, you are looking for !isNull for this case..If it finds the string you are looking for, it will pass an int value (title + header) if it doesn't find the string, it will pass
NULL (data)) or you can say If Column A, B, C,E,F,G...is NULL and D is NOT NULL && Column A = Column A Header string, then Case 1 else default.
The default output would be the real data.
So when sheet 1 is processing, you will have Case 1 output with 2 rows (title + header) and Default output with the rest of the rows (real data).
For other sheets, you will have Case 1 output with 0 rows (no title + header) and Default output with the data rows.
The above sln assumes that the data in each sheet is similar in datatypes and length perspective, you have defined input/output datatypes properly, and etc.

Is it possible to load multiple .xlsx files into a SQL Server Table using SSIS and a Foreach Loop Container when each Excel spreadsheet is comprised of two different worksheets

So we have an Invoice .xlsx File from a 3rd party vendor. It contains two worksheets..."Enrolled" and "Engaged". The data and data columns in each worksheet is different. Is it possible to loop through multiple .xlsx files using SSIS
and a Foreach Loop Container for each spreadsheet, and then another Foreach Loop Container to control each worksheet, and pump the Excel data into a SQL Server Table first for "Enrolled" and then for "Engaged"? How can I control the Foreach
Loop Container in SSIS to process ONLY the "Enrolled" worksheet first? And then the "Engaged" worksheet next?
I know I have multiples out here and I apologize for that...but right now it seems as though I take three steps forward and then two back.
Any help would be GREATLY appreciated!
Thanks in advance!

If the structure of the Excel sheets does not change from file to file then you can by having one ForEach Loop processing always the "Enrolled" sheet and another always the "Engaged" this is doable because the Excel OLEDB connector allows
to pick individual sheets, it is problematic therefore when sheet names themselves change.
MSDN has an example: https://msdn.microsoft.com/en-ca/library/ms345182.aspx
Arthur
MyBlog
Twitter

Excel Source cannot find sheet when using Foreach Loop Container

I have SQL Server 2012 SSIS. I need help with Foreach Loop container.
1) I have C:\\Excel\ folder and multiple Excel.xlsx files are stored there to be imported
2) I have Foreach Loop Container
-Foreach File Enumerator is selected
-Expressions are empty
-Folder is set as C:\\Excel\
-Files is *.*
-Variable is created. User::Filename, 0
2) I have created variable FileName, String,0
3) I have Excel Connection Manager
-ExcelFilePath = @[User::FileName]
4) I have data flow task with Excel Source and OLE DB Destination
Error occured with Execute:
[Excel Source [2]] Error: SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occurred. Error code: 0x80040E37.
[Excel Source [2]] Error: Opening a rowset for "SheetName$" failed. Check that the object exists in the database.
Kenny_I

Hi Kenny_I,
The issue occurs because you have not specified a valid value for the variable @FileName. The error persists even if we set the “DelayValidation” property of the Excel Connection Manager to True. After you assign a value like “C:\Excel\Test1.xlsx” (without
quotes) to the variable, the package should work fine.
Reference:
http://www.bidn.com/blogs/mikedavis/ssis/625/loop-through-excel-file-in-ssis
Regards,
Mike Yin
If you have any feedback on our support, please click
here
Mike Yin
TechNet Community Support

Dynamically specify folder files for Foreach File Enumerator in Foreach Loop Container and load into destination

hi,
I need to load data into tables before loading data I have several folders like a1, a2,a3... each folder 'll contain a several .csv files. I need to load those csv files data into destination based on folder wise
Thanks
AVS

you can use foreachloop for that. It has an option called recursive. So just point it to parent level folder and give recursive option and it will go through all folders
https://www.simple-talk.com/sql/ssis/ssis-basics-introducing-the-foreach-loop-container/
Please Mark This As Answer if it helps to solve the issue Visakh ---------------------------- http://visakhm.blogspot.com/ https://www.facebook.com/VmBlogs

The property XMLDataVariable on the XML Source was empty - Case Foreach Loop Container

I have very simple XML source and OLE DB Source in Foreach Loop Container,
but seem to get error, because XML Source cannot find the XML.
"Error at Data Flow Task. The property XMLDataVariable on the XML Source was empty"
Variables:
FileExtention = *.xml
FilePath =
FolderPath = C:\Customer\XML
XML Source:
Data Access mode = XML file from variable
Variable name = User::filePath
Foreach Loop Container:
Enumerator = Foreach File Enumerator
Expression-Directory = @User:FolderPath
Expression-FileSpec = @User::FileExtension
Enumerator Folder = C:\Customer\XML
Files = *.xml
Variable mapping-User::FilePath = 0
Kenny_I

Have you filled the variable (filePath) with a default value? Otherwise the XML Source cannot validate. Alternative is look for a property "DelayValidation". In most cases this is a property of the connection manager, but you're not using
one for the xml source....
Please mark the post as answered if it answers your question | My SSIS Blog:
http://microsoft-ssis.blogspot.com |
Twitter

SSIS Foreach Loop Container Logic

Similar Messages

Maybe you are looking for