Auto increment variable in foreach loop in ssis

i know this has been asked N number of times, but still i have a question for you guys
i have files sitting in FTP location
i am loading the data from the files into tables (no issues)
but i need to assign value to a particular column as -99999999 and then auto increment that number while loading the files into the database
I cannot have auto increment for the that particular column
should i use a derived column, script component, execute sql task?
can somebody point me to the right direction
if it were a single file, assigning the auto incremental number is easy, but with for each loop, i am not able to get the solution
Thanks

Hi Jack,
To achieve your goal, you can use a Row Count Transformation in the Data Flow Task to get row count for each file, and then use an Execute SQL Task after the Data Flow Task to update the total row count variable. Supposing the variable used in the Row Count
Transformation is InsertCount, the variable used to store the total row count is TotalInsertCount. We need to set the ResultSet of the Execute SQL Task to “Single row”, and the SQLStatement to:
DECLARE @Insert_Count INT, @Total_Insert_Count INT
SET @Insert_Count=?
SET @Total_Insert_Count=?
SET @Total_Insert_Count=@Total_Insert_Count + @Insert_Count
SELECT R_Total_Insert_Count=@Total_Insert_Count
Then, set the Parameter Mapping of the Execute SQL Task, we map variable InsertCount and TotalInsertCount to the parameters in the SQL statement respectively. In the ResultSet page of the Execute SQL Task, map the resultset to variable TotalInsertCount.
Please refer to the following screenshot:
Regards,
Mike Yin
TechNet Community Support

Similar Messages

Auto increment variable (global)

hi all,
let say i've a db table with columns sno, sname, marks. and i want to make sno as global auto incremental variable so tht when i insert record sno should get the current incremented value on screen even though the appn restarted/refreshed. how do i do it?
i've created sno in comp.controller and mapped to view controller. it works on insert action, but when i refresh the page sno will set to ZERO. will it be possbile in wd-java?
tnx,
-JB

Hi
We had this situation some time ago and solved it doing these steps:
-Create a java class like this:
public class GlobalCounter{
     private static long longId = -2;
     private static synchronized long getNext(){
          return ++longId;
     public GlobalCounter() {
          if(this.getNext()==-1){
               //get the last id of your DB or Backend
               long lastId = getLastNumberFromBackend();
               this.longId = ++lastId;
What you do here is that you hava a static value that is initialized to -2 (you can change this if you want) and in the constructor method you ask if the getNext value equals -1 you will ask your backend or database to get you the last number, this is only done once because that number should be at least 0 or greater, this static variable will keep the number from the backend and each time you want to increment that value you will call GlobalCounter.getNext(); method. If the server is restarted the you will call the backend or database just once again to get the last number.
To make it short, you only call your backend one time to get the last number every time the AS is restarted but not each time you want to create a new record.
Hope this helps.
Cheers
JV

Unexpected behavior of Foreach Loop in SSIS 2012

I have come across the unexpected behavior of Foreach Loop in SSIS 2012. While testing package for invalid folder (path asdf:\Vishal_Work does not exists in this case) for source files.
I observed that the control does not enters into Foreach Loop, but also it does not fails the task. The task Foreach Loop runs successfully, though folder was invalid.
The package did not throw any warning regarding invalid folder, but for the unmatched file pattern.
My concern is ‘why Foreach Loop in SSIS did not failed even if folder was invalid or not exists?’

The warning is "The For Each File enumerator is empty. The For Each File enumerator did not find any files that matched the file pattern, or the specified directory was empty."
The warning message states the specified directory was empty, though it does not extists.
We can handle this scenario through script task easily, but I really wonder why Foreach loop does not validate the folder. It is loophole in Foreach loop?
I would not classify it as loophole. I actually agree with Russ, it was a design decision on the side of the development team.
But you may freely file it as a bug or suggestion at MS Connect: https://connect.microsoft.com
I would vote it up because I see how it can lead to undesired package behavior (logical flaw).
Arthur My Blog

How to bypass errors in the control flow in a foreach loop in SSIS?

Hi All,
Can anyone please get me through the solution for the below scenario:
- I have multiple XML files existing in the source system from where I need to read the data from each XML, validate and pull to staging database.
- So, to read the XML files from source, I have a foreach loop which will get the XML files from the source folder one by one and executes.
- There are multiple data flow tasks, script tasks, SQL tasks etc in the foreach loop to validate the business requirements.
Question: If there is an error in any of the component in foreach loop, it should log the error, come out of the loop from that point and pick up the next file with out stopping the execution. How can I achieve this?
Thanks,
Sri

Hi Sri,
According to your description, when there is an error in any of the component in Foreach Loop Container, you want to continue processing rest of the loop after recording the error file details.
To achieve your requirement, the first thing you need to do is create an OnError event handler for the Data Flow (or any other child executable). Once the event handler is created show the system variables in the variables window, locate the Propagate variable
and set it to false.
For more details, please refer to the following two blogs:
http://www.codeproject.com/Articles/384690/In-SSIS-how-to-continue-a-for-each-loop-container
http://visakhm.blogspot.in/2013/03/error-handling-in-ssis-loops.html
Thanks,
Katherine Xiong
Katherine Xiong
TechNet Community Support

FOREACH LOOP IN SSIS

I have a folder structure as below:
Main
--201310
----October language line.xlsx
----October billing report.xlsx
-- 201311
----November language line.xlsx
----November billing report.xlsx
-- 201312
----December language line.xlsx
----December billing report.xlsx
so the folder structure is main -> yearmonth folder -> files.
I have to loop through the yearmonth folders and then pick all the files from respective yearmonth folder.
Could someone help me do this. I am a newbie to SSIS.

Use the Foreach Loop File Enumerator. More details
here.
It fills an SSIS String variable with the complete path of the excel file. You can use that variable in an expression on the
ExcelFilePath property of the excel connection manager.
Please mark the post as answered if it answers your question | My SSIS Blog:
http://microsoft-ssis.blogspot.com |
Twitter

[Solved] Foreach loop that loops over all files in a folder does not consider first file found

Hello,
I have a foreach loop in SSIS (as part of Visual Studio 10 and with SQL Server 2012).
It loops over all files in a folder (ForEach File Enumerator).
I set the folder: OK
Traverse Sub folders: OK
I have set up a Variable in Variable Mappings called FileName so it shows User::FileName: OK
Index of variable is 0: OK
It almost works ok. Except that the first file it finds is never considered.
I set a breakpoint then and the first file it finds, is shown in black in the watch variable window. All subsequent files found are shown in red font. When I change the names of the files so another file is the first file found then it skips the other one
which now is the first file.
What is going on here? Why does SSIS skip the first file it finds in a foreach file loop?
Any suggestions highly appreciated.
Thank you
Andi
Andreas

"red font - interesting, any example to show us (image)
It would be interesting to share with us whether you use any file masks or expressions.
Arthur
MyBlog
Twitter
It appears a variable value turns red when it changes between breakpoints. It started black for the first value (the first file, obtained BEFORE the execution reached the breakpoint), and then turned red. I was able to reproduce this behavior on a test environment.
However, it did not skip any files.
OP, please set up the test shown in the image below. Create a breakpoint in the sequence container for the "OnPreExecute" event.

Add an auto incremental value to TITLE field.

Hi, i am currently working on a custom list in SharePoint 2010 and there is a title field. I would like to append an auto incremental variable to "Request" to make it look like:
Request 1
Request 2
Request 3
and so on...
As you can see below, i tried to use the calculated value Request+ID, but it didn't work out. So is there anyway to solve this problem?
<input name="ctl00$m$g_0bb0fcf5_65ea_4af9_ba76_7c0babad1a82$ctl00$ctl05$ctl00$ctl00$ctl00$ctl04$ctl00$ctl00$TextField"
type="text" value="[Request-]&[ID]"
maxlength="50" id="ctl00_m_g_0bb0fcf5_65ea_4af9_ba76_7c0babad1a82_ctl00_ctl05_ctl00_ctl00_ctl00_ctl04_ctl00_ctl00_TextField"
title="Title" class="ms-long ms-spellcheck-true" /><br />

Hi andylow,
You can also accomplish this without using a workflow but with a custom form on your list. To do this take the following steps:
Make sure that the title field of your list is not a required value
When you're in the list, click on the List tab in the ribbon
There select "Customize Form"
Your form will now open in InfoPath. In the right hand menu you will see all your data fields.
Make the Title number field "Read Only". To do this right click the field, select properties, click on the second tab and select Read Only
In the menu bar, click on Manage Rules
Now click the Data tab in the menu and select Sumit Options
In the menu that opens select the option to run rules before submit
Now there will be a button available in the menu for Submit Options, click on that
Now you can add rules and actions on the submit action
Create a new Action rule and call it Initial Save
As condition for the rule, set the condition that the Title field must be blank
The first action of the rule will be to Submit the form. When the form is submitted it will be assigned an ID, before that will be blank
Now add an Action to set a fields value.
Select the Title field as the field to set
You can set a hard coded value for the value to set but you can also click the function button, click on that
Now another window opens where you can add fields, functions etc.
Now first select the concat function from the function menu
Then double click the first link in the resulting function and type Request
Now double click the second link and select the ID field
Now get rid of the last remaining arguments in the concat function
Click OK until you're back in InfoPath
Now set one more action to submit the form again
Now ad a second rule and call that Edit
As condition for the rule, set the condition that the Title field must be not blank
Add the Submit Form action for this rule
Now you can publish your form with the Quick Publish button. This is a Blue circle with a white upward arrow and a lightning bolt next to the save button in the top menu bar.
Now when you add a new item and save it, the title will be set to Request&ID. When you edit the form no changes will be made.
Cheers,
Eduard Spelier

ForEach Loop Container Mapping Variable From SSIS Package Not Working In SQL-Server StoredProcedure

I have an SSIS package that uses a ForEach Loop Container to enumerate Excel Files in a dir. I also have a Task Flow that inserts data from those Excel files into SQL-Server.
Im trying to insert the file names into a column into the same table in SQL-Server by using a mapping variable in my StoredProcedure.
Im having trouble with my MappingVariable at the end of the script with red squigglies. The following is my StoredProcedure script.
CREATE PROCEDURE [dbo].[Insert_F_STG_v2]
-- Add the parameters for the stored procedure here
@Hrs float,
@Type nvarchar(100),
@SN nvarchar(100),
@Op nvarchar(100),
@[USER::CurrentFileName]
AS
BEGIN
SET NOCOUNT ON;
INSERT INTO [CRM_RC].[dbo].[F_StgTbl]
[Hrs],
[Type],
[SN],
[Op],
[Report_Date]
VALUES
@Hrs ,
@Type,
@SN,
@Op,
@[USER::CurrentFileName]
END
The last @[USER::CurrentFileName] in the Values block at the bottom of the script is the one giving me issues.
The following is the error:
Msg 102, Level 15, State 1, Procedure Insert_F_STG_v2, Line 95
Incorrect syntax near 'USER::CurrentFileName'.

This seems to be the solution, but get the following exception:
[Derived Column [2]] Error: The "Derived Column" failed because truncation occurred, and the truncation row disposition on "Derived Column.Outputs[Derived Column Output].Columns[Derived Column 1]" specifies failure on truncation. A truncation error occurred
on the specified object of the specified component.
AND: [SSIS.Pipeline] Error: SSIS Error Code DTS_E_PROCESSINPUTFAILED. The ProcessInput method on component "Derived Column" (2) failed with error code 0xC020902A while processing input "Derived Column Input" (3). The identified component returned
an error from the ProcessInput method. The error is specific to the component, but the error is fatal and will cause the Data Flow task to stop running. There may be error messages posted before this with more information about the failure.

SSIS Variable Expression not Picking Up the value dynamically set from a Foreach Loop container

Hello, everyone,
I have three variables in my SSIS package:
1) ReportLoc that holds a file path (c:\test, not changed)
2) ReportFileName that holds a file name that get assigned in a Foreach Loop container (file1, file2, etc.)
3) OutputFilePath that is the full file pathname, set in the expression: @[User::LocalPath] + "\\" + @[User::ReportFileName]
All three variable have the package scope.
I expect OutputFilePath will be asigned values like c:\test\file1, c:\test\file2, etc.
However, it alwasy remains an empty string.
Anything I did wrong? Any help and information is greatly appreciated.
Regards

That's it -- I forgot to set Evaluate as Expression to true.
Thank you very much for your reply and solution! Really appreciate it!
Regards.

Using an ssis variable object as a data source in a foreach loop container

hi we run 2012 std. I have an ssis var of type object that is hydrated from a dynamic query in an execute sql task. I can count on one thing in this object and that is that ID will always be the first "col" on each "row".
Otherwise, resultset can contain a variety of things based on params passed to this sub pkg.
I'd like to introduce a for each loop on this object and tap into index 0, ie the id column.
The first question is "will I be able to parse this object's id in a for each component?".
In the for each loop container properties, I c an item enumerator, ado enumerator, ado.net schema rowset and variable enumerator as enumerator choices. Which do I want if the answer to the 1st question is yes?

hoping 2 avoid a data flow. I wonder if what u r saying is that I cannot use an felc to do this. Or if u r showing one of many alternatives.
Sorry, bad title. I just changed that.
I just tried foreach from variable enumerator and aborted with a message that said my variable "coll" doesn't have an enumerator. Going 2 try some of the other choices.
I just tried ado enumerator and from what I can c it isn't actually iterating thru my resultset even once.
I just tried a foreach item enumerator giving "column 0" a data type that matches my ID but from what I can tell the component isn't iterating thru my collection.
ado.net schema rowset doesn't look inviting at all in that it looks like it wants to go back to the db. Not what I had in mind having already gotten this resultset into memory.
according to this article u r supposed to be able to do this using ado enumerator
social.technet.microsoft.com slash wiki slash articles slash sis-looping-over-object-variables-with-as-ado-enumeration-in-foreach-loop-container.aspx

[Forum FAQ] How to calculate the total count of insert rows within a Foreach Loop Container in SSIS?

Introduction
We need to loop through all the flat files that have the same structure in a folder and import all the data to a single SQL Server table. How can we obtain the total count of the rows inserted to the destination SQL Server table?
Solution
We can use Execute SQL Task or Script Task to aggregate the row count increment for each iteration of the Foreach Loop Container. The following steps are the preparations before we add the Execute SQL Task or Script Task:
Create a String type variable FilePath, two Int32 type variables InsertRowCnt and TotalRowCnt.
Drag a Foreach Loop Container to the Control Flow design surface, set the Enumerator to “Foreach File Enumerator”, specify the source folder and the files extension, and set the “Retrieve file name” option to “Fully qualified”.
On the “Variable Mappings” tab of the container, map the variable FilePath to the collection value.
Drag a Data Flow Task to the container, in the Data Flow Task, add a Flat File Source, a Row Count Transformation, and an OLE DB Destination, and join them. Create a Flat File Connection Manager to connect to one of the flat files, and then configure the
Flat File Source as well as the OLE DB Destination adapter. Set the variable for the Row Count Transformation to “User::InsertRowCnt”.
Open the Property Expressions Editor for the Flat File Connection Manager, and set the expression of “ConnectionString” property to
“@[User::FilePath]”.
(I) Execute SQL Task Method:
In the Control Flow, drag an Execute SQL Task under the Data Flow Task and join them.
Create one or using any one existing OLE DB Connection Manager for the Execute SQL Task, set the “ResultSet” option to “Single row”, and then set the “SQLStatement” property to:
DECLARE @InsertRowCnt INT,
               @TotalRowCnt INT
SET @InsertRowCnt=?
SET @TotalRowCnt=?
SET @TotalRowCnt=@InsertRowCnt+@TotalRowCnt
SELECT TotalRowCnt=@TotalRowCnt
On to parameter 1.
On the “Result Set” tab of the Execute SQL Task, map result 0 to variable “User::TotalRowCnt”.
(II) Script Task Method:
In the Control Flow, drag a Script Task under the Data Flow Task and join them.
In the Script Task, select variable InsertRowCnt for “ReadOnlyVariables” option, and select variable TotalRowCnt for “ReadWriteVariables”.
Edit the Main method as follows (C#):
public void Main()
// TODO: Add your code here
int InsertRowCnt = Convert.ToInt32(Dts.Variables["User::InsertRowCnt"].Value.ToString()
int TotalRowCnt = Convert.ToInt32(Dts.Variables["User::TotalRowCnt"].Value.ToString());
TotalRowCnt = TotalRowCnt + InsertRowCnt;
Dts.Variables["User::InsertRowCnt"].Value = TotalRowCnt;
Dts.TaskResult = (int)ScriptResults.Success;
          Or (VB)
          Public Sub Main()
        ' Add your code here
        Dim InsertRowCnt As Integer =
        Convert.ToInt32(Dts.Variables("User::InsertRowCnt").Value.ToString())
        Dim TotalRowCnt As Integer =
        Convert.ToInt32(Dts.Variables("User::TotalRowCnt").Value.ToString())
        TotalRowCnt = TotalRowCnt + InsertRowCnt
        Dts.Variables("User::TotalRowCnt").Value = TotalRowCnt
        Dts.TaskResult = ScriptResults.Success
       End Sub
Applies to
Microsoft SQL Server 2005
Microsoft SQL Server 2008
Microsoft SQL Server 2008 R2
Microsoft SQL Server 2012
Microsoft SQL Server 2014
Please click to vote if the post helps you. This can be beneficial to other community members reading the thread.

Hi ITBobbyP,
If I understand correctly, you want to load data from multiple sheets in an .xlsx file into a SQL Server table.
If in this scenario, please refer to the following tips:
The Foreach Loop container should be configured as shown below:
Enumerator: Foreach ADO.NET Schema Rowset Enumerator
Connection String: The OLE DB Connection String for the excel file.
Schema: Tables.
In the Variable Mapping, map the variable to Sheet_Name, and change the Index from 0 to 2.
The connection string for Excel Connection Manager is the original one, we needn’t make any change.
Change Table Name or View name to the variable Sheet_Name.
If you want to load data from multiple sheets in multiple .xlsx files into a SQL Server table, please refer to following thread:
http://stackoverflow.com/questions/7411741/how-to-loop-through-excel-files-and-load-them-into-a-database-using-ssis-package
Thanks,
Katherine Xiong
Katherine Xiong
TechNet Community Support

How can I load a .xlsx File into a SQL Server Table using a Foreach Loop Container in SSIS?

I know I've REALLY struggled with this before. I just don't understand why this has to be soooooo difficult.
I can very easily do a straight Data Pump of a .xlsX File into a SQL Server Table using a normal Excel Connection and a normal Excel Source...simply converting Unicode to DT_STR and then using an OLE DB Destination of the SQL Server Table.
If I want to make the SSIS Package a little more flexible by allowing multiple .xlsX spreadsheets to be pumped in by using a Foreach Loop Container, the whole SSIS Package seems to go to hell in a hand basket. I simply do the following...
Put the Data Flow Task within the Foreach Loop Container
Add the Variable Mapping Variable User::FilePath that I defined as a Variable and a string within the FOreach Loop Container
I change the Excel Connection and its Expression to be ExcelFilePath ==> @[User::FilePath]
I then try and change the Excel Source and its Data Access Mode to Table Name or view name variable and provide the Variable Name User::FilePath
And that's when I run into trouble...
Exception from HRESULT: 0xC02020E8
Error at Data Flow Task [Excel Source [56]]:SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occured. Error code: 0x80004005.
Error at Data Flow Task [Excel Source [56]]: Opening a rowset for "...(the EXACT Path and .xlsx File Name)...". Check that the object exists in the database. (And I know it's there!!!)
I don't understand by adding a Foreach Loop Container to try and make this as efficient as possible has caused such an error unless I'm overlooking something. I have even tried delaying my validations and that doesn't seem to help.
I have looked hard in Google and even YouTube to try and find a solution for this but for the life of me I cannot seem to find anything on pumping a .xlsX file into SQL Server using a Foreach Loop Container.
Can ANYONE please help me out here? I'm at the end of my rope trying to get this to work. I think the last time I was in this quandry, trying to pump a .xlsX File into a SQL Server Table using a Foreach Loop Container in SSIS, I actually wrote a C# Script
to write the contents of the .xlsX File into a .csv File and then Actually used the .csv File to pump the data into a SQL Server Table.
Thanks for your review and am hoping and praying for a reply and solution.

Hi ITBobbyP,
If I understand correctly, you want to load data from multiple sheets in an .xlsx file into a SQL Server table.
If in this scenario, please refer to the following tips:
The Foreach Loop container should be configured as shown below:
Enumerator: Foreach ADO.NET Schema Rowset Enumerator
Connection String: The OLE DB Connection String for the excel file.
Schema: Tables.
In the Variable Mapping, map the variable to Sheet_Name, and change the Index from 0 to 2.
The connection string for Excel Connection Manager is the original one, we needn’t make any change.
Change Table Name or View name to the variable Sheet_Name.
If you want to load data from multiple sheets in multiple .xlsx files into a SQL Server table, please refer to following thread:
http://stackoverflow.com/questions/7411741/how-to-loop-through-excel-files-and-load-them-into-a-database-using-ssis-package
Thanks,
Katherine Xiong
Katherine Xiong
TechNet Community Support

SSIS BULK INSERT unsing UNC inside of ForEach Loop Container Failed could not be opened. Operating system error code 5(Access is denied.)

Hi,
I am trying to figure out how to fix my problem
Error: Could not be opened. Operating system error code 5(Access is denied.)
Process Description:
Target Database Server Reside on different Server in the Network
SSIS Package runs from a Remote Server
SSIS Package use a ForEachLoop Container to loop into a directory to do Bulk Insert
SSIS Package use variables to specified the share location of the files using UNC like this
\\server\files
Database Service accounts under the Database is runing it has full permission on the share drive were the files reside.
In the Execution Results tab shows the prepare SQL statement for the BULK insert and I can run the same exact the bulk insert in SSMS without errors, from the Database Server and from the server were SSIS package is executed.
I am on a dead end and I don’t want to re-write SSIS to use Data Flow Task because is not flexible to update when metadata of the table changed.
Below post it has almost the same situation:
https://social.msdn.microsoft.com/Forums/sqlserver/en-US/8de13e74-709a-43a5-8be2-034b764ca44f/problem-with-bulk-insert-task-in-foreach-loop?forum=sqlintegrationservices

Insteresting how I fixed the issue, Adding the Application Name into the SQL OLAP Connection String Fixed the issue. I am not sure why SQL Server wasn't able to open the file remotely without this.

SSIS 2008 R2 Foreach Loop Container on ADO object in Endless Loop

SSIS 2008 R2
I have an object Variable populated by a SQL Query. I then attach it to a Foreach Loop Container with "Foreach ADO Enumerator" and define the variable mappings. Inside the Foreach Loop is a series of tasks. It appears that the
tasks are processing each of the rows in the Object but it also appears that the Foreach loop never stops but runs until I kill it. I was under the impression that the Foreach loop would stop when after the last row in the Object was consumed.
As it is now, the Foreach Loop is useless as it runs forever.
Any suggestions would be appreciated.
Thanks,
Michael
P.S. I have determined that the ADO object has a size or count limitation which if exceeded causes the Foreach Loop task to malfunction in both Studio and as an Agent job. I could not find a limit in MS docs, but there sure seems to be
a limit somewhere which everyone should be aware of. 5,000 rows exceeds the limit, in my case.

A typical memory limit rather, especially noticeable in dev env where it is 32 bit
http://www.codeguru.com/csharp/.net/article.php/c16921/Understanding-NET-application-options-for-32-and-64bit-systems.htm
Arthur My Blog

SSIS: ForEach Loop Container Enumerates Same File

Trying to loop an Excel Directory. Have it in a ForEach Loop Container. It keeps looping the same file the amount of times the file pattern recognized; i.e Loop starts at report with 10 lines, there are 7 files that need looped...How
do I get it to loop all the files that need enumerated rather than just the default file path in the connection string?

I have both path and file in foreach loop container...How many variables should I be using in my case? I have one (ExcelInput) declared in the foreach loop container and passing to Excel Connection string expression.
Hi jsorrisone,
One SSIS variable is enough in your scenario. In the Foreach Loop Container, we set the Enumerator to “Foreach File Enumerator”, specify the path of the source folder, keep the “Retrieve file name” option as “Fully qualified”, and store the full path of
the Excel file to the string type variable ExcelInput.
Then, add a Data Flow Task to the Foreach Loop Container, drag an Excel Source, create an Excel Connection Manager. After that, set the expressions for the Excel Connection Manager, and set the expression of “ExcelFilePath” property to @[User::ExcelInput].
Reference:
http://mikedavissql.com/2013/09/16/loop-through-excel-files-in-ssis/
Regards,
Mike Yin
TechNet Community Support

Auto increment variable in foreach loop in ssis

Similar Messages

Maybe you are looking for