UnPivot in the foreach loop container

Hi
I would like to "unpivot" SEQUENTIALLY each row in a table
Before the pivot task in the foreach loop container how to read sequentially each row ?
1/ from a PK ?
2/ from parameters with a WHERE field1 = ? and field2 = ? and field3=?....
These parameters have been set up prevoiouly in the loop
3/ Others ? ...
Thx for your help

Hi Gilles,
To achieve your goal, you can simply use the Unpivot Transformation or UNPIVOT statement in T-SQL.
If you want to use Unpivot Transform, it should be configured as follows:
If you want to use T-SQL, the query is like below:
SELECT u.Name, u.Category
FROM [dbo].[MyTable]
UNPIVOT
(Name for Category in (Field1, Field2, Field3)) u
References:
http://dinesql.blogspot.in/2011/08/pivot-and-unpivot-integration-services.html
http://technet.microsoft.com/en-us/library/ms177410(v=sql.105).aspx
Regards,
Mike Yin
If you have any feedback on our support, please click
here
Mike Yin
TechNet Community Support

Similar Messages

[Forum FAQ] How to calculate the total count of insert rows within a Foreach Loop Container in SSIS?

Introduction
We need to loop through all the flat files that have the same structure in a folder and import all the data to a single SQL Server table. How can we obtain the total count of the rows inserted to the destination SQL Server table?
Solution
We can use Execute SQL Task or Script Task to aggregate the row count increment for each iteration of the Foreach Loop Container. The following steps are the preparations before we add the Execute SQL Task or Script Task:
Create a String type variable FilePath, two Int32 type variables InsertRowCnt and TotalRowCnt.
Drag a Foreach Loop Container to the Control Flow design surface, set the Enumerator to “Foreach File Enumerator”, specify the source folder and the files extension, and set the “Retrieve file name” option to “Fully qualified”.
On the “Variable Mappings” tab of the container, map the variable FilePath to the collection value.
Drag a Data Flow Task to the container, in the Data Flow Task, add a Flat File Source, a Row Count Transformation, and an OLE DB Destination, and join them. Create a Flat File Connection Manager to connect to one of the flat files, and then configure the
Flat File Source as well as the OLE DB Destination adapter. Set the variable for the Row Count Transformation to “User::InsertRowCnt”.
Open the Property Expressions Editor for the Flat File Connection Manager, and set the expression of “ConnectionString” property to
“@[User::FilePath]”.
(I) Execute SQL Task Method:
In the Control Flow, drag an Execute SQL Task under the Data Flow Task and join them.
Create one or using any one existing OLE DB Connection Manager for the Execute SQL Task, set the “ResultSet” option to “Single row”, and then set the “SQLStatement” property to:
DECLARE @InsertRowCnt INT,
               @TotalRowCnt INT
SET @InsertRowCnt=?
SET @TotalRowCnt=?
SET @TotalRowCnt=@InsertRowCnt+@TotalRowCnt
SELECT TotalRowCnt=@TotalRowCnt
On to parameter 1.
On the “Result Set” tab of the Execute SQL Task, map result 0 to variable “User::TotalRowCnt”.
(II) Script Task Method:
In the Control Flow, drag a Script Task under the Data Flow Task and join them.
In the Script Task, select variable InsertRowCnt for “ReadOnlyVariables” option, and select variable TotalRowCnt for “ReadWriteVariables”.
Edit the Main method as follows (C#):
public void Main()
// TODO: Add your code here
int InsertRowCnt = Convert.ToInt32(Dts.Variables["User::InsertRowCnt"].Value.ToString()
int TotalRowCnt = Convert.ToInt32(Dts.Variables["User::TotalRowCnt"].Value.ToString());
TotalRowCnt = TotalRowCnt + InsertRowCnt;
Dts.Variables["User::InsertRowCnt"].Value = TotalRowCnt;
Dts.TaskResult = (int)ScriptResults.Success;
          Or (VB)
          Public Sub Main()
        ' Add your code here
        Dim InsertRowCnt As Integer =
        Convert.ToInt32(Dts.Variables("User::InsertRowCnt").Value.ToString())
        Dim TotalRowCnt As Integer =
        Convert.ToInt32(Dts.Variables("User::TotalRowCnt").Value.ToString())
        TotalRowCnt = TotalRowCnt + InsertRowCnt
        Dts.Variables("User::TotalRowCnt").Value = TotalRowCnt
        Dts.TaskResult = ScriptResults.Success
       End Sub
Applies to
Microsoft SQL Server 2005
Microsoft SQL Server 2008
Microsoft SQL Server 2008 R2
Microsoft SQL Server 2012
Microsoft SQL Server 2014
Please click to vote if the post helps you. This can be beneficial to other community members reading the thread.

Hi ITBobbyP,
If I understand correctly, you want to load data from multiple sheets in an .xlsx file into a SQL Server table.
If in this scenario, please refer to the following tips:
The Foreach Loop container should be configured as shown below:
Enumerator: Foreach ADO.NET Schema Rowset Enumerator
Connection String: The OLE DB Connection String for the excel file.
Schema: Tables.
In the Variable Mapping, map the variable to Sheet_Name, and change the Index from 0 to 2.
The connection string for Excel Connection Manager is the original one, we needn’t make any change.
Change Table Name or View name to the variable Sheet_Name.
If you want to load data from multiple sheets in multiple .xlsx files into a SQL Server table, please refer to following thread:
http://stackoverflow.com/questions/7411741/how-to-loop-through-excel-files-and-load-them-into-a-database-using-ssis-package
Thanks,
Katherine Xiong
Katherine Xiong
TechNet Community Support

Can I append the file name from a Foreach Loop Container Enumerator into a SQL Server Statement?

I would like to pass back via an Email the name of the file that was successfully FTP'ed to a remote server. And I was hoping to do this via an Execute SQL Task with the following SQL Statement...but I just don't know if it will append the Foreach Loop Container
variable enumerator to the appended Email message that I am building in SQL Server or if I can and how I can...
I sort of get the feeling that I can try and do this via Dynamic SQL...I just don't know if by building dynamic SQL if my Foreach Loop Container enumerator variable will be appended to my Email Message via the SQL Server UPDATE
@[User::FTPFHFileName]
Or do I add a Data Flow Task and an OLE DB Command and pass the Foreach Loop Container enumerator variable to a Stored Procedure referenced in the OLE DB Command via a "?"
Thanks for your review and am hopeful for a reply.

Hello,
It seems that the issue had been solved and thanks for your sharing. It will be very beneficial for other community members who have similar questions.
I’d like to mark this issue as "Answered". Please also feel free to unmark the issue, with any new findings or concerns you may have.
Regards,
Katherine Xiong
Katherine Xiong
TechNet Community Support

How can I load a .xlsx File into a SQL Server Table using a Foreach Loop Container in SSIS?

I know I've REALLY struggled with this before. I just don't understand why this has to be soooooo difficult.
I can very easily do a straight Data Pump of a .xlsX File into a SQL Server Table using a normal Excel Connection and a normal Excel Source...simply converting Unicode to DT_STR and then using an OLE DB Destination of the SQL Server Table.
If I want to make the SSIS Package a little more flexible by allowing multiple .xlsX spreadsheets to be pumped in by using a Foreach Loop Container, the whole SSIS Package seems to go to hell in a hand basket. I simply do the following...
Put the Data Flow Task within the Foreach Loop Container
Add the Variable Mapping Variable User::FilePath that I defined as a Variable and a string within the FOreach Loop Container
I change the Excel Connection and its Expression to be ExcelFilePath ==> @[User::FilePath]
I then try and change the Excel Source and its Data Access Mode to Table Name or view name variable and provide the Variable Name User::FilePath
And that's when I run into trouble...
Exception from HRESULT: 0xC02020E8
Error at Data Flow Task [Excel Source [56]]:SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occured. Error code: 0x80004005.
Error at Data Flow Task [Excel Source [56]]: Opening a rowset for "...(the EXACT Path and .xlsx File Name)...". Check that the object exists in the database. (And I know it's there!!!)
I don't understand by adding a Foreach Loop Container to try and make this as efficient as possible has caused such an error unless I'm overlooking something. I have even tried delaying my validations and that doesn't seem to help.
I have looked hard in Google and even YouTube to try and find a solution for this but for the life of me I cannot seem to find anything on pumping a .xlsX file into SQL Server using a Foreach Loop Container.
Can ANYONE please help me out here? I'm at the end of my rope trying to get this to work. I think the last time I was in this quandry, trying to pump a .xlsX File into a SQL Server Table using a Foreach Loop Container in SSIS, I actually wrote a C# Script
to write the contents of the .xlsX File into a .csv File and then Actually used the .csv File to pump the data into a SQL Server Table.
Thanks for your review and am hoping and praying for a reply and solution.

Hi ITBobbyP,
If I understand correctly, you want to load data from multiple sheets in an .xlsx file into a SQL Server table.
If in this scenario, please refer to the following tips:
The Foreach Loop container should be configured as shown below:
Enumerator: Foreach ADO.NET Schema Rowset Enumerator
Connection String: The OLE DB Connection String for the excel file.
Schema: Tables.
In the Variable Mapping, map the variable to Sheet_Name, and change the Index from 0 to 2.
The connection string for Excel Connection Manager is the original one, we needn’t make any change.
Change Table Name or View name to the variable Sheet_Name.
If you want to load data from multiple sheets in multiple .xlsx files into a SQL Server table, please refer to following thread:
http://stackoverflow.com/questions/7411741/how-to-loop-through-excel-files-and-load-them-into-a-database-using-ssis-package
Thanks,
Katherine Xiong
Katherine Xiong
TechNet Community Support

Foreach Loop Container with a Data Flow Task looking for file from Connection Manager

So I have a Data Flow Task within a Foreach Loop Container. The Foreach Loop Container has a Variable Mapping of User:FileName to pass to the Data Flow Task.
The Data Flow Task has a Flat File Source since we're looking to process .csv Files. And the Flat File Source has a Flat File Connection Manager where I specified the File name when I created it. I thought you needed to do this even though it won't really
use it since it should be getting its File name from the Foreach Loop Container. But when attempting to execute, it is blowing up because it seems to be looking for my test file name that I indicated in the Flat File Connection Manager rather than the file
it should be trying to process in User:FileName from the Foreach Loop Container.
What am I doing wrong here??? I thought you needed to indicate a File name within the Flat File Connection Manager even though it really won't be using it.
Thanks for your review...I hope I've been clear...and am hopeful for a reply.
PSULionRP

The Flat File Connection manager's Connection String needs to be set to reference the variable used in the ForEach Loop:
Arthur My Blog

Dynamically specify folder files for Foreach File Enumerator in Foreach Loop Container and load into destination

hi,
I need to load data into tables before loading data I have several folders like a1, a2,a3... each folder 'll contain a several .csv files. I need to load those csv files data into destination based on folder wise
Thanks
AVS

you can use foreachloop for that. It has an option called recursive. So just point it to parent level folder and give recursive option and it will go through all folders
https://www.simple-talk.com/sql/ssis/ssis-basics-introducing-the-foreach-loop-container/
Please Mark This As Answer if it helps to solve the issue Visakh ---------------------------- http://visakhm.blogspot.com/ https://www.facebook.com/VmBlogs

SSIS: ForEach Loop Container Enumerates Same File

Trying to loop an Excel Directory. Have it in a ForEach Loop Container. It keeps looping the same file the amount of times the file pattern recognized; i.e Loop starts at report with 10 lines, there are 7 files that need looped...How
do I get it to loop all the files that need enumerated rather than just the default file path in the connection string?

I have both path and file in foreach loop container...How many variables should I be using in my case? I have one (ExcelInput) declared in the foreach loop container and passing to Excel Connection string expression.
Hi jsorrisone,
One SSIS variable is enough in your scenario. In the Foreach Loop Container, we set the Enumerator to “Foreach File Enumerator”, specify the path of the source folder, keep the “Retrieve file name” option as “Fully qualified”, and store the full path of
the Excel file to the string type variable ExcelInput.
Then, add a Data Flow Task to the Foreach Loop Container, drag an Excel Source, create an Excel Connection Manager. After that, set the expressions for the Excel Connection Manager, and set the expression of “ExcelFilePath” property to @[User::ExcelInput].
Reference:
http://mikedavissql.com/2013/09/16/loop-through-excel-files-in-ssis/
Regards,
Mike Yin
TechNet Community Support

Is it possible to load multiple .xlsx files into a SQL Server Table using SSIS and a Foreach Loop Container when each Excel spreadsheet is comprised of two different worksheets

So we have an Invoice .xlsx File from a 3rd party vendor. It contains two worksheets..."Enrolled" and "Engaged". The data and data columns in each worksheet is different. Is it possible to loop through multiple .xlsx files using SSIS
and a Foreach Loop Container for each spreadsheet, and then another Foreach Loop Container to control each worksheet, and pump the Excel data into a SQL Server Table first for "Enrolled" and then for "Engaged"? How can I control the Foreach
Loop Container in SSIS to process ONLY the "Enrolled" worksheet first? And then the "Engaged" worksheet next?
I know I have multiples out here and I apologize for that...but right now it seems as though I take three steps forward and then two back.
Any help would be GREATLY appreciated!
Thanks in advance!

If the structure of the Excel sheets does not change from file to file then you can by having one ForEach Loop processing always the "Enrolled" sheet and another always the "Engaged" this is doable because the Excel OLEDB connector allows
to pick individual sheets, it is problematic therefore when sheet names themselves change.
MSDN has an example: https://msdn.microsoft.com/en-ca/library/ms345182.aspx
Arthur
MyBlog
Twitter

Using foreach loop container with file system task to rename and move files

the foreach loop container will not rename and move files for me
sukai

duplicate of
http://social.msdn.microsoft.com/Forums/sqlserver/en-US/8f2899f1-43b0-47e0-8bd0-e082989cdcb8/file-system-task-and-foreach-loop-container?forum=sqlintegrationservices
Please Mark This As Answer if it helps to solve the issue Visakh ---------------------------- http://visakhm.blogspot.com/ https://www.facebook.com/VmBlogs

Use a external parameter in a foreach loop container

Hi
I use a parameter P1 in a package
This package contains also a foreach loop container including a parameter P2 (P1 is on outside of the foreach loop container)
In this foreach loop container, I need to insert into a table T1 the parameter P1 and P2 from an execute sql task :
INSERT INTO T1 (F1, F2) VALUES (?,?)
My question : how to access to the P1 parameter in the container ?
Thx for your help

I assume when you say parameter its in term of USER Variables defined in packages if yes then its essentially a scope thing, just define the scope of variable to outside and you should be able to access it.
If you mean P1 is something you assign from the Foreach look then assign then store this parameter as variable then assign the variable as you want to use it.
Abhinav http://bishtabhinav.wordpress.com/

SSIS Variable Expression not Picking Up the value dynamically set from a Foreach Loop container

Hello, everyone,
I have three variables in my SSIS package:
1) ReportLoc that holds a file path (c:\test, not changed)
2) ReportFileName that holds a file name that get assigned in a Foreach Loop container (file1, file2, etc.)
3) OutputFilePath that is the full file pathname, set in the expression: @[User::LocalPath] + "\\" + @[User::ReportFileName]
All three variable have the package scope.
I expect OutputFilePath will be asigned values like c:\test\file1, c:\test\file2, etc.
However, it alwasy remains an empty string.
Anything I did wrong? Any help and information is greatly appreciated.
Regards

That's it -- I forgot to set Evaluate as Expression to true.
Thank you very much for your reply and solution! Really appreciate it!
Regards.

The property XMLDataVariable on the XML Source was empty - Case Foreach Loop Container

I have very simple XML source and OLE DB Source in Foreach Loop Container,
but seem to get error, because XML Source cannot find the XML.
"Error at Data Flow Task. The property XMLDataVariable on the XML Source was empty"
Variables:
FileExtention = *.xml
FilePath =
FolderPath = C:\Customer\XML
XML Source:
Data Access mode = XML file from variable
Variable name = User::filePath
Foreach Loop Container:
Enumerator = Foreach File Enumerator
Expression-Directory = @User:FolderPath
Expression-FileSpec = @User::FileExtension
Enumerator Folder = C:\Customer\XML
Files = *.xml
Variable mapping-User::FilePath = 0
Kenny_I

Have you filled the variable (filePath) with a default value? Otherwise the XML Source cannot validate. Alternative is look for a property "DelayValidation". In most cases this is a property of the connection manager, but you're not using
one for the xml source....
Please mark the post as answered if it answers your question | My SSIS Blog:
http://microsoft-ssis.blogspot.com |
Twitter

Strange issue with rights of account, used by SSIS (Foreach Loop Container does not return file names without Admin rights on server)

Hello everyone.
Faced very strange issue with account, which is used to run SSIS package.
The specific package uses Foreach Loop Container to retrieve file names within the specified folder, and put them into Import file task.
The package is set up to run under specific service account. This service account is given all permissions (Full control) to the folder where source files reside.
So the issue is: SSIS package fails to execute this task (Foreach Loop Container and then Import), and shows that no files are found in the directory (although files ARE there).
Once we're adding the service account into local Administrators group on the SQL Server, it works! Removing - does not work again. We cannot leave the service account as SQL server's admin as it's prohibited by our IT policies, and is just a bad practice.
Any ideas, please?
MCP

Here's the real log output:
Date 16.04.2014 12:47:09
Log Job History (RU-BW: Update)
Step ID 1
Server Server
Job Name RU-BW: Update
Step Name bw_import_cust_master_data
Duration 00:00:02
Sql Severity 0
Sql Message ID 0
Operator Emailed
Operator Net sent
Operator Paged
Retries Attempted 0
Message
Executed as user: service_account Microsoft (R) SQL Server Execute Package Utility Version 10.50.4286.0 for 64-bit Copyright (C) Microsoft Corporation 2010. All rights reserved. Started: 12:47:09 Error: 2014-04-16 12:47:11.45
Code: 0xC0202070 Source: bw_import_cust_master_data Connection manager "Input" Description: The file name property is not valid. The file name is a device or contains invalid characters. End Error Error:
2014-04-16 12:47:11.47 Code: 0xC0202070 Source: bw_import_cust_master_data Connection manager "Input" Description: The file name property is not valid. The file name is a device or contains invalid characters. End
Error Error: 2014-04-16 12:47:11.48 Code: 0xC0202070 Source: bw_import_cust_master_data Connection manager "Input" Description: The file name property is not valid. The file name is a device or contains invalid
characters. End Error Error: 2014-04-16 12:47:11.48 Code: 0xC020207E Source: Import file Flat File Source [1] Description: The file name is not valid. The file name is a device or contains invalid characters.
End Error Error: 2014-04-16 12:47:11.48 Code: 0xC004701A Source: Import file SSIS.Pipeline Description: component "Flat File Source" (1) failed the pre-execute phase and returned error code 0xC020207E.
End Error DTExec: The package execution returned DTSER_FAILURE (1). Started: 12:47:09 Finished: 12:47:11 Elapsed: 2.324 seconds. The package execution failed. The step failed.
MCP

ForEach Loop Container Mapping Variable From SSIS Package Not Working In SQL-Server StoredProcedure

I have an SSIS package that uses a ForEach Loop Container to enumerate Excel Files in a dir. I also have a Task Flow that inserts data from those Excel files into SQL-Server.
Im trying to insert the file names into a column into the same table in SQL-Server by using a mapping variable in my StoredProcedure.
Im having trouble with my MappingVariable at the end of the script with red squigglies. The following is my StoredProcedure script.
CREATE PROCEDURE [dbo].[Insert_F_STG_v2]
-- Add the parameters for the stored procedure here
@Hrs float,
@Type nvarchar(100),
@SN nvarchar(100),
@Op nvarchar(100),
@[USER::CurrentFileName]
AS
BEGIN
SET NOCOUNT ON;
INSERT INTO [CRM_RC].[dbo].[F_StgTbl]
[Hrs],
[Type],
[SN],
[Op],
[Report_Date]
VALUES
@Hrs ,
@Type,
@SN,
@Op,
@[USER::CurrentFileName]
END
The last @[USER::CurrentFileName] in the Values block at the bottom of the script is the one giving me issues.
The following is the error:
Msg 102, Level 15, State 1, Procedure Insert_F_STG_v2, Line 95
Incorrect syntax near 'USER::CurrentFileName'.

This seems to be the solution, but get the following exception:
[Derived Column [2]] Error: The "Derived Column" failed because truncation occurred, and the truncation row disposition on "Derived Column.Outputs[Derived Column Output].Columns[Derived Column 1]" specifies failure on truncation. A truncation error occurred
on the specified object of the specified component.
AND: [SSIS.Pipeline] Error: SSIS Error Code DTS_E_PROCESSINPUTFAILED. The ProcessInput method on component "Derived Column" (2) failed with error code 0xC020902A while processing input "Derived Column Input" (3). The identified component returned
an error from the ProcessInput method. The error is specific to the component, but the error is fatal and will cause the Data Flow task to stop running. There may be error messages posted before this with more information about the failure.

Excel Source cannot find sheet when using Foreach Loop Container

I have SQL Server 2012 SSIS. I need help with Foreach Loop container.
1) I have C:\\Excel\ folder and multiple Excel.xlsx files are stored there to be imported
2) I have Foreach Loop Container
-Foreach File Enumerator is selected
-Expressions are empty
-Folder is set as C:\\Excel\
-Files is *.*
-Variable is created. User::Filename, 0
2) I have created variable FileName, String,0
3) I have Excel Connection Manager
-ExcelFilePath = @[User::FileName]
4) I have data flow task with Excel Source and OLE DB Destination
Error occured with Execute:
[Excel Source [2]] Error: SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occurred. Error code: 0x80040E37.
[Excel Source [2]] Error: Opening a rowset for "SheetName$" failed. Check that the object exists in the database.
Kenny_I

Hi Kenny_I,
The issue occurs because you have not specified a valid value for the variable @FileName. The error persists even if we set the “DelayValidation” property of the Excel Connection Manager to True. After you assign a value like “C:\Excel\Test1.xlsx” (without
quotes) to the variable, the package should work fine.
Reference:
http://www.bidn.com/blogs/mikedavis/ssis/625/loop-through-excel-file-in-ssis
Regards,
Mike Yin
If you have any feedback on our support, please click
here
Mike Yin
TechNet Community Support

UnPivot in the foreach loop container

Similar Messages

Maybe you are looking for