SSIS 2008 – Read roughly 50 CSV files from a folder, create SQL table from them dynamically, and dump data.

Question

Hello everyone,
I’ve been assigned one requirement wherein I would like to read around 50 CSV files from a specified folder.
In step 1 I would like to create schema for this files, meaning take the CSV file one by one and create SQL table for it, if it does not exist at destination.
In step 2 I would like to append the data of these 50 CSV files into respective table.
In step 3 I would like to purge data older than a given date.
Please note, the data in these CSV files would be very bulky, I would like to know the best way to insert bulky data into SQL table.
Also, in some of the CSV files, there will be 4 rows at the top of the file which have the header details/header rows.
According to my knowledge I would be asked to implement this on SSIS 2008 but I’m not 100% sure for it.
So, please feel free to provide multiple approaches if we can achieve these requirements elegantly in newer versions like SSIS 2012.
Any help would be much appreciated.
Thanks,
Ankit
Thanks, <b>Ankit Shah</b> <hr> Inkey Solutions, India. <hr> Microsoft Certified Business Management Solutions Professionals <hr> http://ankit.inkeysolutions.com

Answer 1

Hello Harry and Aamir,
Thank you for the responses.
@Aamir, thank you for sharing the link, yes I'm going to use Script task to read header columns of CSV files, preparing one SSIS variable which will be having SQL script to create the required table with if exists condition inside script task itself.
I will be having "Execute SQL task" following the script task. And this will create the actual table for a CSV.
Both these components will be inside a for each loop container and execute all 50 CSV files one by one.
Some points to be clarified,
1. In the bunch of these 50 CSV files there will be some exception for which we first need to purge the tables and then insert the data. Meaning for 2 files out of 50, we need to first clean the tables and then perform data insert, while for the rest 48
files, they should be appended on daily basis.
Can you please advise what is the best way to achieve this requirement? Where should we configure such exceptional cases for the package?
2. For some of the CSV files we would be having more than one file with the same name. Like out of 50 the 2nd file is divided into 10 different CSV files. so in total we're having 60 files wherein the 10 out of 60 have repeated file names. How can we manage
this criteria within the same loop, do we need to do one more for each looping inside the parent one, what is the best way to achieve this requirement?
3. There will be another package, which will be used to purge data for the SQL tables. Meaning unlike the above package, this package will not run on daily basis. At some point we would like these 50 tables to be purged with older than criteria, say remove
data older than 1st Jan 2015. what is the best way to achieve this requirement?
Please know, I'm very new in SSIS world and would like to develop these packages for client using best package development practices.
Any help would be greatly appreciated.
Thanks, <b>Ankit Shah</b> <hr> Inkey Solutions, India. <hr> Microsoft Certified Business Management Solutions Professionals <hr> http://ankit.inkeysolutions.com
1. In the bunch of these 50 CSV files there will be some exception for which we first need to purge the tables and then insert the data. Meaning for 2 files out of 50, we need to first clean the tables and then perform
data insert, while for the rest 48 files, they should be appended on daily basis.
Can you please advise what is the best way to achieve this requirement? Where should we configure such exceptional cases for the package?
How can you identify these files? Is it based on file name or are there some info in the file which indicates
that it required a purge? If yes you can pick this information during file name or file data parsing step and set a boolean variable. Then in control flow have a conditional precedence constraint which will check the boolean variable and if set it will execute
a execte sql task to do the purge (you can use TRUNCATE TABLE or DELETE FROM TableName statements)
2. For some of the CSV files we would be having more than one file with the same name. Like out of 50 the 2nd file is divided into 10 different CSV files. so in total we're having 60 files wherein the 10 out of 60 have
repeated file names. How can we manage this criteria within the same loop, do we need to do one more for each looping inside the parent one, what is the best way to achieve this requirement?
The best way to achieve this is to append a sequential value to filename (may be timestamp) and then process
them in sequence. This can be done prior to main loop so that you can use same loop to process these duplicate filenames also. The best thing would be to use file creation date attribute value so that it gets processed in the right sequence. You can use a
script task to get this for each file as below
http://microsoft-ssis.blogspot.com/2011/03/get-file-properties-with-ssis.html
3. There will be another package, which will be used to purge data for the SQL tables. Meaning unlike the above package, this package will not run on daily basis. At some point we would like these 50 tables to be purged
with older than criteria, say remove data older than 1st Jan 2015. what is the best way to achieve this requirement?
You can use a SQL script for this. Just call a sql procedure
with a single parameter called @Date and then write logic like below
CREATE PROC PurgeTableData
@CutOffDate datetime
AS
DELETE FROM Table1 WHERE DateField < @CutOffDate;
DELETE FROM Table2 WHERE DateField < @CutOffDate;
DELETE FROM Table3 WHERE DateField < @CutOffDate;
GO
@CutOffDate which denote date from which older data have to be purged
You can then schedule this SP in a sql agent job to get executed based on your required frequency
Please Mark This As Answer if it solved your issue
Please Vote This As Helpful if it helps to solve your issue
Visakh
My Wiki User Page
My MSDN Page
My Personal Blog
My Facebook Page