SSIS - Import Multiple flat files with different metadata
Hi ,
I have set of flat files with different metadata structure, I would like to load them into staging tables.
1. ) Can we load the flatfiles into the staging tables with out having multiple data flow task.
2.) If possible , can we programmatically select the staging table based on the metadata of the flatfile and load them.
Please advise.
Thanks
Thiya
Nope in SSIS a data flow task needs to have a fixed metadata. So if your file metadata varies then best option would be use OPENROWSET syntax to pull the data and populate into your staging table. You may also use
SELECT .. INTO StagingTable ... FROM OPENROWSET (...)
syntax to create staging table at runtime based on the file metadata
http://sqlmate.wordpress.com/2012/08/09/use-your-text-csv-files-in-your-queries-via-openrowset/
If you want to do this in SSIS you need to create data flow dynamically using script task and build the metadata
see
http://www.selectsifiso.net/?p=288
Please Mark This As Answer if it helps to solve the issue Visakh ---------------------------- http://visakhm.blogspot.com/ https://www.facebook.com/VmBlogs
Similar Messages
-
SSIS project - read multiple flat files with different formats
hi all,
i need to import multiple flat files with different formats into different tables of the sql server database and not able to figure out the best way out in ssis to do so...
please advise the possible methods in ssis to do so and if possible the process which can be dynamic as file names or columns might change in future.Hi AK1987,
To import flat files with dynamic columns, we can use Script Task inside a Foreach Loop Container to parse the first row of the flat file to get the columns names and save them into a .NET variable, then, we can create “Create Table” script based on this
variable, and then store the script into a SSIS package variable. After that, we create a staging table based on the package variable, load the flat file data to the staging table. Eventually, we load data from the staging table to the destination table. For
the detail steps, please walk through the following blog:
http://www.citagus.com/citagus/blog/importing-from-flat-file-with-dynamic-columns/
Regards,
Mike Yin
TechNet Community Support -
Using ssis import a multiple flat files with different mappings
I have an scenario for import a file
20 different flat file source and having XML mapping document in a table.
get files from that path(i have path and file name extension in table)
i have map this file based on the file name extension in dynamically
create a 20 different staging table based on mapping in dynamically
kindly help for this
Hi Karthick,
As Arthur said, if you don't want to hard code the data flow, you need to read XML from the SQL Server table and use Script Task to implement the dynamic columns mapping. To read XML from a SQL table, you can refer to:
http://blog.sqlauthority.com/2009/02/13/sql-server-simple-example-of-reading-xml-file-using-t-sql/
In the Script Task, we need to parse the first row of the flat file to get the columns information and save that to a .NET variable. Then, the variable will be used to create the query that will be used to create SQL table. For more information, please see:
http://www.citagus.com/citagus/blog/importing-from-flat-file-with-dynamic-columns/
Regards,
Mike Yin
TechNet Community Support -
Too many flat files with different formats.
Hi gurus,
i have a headache problems. that is follows:
We will upload data from flat files,but you know there are about 100 .xls files and with different formats.
and as you know the datasource of the flat file must correspond to every fields of the flat file.
but you know there are 100 flat files and if we donot do any optimise,we will create about 100 transformation and 100 datasource to meet our requirements.
but that seems impossible.
is there any good idea to decrease the number of transfomation ?Hi AK1987,
To import flat files with dynamic columns, we can use Script Task inside a Foreach Loop Container to parse the first row of the flat file to get the columns names and save them into a .NET variable, then, we can create “Create Table” script based on this
variable, and then store the script into a SSIS package variable. After that, we create a staging table based on the package variable, load the flat file data to the staging table. Eventually, we load data from the staging table to the destination table. For
the detail steps, please walk through the following blog:
http://www.citagus.com/citagus/blog/importing-from-flat-file-with-dynamic-columns/
Regards,
Mike Yin
TechNet Community Support -
Importing From Flat File with Dynamic Columns
HI
I am using ssis 2008,i have folder in which I have Four(4) “.txt” files each file will have 2 columns(ID, NAME). I loaded 4
files in one destination, but today I receive one more “.txt” file here we have 3 columns (ID, NAME, JOB) how can I get a message new column will receive in source. And how can I create in extra column in my destination table dynamically …please help meHi Sasidhar,
You need a Script Task to read the names and number of columns in the first row of the flat file each time and store it in a variable, then create a staging table dynamically based on this variable and modify the destination table definition if one ore more
new columns need to be added, and then use the staging table to load the destination table. I am afraid there is no available working script for your scenario, and you need some .NET coding experience to achieve your goal. Here is an example you can refer
to:
http://www.citagus.com/citagus/blog/importing-from-flat-file-with-dynamic-columns/
Regards,
Mike Yin
TechNet Community Support -
InfoPackage to load multiple flat file for different plant
Hi
There is a requirements to load the data from multiple flat files by different plant into BI which source data is coming from other system. Each flat file has it own naming convention.
As data has to load via infopackage into the targeted cube. Then I have to create multiple infopackage in order to load the data according to the number of flat files that provided. However, new plant will be created due to expansion of bussiness. If that is the case, additional infopackage is required to create as and when there is a new plant.
Is there a way to create a infopackage to load the data from flat file by plant? Your advice is much appreciated.
Thanks and regards
Kang RingHi,
The best way I can suggest is....
1. Create a folder in AL11 for your requirements.
2. In InfoPackage under Extraxction Tab you can have File Name , just side of this you can fine ABAP Routines ICon there you click on that and asks Name give some and then it will takes to your into ABAP Code there you can access the AL11 and then write code in such way that to take file irrespective of the File name, whatever the file is there in that Folder it ahoud take from that AL11 Folder path.
3. Load the Data using PC.
4.At the end of the PC Delete the File in AL11 folder , becasue we don't need it for next load.
Else
See the thread
Re: Automation of load from application server
Thanks
Reddy
Edited by: Surendra Reddy on Jun 4, 2010 11:03 AM -
Multiple flat files with Comma delimiter and Pipe Delimiter in the sub folders.
Hi,
I have a directory C:\doc\Outcomes\Health --(This is the main path).
In the path above i have multiple subfolders like
A
B
C
D
Folder A & B have 20 flat files each which are comma separated and pipe delimiter files.
Folder C&D have 20 excel files each.
1) So, In SSIS while looping through the subfolders how do i limit to loop only excel files one time and flat files one time.
2) In folder A&B, how do i loop only Pipe delimiter files (neglecting comma saperated files). I want to loop only pipe delimiter files while using for each loop container.
ThanksBoth are txt files, but the data inside the files is saperated by ',' and '|'. ( comma and pipe)
Thats ok
If delimiters are not consistent you can use this method
http://visakhm.blogspot.in/2014/07/ssis-tips-handling-inconsistent-text.html
Please Mark This As Answer if it solved your issue
Please Mark This As Helpful if it helps to solve your issue
Visakh
My MSDN Page
My Personal Blog
My Facebook Page -
Flat file with different layouts in a single file
Hello Friends,
Good Morning.
I am new to ODI - Oralce Data Integrator 11G.
I have a flat file where I had to load the flat file data to a oracle DB ( 11G ).
The flat file has 5 different layouts ( Data of 5 different suppliers coming as a single file and each layout is different ) and each layout corrresponds to one supplier information.
I had to load each layout data into a different table using a single source file ( That has all supplier information).
Please advice.
Thanks/KumarOk below are the layouts.
BASE record
DS01 record
RM01 record
CR01 record
NW01 record
Few more question to you.
Are they always come in sequential like below ?
BASE record
DS01 record
RM01 record
CR01 record
NW01 record
BASE record
DS01 record
RM01 record
CR01 record
NW01 record
BASE record
DS01 record
RM01 record
CR01 record
NW01 record
Do you want only insert or insert/update as well ? -
Flat File with unequal spacing as delimiter
I am trying to import multiple flat files with unequal spacing as delimiter in sql server using SSIS 2012, however when I import the file, all the data in the file is imported into only one column. None of the predefined delimiters in SSIS works with the
way the flat file was configured. I read that I can use the script component in SSIS 2012 to create the additional columns and somehow manipulate the file using the spacing to assign a new delimiter, such as a common, but I can't find any resources on it.
Anyone have any experience/knowledge of this situation can provide me with an example or some resources that can help me to solve this situation?
Example of file format:
26 SM FRIES 0 1 0 0.00 0.0
30 PEPSI 0 22 0 0.00 0.0Looks like a fixed width file to me seeing the data posted
If you can determine the width of the columns then you can use fixed width (or ragged right) option in flat file connection manager and identify the lengths of the individual fields to get values out.
See this example
http://gaurangpatel.net/importing-fixed-width-file-into-database-using-ssis-package
Please Mark This As Answer if it solved your issue
Please Vote This As Helpful if it helps to solve your issue
Visakh
My Wiki User Page
My MSDN Page
My Personal Blog
My Facebook Page -
1 xml file to multiple xml files with 200 records in each
Hello Experts,
I have below scenario.
Please suggest me which might be the best approch to work on.
1) XML file to XML file
I will have 1 pretty huge XML file which i need to break in multiple XML files with 200 records each.
1) first approch i can have bpm in which i can split the file according to my requirement.
2) Second approch i can create 2 scenarios in which 1st scenario will pick up XML file and create multiple flat files with File content conversion. Second scenario will pick up all these flat files and create XML files.
2) XML file to XML file
Or i can have multiple files with 1 record per file and i need to merge in multiple XML files with 200 records in each.
So its kind of 1:N or M:N scenarios.
Please tell me which is might be better performance and design wise.
Or if you have any idea in any other way i can do this, then please reply as soon as possbile.
Please tell me if you have OS command for the same or some script to run or anything which i can implement.
Thanks,
Hetalwhat is your senario? is it File to File?
You can use multi mapping concept without BPM. You can handle the 200 records per message logic in the multimapping.
Regards,
Praveen Gujjeti. -
Loading Multiple Flat File in to BI System with one InfoPackage
Greetings,
i have developed a routine to Load Multiple Flate File in one InfoPackage, but it doesn't work. Only the Last File are loaded. Who can Help me?
Loading 'R:\Verzeichnis \ing_wan_b_002.fall.csv' to 'R:\Verzeichnis \ing_wan_b_240*.fall.csv'
Routine:
DATA: VerzUndDateiname TYPE string value 'R:\Verzeichnis \ing_wan_b_000.fall.csv',
VerzUndDateinameKomplett TYPE string,
count TYPE i value 2,
countN(3) TYPE n.
WHILE count <= 240.
countN = count.
VerzUndDateinameKomplett = VerzUndDateiname.
REPLACE '000' WITH zaehlerN
INTO VerzUndDateinameKomplett.
p_filename = VerzUndDateinameKomplett.
count = count + 1.
ENDWHILE.
Best Regards
Jens
Edited by: JB6493 on May 18, 2009 1:03 PM
Edited by: JB6493 on May 18, 2009 1:07 PMHello Jens,
you have to process the InfoPackage 239 times. Your routine would be executed once during one processing of the InfoPackage. This would run the WHILE statement and end up with the last filename after 239 iterations.
Either try to concatenate the files you want to load (if possible) and load them in one go. Alternatively you could try to use a process chain to run the InfoPackage 239 times.
Kind regards,
Christoph -
Create a flat file with multiple characters for enclosures
Hello,
we use OWB 11g2 (11.2.02).
Now we try to create a flat file with multiple characters for enclosures. The manual wrote:
"Enclosures (Left and Right): Some delimited files contain enclosures that denote text strings within a field. If the file contains enclosures, enter an enclosure character in the text box or select one from the list. The list displays commonenclosures. However, you may enter any character. The default for both the left and right enclosure is the double quotation mark ("). You can specify multiple characters and hexadecimal characters as field enclosures."
But it will not work. The OWB use the first character from the left enclosure definition as left enclosure and the second one as right enclosure !?!
Did anyone know this behavior? Is there a solution for this problem?
Thanks and regards
NorbertHI Raghu,
Use the function module 'GUI_UPLOAD'.
In that you have to specify the field_separator value = 'X' in export section.
Regards,
S.C.K -
How to import flat file with date in filename on a regular basis
Hi,
Using OWB 11gR1
I have a file that will be delivered to an FTP each night with the date in the filename having the form YYYYMMDD-FILE.txt (ex: 20100326-FILE.txt) that I want to import to an external table.
Now I've set up the import to the external table but am only able to import files that I specify the name for exactly. I've tried pointing to filenames such as "*-FILE.txt" and "%-FILE.txt" but that only results in errors.
It must be possible to automatically import files with different filenames but the same structure, or isn't it? Could anyone help me solve this it'd be greatly appreciated.
Thank you in advance.Hi
For dynamic files you can;
1. either do the DDL on the external table to point to the file with the changing name
2Copy the file to a fixed name before using the external table/maps
3. Use the preprocessor to cat/pipe these files for the external table. See the post here http://blogs.oracle.com/warehousebuilder/2009/06/file_staging_using_external_table_preprocessor.html it shows using gunzip but could simply be doing 'cat' on a bunch of files to standard output
Cheers
David -
Flat File with multiple record types (OWB 10.2.0.2)
Hi!
I`m using OWB 10.2.0.2 and I`m trying to load a flat file with multiple record types, using SQL LOADER.
In the flat file editor in the Record tab, I`ve set the type values and the corresponding record names like this:
Type Value Record Name
======== ===========
T TRAILER
0 DETAILS
1 DETAILS
2 DETAILS
When using this flat file in a mapping to load the data in a staging table, the generated code looks like this:
INTO TABLE TRAILER
TRUNCATE
REENABLE DISABLED_CONSTRAINTS
WHEN (1:1) = 'T'
INTO TABLE DETAILS
APPEND
REENABLE DISABLED_CONSTRAINTS
WHEN (1:1) = '0,1,2'
The above clause (WHEN (1:1) = '0,1,2') is wrong as I expect one "INTO TABLE..." clause for each record type.
Could this be a bug or am I doing something wrong?
Thanks a lot for your help,
YorgosWe`re using two target tables, one for the trailer record and the other for the details records.
We are facing this problem from the moment we upgraded from OWB 10.1 to OWB 10.2.0.2, so we think it must be something with the way the sql loader code is generated in the new version.
As our data sources are mainly flat files coming from mainframes, this is a huge problem for us. We even asked an expert in DW from Oracle to help us on this, but still haven`t found a solution.
Is there any workaround for this or should we forget sql loader and go with an external tables + custom PL/SQL code solution?
Your help is greatly appreciated Jean-Pierre.
Regards,
Yorgos -
Re: Upload a flat file with multiple field seperators
HI Experts,
In my reports program iam trying to upload the data with in flat file with multiple seperators
but data is not uploading can ypu please suggest the solution.
Is there any function module to upload with multiple field seperators
Example of flat file:
E1234 JOHN BB HYD IN EN
E1234,BURE,BH,DSNR,IN,EN
E1236 KIWI KW HYT IN EN
E1237;BRAD;BW;MSRD;IN;EN
Thanks & Regards,
Raghusagar.GHI Raghu,
Use the function module 'GUI_UPLOAD'.
In that you have to specify the field_separator value = 'X' in export section.
Regards,
S.C.K
Maybe you are looking for
-
Adding pagination in report build from PL/SQL dynamic content using htp.p
Hi, I have a requirement for which i used PL/SQL dynamic content to build my report and i displayed my report using a cursor and loop by HTP.P function. Now problem is I have report with more than 500 rows and I want to add pagination concept for thi
-
HT1446 How do we charge MacBook Pro 13 inch?
I don't know how to charge it. Can anyone help me?
-
Macbook not starting when in sleep mode
So my white 13 inch macbook was in sleep mode for a couple hours and usually I just press either the enter/return button or the power/startup button to wake the computer but for the past two times, it wasn't starting up unless I held the power/startu
-
Exchange 2013 disconnects externally after every hour but works fine internally
Hello All, I have a very strange problem in my office. At the new years eve, there was a disconnection in our Internet from our ISP and everything came back online again after 1 hour. before this happened everything was working fine such as externall
-
Archive application logs - BC_SBAL
Hi We have a lot of entries in our BALHDR table that we want to archive. Problem is that millions of these are registered with expiry date = 31.12.9999. How can we archive these entries? Note 195157 describes deletion of entries - that is not an opti