SSIS project - read multiple flat files with different formats

hi all,
i need to import multiple flat files with different formats into different tables of the sql server database and not able to figure out the best way out in ssis to do so...
please advise the possible methods in ssis to do so and if possible the process which can be dynamic as file names or columns might change in future.

Hi AK1987,
To import flat files with dynamic columns, we can use Script Task inside a Foreach Loop Container to parse the first row of the flat file to get the columns names and save them into a .NET variable, then, we can create “Create Table” script based on this
variable, and then store the script into a SSIS package variable. After that, we create a staging table based on the package variable, load the flat file data to the staging table. Eventually, we load data from the staging table to the destination table. For
the detail steps, please walk through the following blog:
http://www.citagus.com/citagus/blog/importing-from-flat-file-with-dynamic-columns/ 
Regards,
Mike Yin
TechNet Community Support

Similar Messages

  • Using ssis import a multiple flat files with different mappings

    I have an scenario for import a file
    20 different flat file source and having XML mapping document in a table.
    get files from that path(i have path and file name extension in table)
    i have map this file based on the file name extension in dynamically
    create a 20 different staging table based on mapping in dynamically
    kindly help for this
     

    Hi Karthick,
    As Arthur said, if you don't want to hard code the data flow, you need to read XML from the SQL Server table and use Script Task to implement the dynamic columns mapping. To read XML from a SQL table, you can refer to:
    http://blog.sqlauthority.com/2009/02/13/sql-server-simple-example-of-reading-xml-file-using-t-sql/ 
    In the Script Task, we need to parse the first row of the flat file to get the columns information and save that to a .NET variable. Then, the variable will be used to create the query that will be used to create SQL table. For more information, please see:
    http://www.citagus.com/citagus/blog/importing-from-flat-file-with-dynamic-columns/ 
    Regards,
    Mike Yin
    TechNet Community Support

  • SSIS - Import Multiple flat files with different metadata

    Hi ,
    I have set of flat files with different metadata structure, I would like to load them into staging tables. 
    1. ) Can we load the flatfiles into the staging tables with out having multiple data flow task.
    2.) If possible , can we programmatically select the staging table based on the metadata of the flatfile and load them.
    Please advise.
    Thanks
    Thiya

    Nope in SSIS a data flow task needs to have a fixed metadata. So if your file metadata varies then best option would be use OPENROWSET syntax to pull the data and populate into your staging table. You may also use 
    SELECT .. INTO StagingTable ... FROM OPENROWSET (...)
    syntax to create staging table at runtime based on the file metadata
    http://sqlmate.wordpress.com/2012/08/09/use-your-text-csv-files-in-your-queries-via-openrowset/
    If you want to do this in SSIS you need to create data flow dynamically using script task and build the metadata
    see
    http://www.selectsifiso.net/?p=288
    Please Mark This As Answer if it helps to solve the issue Visakh ---------------------------- http://visakhm.blogspot.com/ https://www.facebook.com/VmBlogs

  • Too many flat files with different formats.

    Hi gurus,
    i have a headache problems. that is follows:
    We will upload data from flat files,but you know there are about 100 .xls files and with different formats.
    and as you know  the datasource of the flat file must correspond to every fields of the flat file.
    but you know there are 100 flat files and if we donot do any optimise,we will create about 100 transformation and 100 datasource to meet our requirements.
    but that seems impossible.
    is there any good idea to decrease the number of transfomation ?

    Hi AK1987,
    To import flat files with dynamic columns, we can use Script Task inside a Foreach Loop Container to parse the first row of the flat file to get the columns names and save them into a .NET variable, then, we can create “Create Table” script based on this
    variable, and then store the script into a SSIS package variable. After that, we create a staging table based on the package variable, load the flat file data to the staging table. Eventually, we load data from the staging table to the destination table. For
    the detail steps, please walk through the following blog:
    http://www.citagus.com/citagus/blog/importing-from-flat-file-with-dynamic-columns/ 
    Regards,
    Mike Yin
    TechNet Community Support

  • InfoPackage to load multiple flat file for different plant

    Hi
    There is a requirements to load the data from multiple flat files by different plant into BI which source data is coming from other system. Each flat file has it own naming convention.
    As data has to load via infopackage into the targeted cube. Then I have to create multiple infopackage in order to load the data according to the number of flat files that provided. However, new plant will be created due to expansion of bussiness. If that is the case, additional infopackage is required to create as and when there is a new plant. 
    Is there a way to create a infopackage to load the data from flat file by plant? Your advice is much appreciated.
    Thanks and regards
    Kang Ring

    Hi,
    The best way I can suggest is....
    1. Create a folder in AL11 for your requirements.
    2. In InfoPackage under Extraxction Tab you can have File Name , just side of this you can fine ABAP Routines ICon there you click on that and asks Name give some and then it will takes to your into ABAP Code there you can access the AL11 and then write code in such way that to take file irrespective of the File name, whatever the file is there in that Folder it ahoud take from that AL11 Folder path.
    3. Load the Data using PC.
    4.At the end of the PC Delete the File in AL11 folder , becasue we don't need it for next load.
    Else
    See the thread
    Re: Automation of load from application server
    Thanks
    Reddy
    Edited by: Surendra Reddy on Jun 4, 2010 11:03 AM

  • How to load a flat file with utf8 format in odi as source file?

    Hi All,
    Anybody knows how we can load a flat file with utf8 format in odi as source file.Please everybody knows to guide me.
    Regards,
    Sahar

    Could you explain which problem are you facing?
    Francesco

  • XML file with different format

    Hello
       I have a requirement where the customer needs XML file with different format.  How do I achieve the second format mentioned below instead of first format.
    Normally we create the XML file in the following format:
    - <Entries>
        - <Organization>
            <MemberName>0000000002</MemberName>
            <FullName>BAY GYNECOLOGICAL ASSOCIATES</FullName>
            <OrgId>0000000002</OrgId>
            <OrgType>CUST</OrgType>
       - <AssociatedToOrg>
            <Name>0002</Name>
       - </AssociatedToOrg>
      </Organization>
    Now the customer wants like the following format:
    <Organization MemberName="522_Community_Customer" FullName="522 Community Customer" OrgId="DEA123456" OrgIdType="DEA" >
          <AssociatedToOrg Name="MN"/>
        </Organization>
    Thanks
    Naga

    you can solve it with a XSL mapping. refer to the SDN to know how develop it
    /people/aleksey.popov2/blog/2010/01/26/consuming-webservices-with-tag-in-wsdl-using-xslt
    XSLT Editor for creating xlst mappings
    http://www.sdn.sap.com/irj/scn/index?rid=/library/uuid/190eb190-0201-0010-0ab3-e69f70b6c257
    http://wiki.sdn.sap.com/wiki/display/XI/XSLTMappingSteps
    http://wiki.sdn.sap.com/wiki/display/XI/FileTOFile-UsingXSLTmapping%28forBeginners%29
    http://wiki.sdn.sap.com/wiki/display/Snippets/ConvertflatXMLfiletonestedusing+XSLT
    http://wiki.sdn.sap.com/wiki/display/Java/MultiMappingwithJavaandXSLTmappings
    Edited by: Rodrigo Alejandro Pertierra on Oct 1, 2010 5:26 PM

  • Reading fixed length file with different record types

    Hi,
    I need to read a fixed-length file with different record types, but the record identifier is in 31st position and not in 1st position.
    But if I give 31 as position in File adpater wizard, BPEL takes whole 1-31 as identifier.
    How we need to read such files.
    Thanks
    Ravdeep

    hi ,
    u cannot use the default wzard for this
    use some thing like this nxsd:lookAhead="30" nxsd:lookFor="S"have a look at the below link it has some examples
    http://download.oracle.com/docs/cd/B31017_01/integrate.1013/b28994/nfb.htm

  • How to create a dynamic SSIS package for multiple flat file destinations

    Hi,
    I have to create a ssis package which has single data flow task and inside that I have 23 source (sql- select * from - statements)- destination (flat files, 23 distinct) connection.
    Now for each product I have to create separate SSIS package (i.e. if prod=abc then these read select * from abc_tables and 23 abc_ txt files)
    I want to do it dynamically, means only single package and inside that variables will take select * values for each source-dest connection (so i believe 23 variables) and same for destination flat files.
    Let me know. :)
    ANK HIT - if reply helps, please mark it as ANSWER or helpful post

    Sorry It seems you're contradicting yourself. you say I know my source and dest structure and the you're
    asking all I want is to have a dynamic structure
    what does that mean?
    and reading your next sentence
    I want to run a package for 5 products, instead of creating 5 ssis packages with 23 source- dest connection, I would
    like to have one with only 23 source- dest connections 
    What I feel is what you're looking for is to  have a looping structure to loop through each of the 5
    products.
    In that case what you could do is this
    1. Create a object variable in SSIS 
    2. Use a Execute SQL Task to populate the variable with all available products (I think you'll have a master table for that). Set ResultSet property to Full ResultSet and then in ResultSet tab map Object variable to 0 th index
    3. use a ForEachLoop container with ADO .NET recordset enumerator and map to object variable. Create a variable of datatype same as that of Product identifier field to get individual values out
    4. Inside loop create your data flow task with 23 source destination connection. In the query part use a parameter for product field and map it to the variable containing product value to get only data for the product.
    Please Mark This As Answer if it helps to solve the issue Visakh ---------------------------- http://visakhm.blogspot.com/ https://www.facebook.com/VmBlogs

  • Flat file with different  layouts in a single file

    Hello Friends,
    Good Morning.
    I am new to ODI - Oralce Data Integrator 11G.
    I have a flat file where I had to load the flat file data to a oracle DB ( 11G ).
    The flat file has 5 different layouts ( Data of 5 different suppliers coming as a single file and each layout is different ) and each layout corrresponds to one supplier information.
    I had to load each layout data into a different table using a single source file ( That has all supplier information).
    Please advice.
    Thanks/Kumar

    Ok below are the layouts.
    BASE record                                                                                                    
    DS01 record                                                                                                              
    RM01 record                                                                                                         
    CR01 record                                                       
    NW01 record                                                       
    Few more question to you.
    Are they always come in sequential like below ?
    BASE record                                                                                                    
    DS01 record                                                                                                              
    RM01 record                                                                                                         
    CR01 record                                                       
    NW01 record
    BASE record                                                                                                    
    DS01 record                                                                                                              
    RM01 record                                                                                                         
    CR01 record                                                       
    NW01 record
    BASE record                                                                                                    
    DS01 record                                                                                                              
    RM01 record                                                                                                         
    CR01 record                                                       
    NW01 record
    Do you want only insert or insert/update as well ?

  • Multiple flat files with Comma delimiter and Pipe Delimiter in the sub folders.

    Hi,
    I have a directory C:\doc\Outcomes\Health  --(This is the main path). 
    In the path above i have multiple subfolders like 
    A
    B
    C
    D
    Folder A & B have 20 flat files each which are comma separated and pipe delimiter files. 
    Folder C&D have 20 excel files each.
    1) So, In SSIS while looping through the subfolders how do i limit to loop only excel files one time and flat files one time.
    2) In folder A&B, how do i loop only Pipe delimiter files (neglecting comma saperated files). I want to loop only pipe delimiter files while using for each loop container.
    Thanks 

    Both are txt files, but the data inside the files is saperated by ',' and '|'. ( comma and pipe)
    Thats ok 
    If delimiters are not consistent you can use this method
    http://visakhm.blogspot.in/2014/07/ssis-tips-handling-inconsistent-text.html
    Please Mark This As Answer if it solved your issue
    Please Mark This As Helpful if it helps to solve your issue
    Visakh
    My MSDN Page
    My Personal Blog
    My Facebook Page

  • Read two lvm files with different length

    I'm trying to read to same graph two different measurements with same starting point, starting point is OK.
    But when the shorter file is coming to the end the VI stops with following message:
    "Possible reason(s):
    LabVIEW:  End of file encountered.
    =========================
    NI-488:  Invalid argument or arguments to function call."
    How can I come over this and get both measurements into same graph in their full length with same starting point?
    PS I have attached the VI file, a short and a long measurement file. Changed the file extension from LVM to TXT
    Attachments:
    Read_duo.vi ‏128 KB
    SE1100_ny_1.txt ‏249 KB
    SE1100_ref_1.txt ‏468 KB

    Hi,
    Apprently your LabVIEW is set with the option "enable automatic error handling", you can change this in tools >> options block diagram untick the option.
    Doing this will only get ride of the dialog when the error occurs but not get read of the error.
    So you are reading your files chunk by chunk in your while loop, so at some point you will get to the end of your files, when you do the VI you are using return an error 4 "end of file reached", from here you have different possibilities, if you want to be able to read files that have different length, here is a quick way to ignore the "end of file" error on both files independantly.
    Hope this helps
    When my feet touch the ground each morning the devil thinks "bloody hell... He's up again!"

  • Reading a text file with different delimiters using Scanner

    Hello,
    I'm trying to read text from a file using ", " as a delimiter between Strings using a Scanner. I tried reading one line and then using the split() method in the string class, split each string by it's delimiter. I tried using an ArrayList to store each split token and the compiler is giving me some sort of syntax error when I'm pretty sure its right. If help can be given, it would be greatly appreciated. Thank you
    Here's my code
    Scanner input = new Scanner(backupFile);
    input.useDelimiter("[, ]");
    while (input.hasNext()) {
    ArrayList<String> list = new ArrayList<String>();
    String s = input.nextLine();
    String delims = "[, ]";
    list.add(s.split(delims));
    for(int i = 0; i < list.size(); i++)
    System.out.println(list.get(i));
    list.clear();
    }

    s.split(delims) will return an array of strings, not a string. So you need to declare list as an array list of such arrays:ArrayList<String[]> list = new ArrayList<String[]>();(It's a good idea to copy and post compiler messages here, so no-one has to guess. Likewise indicate which line of your code they are referring to.)
    [Edit]If it only the contents of the array you wish to put in the array, then you can just say List list = Arrays.asList(s.split(delims));

  • Flat File with unequal spacing as delimiter

    I am trying to import multiple flat files with unequal spacing as delimiter in sql server using SSIS 2012, however when I import the file, all the data in the file is imported into only one column. None of the predefined delimiters in SSIS works with the
    way the flat file was configured. I read that I can use the script component in SSIS 2012 to create the additional columns and somehow manipulate the file using the spacing to assign a new delimiter, such as a common, but I can't find any resources on it.
    Anyone have any experience/knowledge of this situation can provide me with an example or some resources that can help me to solve this situation?
    Example of file format:
    26   SM FRIES         0      1     0       0.00        0.0
    30   PEPSI               0    22     0       0.00        0.0

    Looks like a fixed width file to me seeing the data posted
    If you can determine the width of the columns then you can use fixed width (or ragged right) option in flat file connection manager and identify the lengths of the individual fields to get values out.
    See this example
    http://gaurangpatel.net/importing-fixed-width-file-into-database-using-ssis-package
    Please Mark This As Answer if it solved your issue
    Please Vote This As Helpful if it helps to solve your issue
    Visakh
    My Wiki User Page
    My MSDN Page
    My Personal Blog
    My Facebook Page

  • 1 xml file to multiple xml files with 200 records in each

    Hello Experts,
    I have below scenario.
    Please suggest me which might be the best approch to work on.
    1) XML file to XML file
    I will have 1 pretty huge XML file which i need to break in multiple XML files with 200 records each.
      1) first approch i can have bpm in which i can split the file according to my requirement.
      2) Second approch i can create 2 scenarios in which 1st scenario will pick up XML file and create multiple flat files with File content conversion. Second scenario will pick up all these flat files and create XML files.
    2) XML file to XML file
    Or i can have multiple files with 1 record per file and i need to merge in multiple XML files with 200 records in each.
    So its kind of 1:N or M:N scenarios.
    Please tell me which is might be better performance and design wise.
    Or if you have any idea in any other way i can do this, then please reply as soon as possbile.
    Please tell me if you have OS command for the same or some script to run or anything which i can implement.
    Thanks,
    Hetal

    what is your senario? is it File to File?
    You can use multi mapping concept without BPM. You can handle the 200 records per message logic in the multimapping.
    Regards,
    Praveen Gujjeti.

Maybe you are looking for

  • How to re-install CS5 Design Premium after MacBook crash

    Hello All! My MacBook pro crashed after a failed Yosemite update and I have to erase my hard drive. When we re-installed it I had lost all of my applications. I cannot find my discs for CS5 but I have my serial number online. Can anyone please tell m

  • Safari will not load numerous pages

    Ever since the 10.4.9 update I've had trouble with Safari not loading certain pages. Yahoo!, YouTube and numerous other pages just will not load. The problem carries over to other browsers as well. I've tried numerous things to resolve this issue. Un

  • Missing files in SRDEMO ServiceLocator and UserInfo

    Hi there Using jdev 10.1.3.1 and working my way through the JavaEE_Tutorial[.pdf]. On page 67 it refers to two files: ServiceLocatorJava.txt and UserInfoJava.txt that contain the code for the two corresponding classes. I cannot find these anywhere an

  • Questions about Synchronization

    Hello. I have two questions I need to understand. 1) Is possible to synchronize two iPhones in the same computer using iTunes? 2) I'm using a computer to synchronize my iPhone and I need to change the computer, how can I do it? Is there an easy way t

  • New Idea For Comf

    I got an idea for easy use, Im using ZEN MICRO and searching music with only 300 songs takes forever.. My idea is that we will be able to choose the number skipping songs, normaly its one by one can creative make a choice for like by 5/0/20? It will