Importing From Flat File with Dynamic Columns

HI
I am using ssis 2008,i have folder in which I have Four(4) “.txt” files each file will have 2 columns(ID, NAME). I loaded 4
files in one destination, but today I receive one more “.txt” file here we have 3 columns (ID, NAME, JOB) how can I get a message new column will receive in source. And how can I create in extra column in my destination table dynamically …please help me

Hi Sasidhar,
You need a Script Task to read the names and number of columns in the first row of the flat file each time and store it in a variable, then create a staging table dynamically based on this variable and modify the destination table definition if one ore more
new columns need to be added, and then use the staging table to load the destination table. I am afraid there is no available working script for your scenario, and you need some .NET coding experience to achieve your goal. Here is an example you can refer
to:
http://www.citagus.com/citagus/blog/importing-from-flat-file-with-dynamic-columns/ 
Regards,
Mike Yin
TechNet Community Support

Similar Messages

  • Logic to upload file with dynamic columns

    hi
    in my requirement i hav given to add logic to upload file with dynamic columns so that this upload program can be reused.
    this way the program is flexible, irrespective of the number of columns in the file.
    can any one explain this?
    and let me know what actually i hav to do.

    Check the program and the dynamic column is in the col_pos internal table and in the routines get_structure onwards.
    https://www.sdn.sap.com/irj/sdn/wiki?path=/display/scm/dynamic%2bstructures%2band%2bcomponents
    cheers
    Aveek

  • SSIS - Import Multiple flat files with different metadata

    Hi ,
    I have set of flat files with different metadata structure, I would like to load them into staging tables. 
    1. ) Can we load the flatfiles into the staging tables with out having multiple data flow task.
    2.) If possible , can we programmatically select the staging table based on the metadata of the flatfile and load them.
    Please advise.
    Thanks
    Thiya

    Nope in SSIS a data flow task needs to have a fixed metadata. So if your file metadata varies then best option would be use OPENROWSET syntax to pull the data and populate into your staging table. You may also use 
    SELECT .. INTO StagingTable ... FROM OPENROWSET (...)
    syntax to create staging table at runtime based on the file metadata
    http://sqlmate.wordpress.com/2012/08/09/use-your-text-csv-files-in-your-queries-via-openrowset/
    If you want to do this in SSIS you need to create data flow dynamically using script task and build the metadata
    see
    http://www.selectsifiso.net/?p=288
    Please Mark This As Answer if it helps to solve the issue Visakh ---------------------------- http://visakhm.blogspot.com/ https://www.facebook.com/VmBlogs

  • Upload file with dynamic columns

    Hi,
    My requirement is, I have to upload a file with number of columns being dynamic.
    My Input file column number can vary based on the input data. For one entry it can have 10 columns of input and for another entry it can have just 5. First few columns are common for all entries and then from one particular column the next columns can vary based on the input data.
    I have come across FM 'ALSM_EXCEL_TO_INTERNAL_TABLE' , but for this FM we have to pass the scope of the excel, like the column and row details which is not the required thing in my case.
    Please let me know as to how to achieve the requirement of uploading a file with he number of columns varying.
    Thanks in advance.
    Regards,
    Dedeepya Thota

    Hi,
    Your saying in you excel sheet your entering the details.
    As you said, Suppose your data is 5 columns. And each entry various
    there are 3 records in excel sheet
    one entry contains 5 columns of data
    second entry contains only 2 columns of data
    third entry contains 4 columns of data
    As shown below
    Column1   Column2   Column3  Column4  Column5
      100       AA      AA1         AA2        AA3
      200       BB       BB2      
      300       CC       CC2           CC3
    Now your excel sheet contains data as shown above.
    When you upload this file to your internal table 3 entries will be uploaded and every column contains according to that column value in itab.
    In this case record second entry column3 contains value BB2 will store in itab-column3, But as per our requirement it should store in itab-column4. Same will happen for third entry also.
    In above case you report will not work.
    So wat you have to do is, definatly we know there is some maximum colums in entries.
    You fix for that maximum no.columns in excel and declare your itab.
    In excel you maintian the values according to the columns value which has to maintian.
    For example like below.
    Column1   Column2   Column3  Column4  Column5
      100        AA       AA1         AA2        AA3
      200        BB                        BB2
      300        CC                        CC2        CC3
    If some columns are not having data that will be empty. It wont be any problem. Just you have to maintain your template for data
    and according to your template you have declare itab in your report.
    Regards,
    Shankar.

  • Export query results to flat file with dynamic filename

    Hi
    Can anybody can point me how to dynamic export query serults set to for example txt file using process flows in OWB.
    Let say I have simple select query
    select * from table1 where daterange >= sysdate -1 and daterange < sysdate
    so query results will be different every day because daterange will be different. Also I would like to name txt file dynamicly as well
    eg. results_20090601.txt, results_20090602.txt, results_20090603.txt
    I cant see any activity in process editor to enter custom sql statment, like it is in MSSQL 2000 or 2005
    thanks in advance

    You can call existing procedures from a process flow the procedure can create the filename with whatever name you desire. OWB maps with file as target can also create a file with a dynamic name defined by an expression (see here ).
    Cheers
    David

  • Import data from txt-file with reading column names

    Hello,
    after changing the the export of my simulation programm I can't use the solution postet here, wich works well. (I branded the post as soluted thats why I start a ne post)
    Can anybode please tell me how to adept the code to import the new type of txt file. Here the name of the columns are in the first line aof the columns, like this (with ignoring the first lines).
    "TIME      "    "Sitzbasis_X"    "Sitzbasis_Y"    "Sitzbasis_Z"    "Achsklammer_Z"    "Antriebsachse_Z"    "Gegengewicht_Z"    "Lenkachse_Z"
    0.000000e+000    -5.472285e+003    -1.886681e+002    1.812099e+004    4.539113e+004    -1.252936e+005    -1.499764e+004    -9.806650e+003
    1.000000e-003    -1.722846e+003    -1.141275e+002    8.443419e+003    2.475389e+004    -7.533712e+004    -1.138092e+004    -9.624729e+003
    2.000000e-003    -3.964053e+002    -7.694600e+001    3.186345e+003    1.399086e+004    -5.653730e+004    -9.956198e+003    -9.311346e+003
    I added an example file
    Thanks for Help
    Gabriel
    Solved!
    Go to Solution.
    Attachments:
    TXT-File.txt ‏115 KB

    Hello Gabriel
    You can use the DataPlugin Wizard (Navigator>>File>>DataPlugin Wizard) to define a new file import filter for your data. It will take only one minute.
     Please found the uri file I create for you. Copy the attached file to your disk, unpack it and double click the uri file.
    Then you can open your file by using "open with".
    Hope this help
    Attachments:
    TXT_File_DataPlugin.zip ‏4 KB

  • Using ssis import a multiple flat files with different mappings

    I have an scenario for import a file
    20 different flat file source and having XML mapping document in a table.
    get files from that path(i have path and file name extension in table)
    i have map this file based on the file name extension in dynamically
    create a 20 different staging table based on mapping in dynamically
    kindly help for this
     

    Hi Karthick,
    As Arthur said, if you don't want to hard code the data flow, you need to read XML from the SQL Server table and use Script Task to implement the dynamic columns mapping. To read XML from a SQL table, you can refer to:
    http://blog.sqlauthority.com/2009/02/13/sql-server-simple-example-of-reading-xml-file-using-t-sql/ 
    In the Script Task, we need to parse the first row of the flat file to get the columns information and save that to a .NET variable. Then, the variable will be used to create the query that will be used to create SQL table. For more information, please see:
    http://www.citagus.com/citagus/blog/importing-from-flat-file-with-dynamic-columns/ 
    Regards,
    Mike Yin
    TechNet Community Support

  • SSIS project - read multiple flat files with different formats

    hi all,
    i need to import multiple flat files with different formats into different tables of the sql server database and not able to figure out the best way out in ssis to do so...
    please advise the possible methods in ssis to do so and if possible the process which can be dynamic as file names or columns might change in future.

    Hi AK1987,
    To import flat files with dynamic columns, we can use Script Task inside a Foreach Loop Container to parse the first row of the flat file to get the columns names and save them into a .NET variable, then, we can create “Create Table” script based on this
    variable, and then store the script into a SSIS package variable. After that, we create a staging table based on the package variable, load the flat file data to the staging table. Eventually, we load data from the staging table to the destination table. For
    the detail steps, please walk through the following blog:
    http://www.citagus.com/citagus/blog/importing-from-flat-file-with-dynamic-columns/ 
    Regards,
    Mike Yin
    TechNet Community Support

  • Too many flat files with different formats.

    Hi gurus,
    i have a headache problems. that is follows:
    We will upload data from flat files,but you know there are about 100 .xls files and with different formats.
    and as you know  the datasource of the flat file must correspond to every fields of the flat file.
    but you know there are 100 flat files and if we donot do any optimise,we will create about 100 transformation and 100 datasource to meet our requirements.
    but that seems impossible.
    is there any good idea to decrease the number of transfomation ?

    Hi AK1987,
    To import flat files with dynamic columns, we can use Script Task inside a Foreach Loop Container to parse the first row of the flat file to get the columns names and save them into a .NET variable, then, we can create “Create Table” script based on this
    variable, and then store the script into a SSIS package variable. After that, we create a staging table based on the package variable, load the flat file data to the staging table. Eventually, we load data from the staging table to the destination table. For
    the detail steps, please walk through the following blog:
    http://www.citagus.com/citagus/blog/importing-from-flat-file-with-dynamic-columns/ 
    Regards,
    Mike Yin
    TechNet Community Support

  • Flat File with unequal spacing as delimiter

    I am trying to import multiple flat files with unequal spacing as delimiter in sql server using SSIS 2012, however when I import the file, all the data in the file is imported into only one column. None of the predefined delimiters in SSIS works with the
    way the flat file was configured. I read that I can use the script component in SSIS 2012 to create the additional columns and somehow manipulate the file using the spacing to assign a new delimiter, such as a common, but I can't find any resources on it.
    Anyone have any experience/knowledge of this situation can provide me with an example or some resources that can help me to solve this situation?
    Example of file format:
    26   SM FRIES         0      1     0       0.00        0.0
    30   PEPSI               0    22     0       0.00        0.0

    Looks like a fixed width file to me seeing the data posted
    If you can determine the width of the columns then you can use fixed width (or ragged right) option in flat file connection manager and identify the lengths of the individual fields to get values out.
    See this example
    http://gaurangpatel.net/importing-fixed-width-file-into-database-using-ssis-package
    Please Mark This As Answer if it solved your issue
    Please Vote This As Helpful if it helps to solve your issue
    Visakh
    My Wiki User Page
    My MSDN Page
    My Personal Blog
    My Facebook Page

  • Error in loading data from flat file....

    Hi,
    While loading the data from flat file , only one column is getting populated in the ODS and after that the sixth column is getting populated with the null values. Also when I save the changes in CSV file .. its not getting saved.
    Could you please tell me what could be the probable reason for this.
    Also what are the points we should keep in mind while loading from flat files.
    Regards,
    Jeetu

    Hi,
    You need to take care of -
    1. you flat file structure ( left to right columns) should match with the datasource ( top to down) column to column. if you don't wish to laod some column, leave it as blank column in flat file.
    2. your file should not be open when loading data
    3. make sure your transfer rules & update rules are defined properly.
    4. do the simulation in infopakage first & that will give you fair idea.
    5. to start with load data first in PSA Only, check & then take it forward.
    hope it helps
    regards
    Vikash

  • Extra column to be added while importing data from flat file to SQL Server

    I have 3 flat files with only one column in a table.
    The file names are bars, mounds & mini-bars.
    Table 'prd_type' has columns 'typeid' & 'typename' where typeid is auto-incremented and typename is bars, mounds & mini-bars.
    I Import data from 3 files to prd_details table. This table has columns 'pid', 'typeid' & 'pname' where pid is auto-incremented and pname is mapped to flat files and get info from them, now i wanted the typeid info to be received from prd_type table.
    Can someone please suggest me on this?

    You can get it as follows
    Assuming you've three separate data flow tasks for three files you can do this
    1. Add a new column to pipeline using derived column transformation and hardcode it to bars, mounds or mini-bars depending on the source
    2. Add a look task based on prd_type table. use query as
    SELECT typeid,typename
    FROM prd_type
    Use full cache option
    Add lookup based on derived column. new column -> prd_type.typename relationship. Select typeid as output column
    3. In the final OLEDB destination task map the typeid column to tables typeid column.
    In case you use single data flow task you need to include a logic based on filename or something to get the hardcoded type value as bars, mounds or mini-bars in the data pipeline
    Please Mark This As Answer if it helps to solve the issue Visakh ---------------------------- http://visakhm.blogspot.com/ https://www.facebook.com/VmBlogs

  • How to import a logo from flat file to SAP in SMARTFORM.

    how to import a logo from flat file to SAP in SMARTFORM. i need to know whether TIFF format is used and also the full steps.
    Thanks,

    Hi,
    You can not directly use a logo from a file into a Smart form. First you will have to upload it to SAP. This has to be done only once. Once it is imported into SAP then you can use it in smart form. To import a logo to SAP, goto SE78. Open the tree under GRAPHICS. Select the file type and click on the Import (first button on the toolbar, just before delete button). Select the logo file and give it some appropriate name along with description. Select appropriate type of the image and click Continue. Your logo will be uploaded into SAP from a file. Now in your smartform, Create a Graphic Node, give the name of the logo along with other properties. In this way you can import a logo from a flat file into SAP and consequently in a smart form.
    Reward points if the answer is helpful.
    Regards,
    Mukul

  • Need a script to import the data from flat file

    Hi Friends,
    Any one have any scripts to import the data from flat files into oracle database(Linux OS). I have to automate the script for every 30min to check any flat files in Incoming directory process them with out user interaction.
    Thanks.
    Srini

    Here is my init.ora file
    # $Header: init.ora 06-aug-98.10:24:40 atsukerm Exp $
    # Copyright (c) 1991, 1997, 1998 by Oracle Corporation
    # NAME
    # init.ora
    # FUNCTION
    # NOTES
    # MODIFIED
    # atsukerm 08/06/98 - fix for 8.1.
    # hpiao 06/05/97 - fix for 803
    # glavash 05/12/97 - add oracle_trace_enable comment
    # hpiao 04/22/97 - remove ifile=, events=, etc.
    # alingelb 09/19/94 - remove vms-specific stuff
    # dpawson 07/07/93 - add more comments regarded archive start
    # maporter 10/29/92 - Add vms_sga_use_gblpagfile=TRUE
    # jloaiza 03/07/92 - change ALPHA to BETA
    # danderso 02/26/92 - change db_block_cache_protect to dbblock_cache_p
    # ghallmar 02/03/92 - db_directory -> db_domain
    # maporter 01/12/92 - merge changes from branch 1.8.308.1
    # maporter 12/21/91 - bug 76493: Add control_files parameter
    # wbridge 12/03/91 - use of %c in archive format is discouraged
    # ghallmar 12/02/91 - add global_names=true, db_directory=us.acme.com
    # thayes 11/27/91 - Change default for cache_clone
    # jloaiza 08/13/91 - merge changes from branch 1.7.100.1
    # jloaiza 07/31/91 - add debug stuff
    # rlim 04/29/91 - removal of char_is_varchar2
    # Bridge 03/12/91 - log_allocation no longer exists
    # Wijaya 02/05/91 - remove obsolete parameters
    # Example INIT.ORA file
    # This file is provided by Oracle Corporation to help you customize
    # your RDBMS installation for your site. Important system parameters
    # are discussed, and example settings given.
    # Some parameter settings are generic to any size installation.
    # For parameters that require different values in different size
    # installations, three scenarios have been provided: SMALL, MEDIUM
    # and LARGE. Any parameter that needs to be tuned according to
    # installation size will have three settings, each one commented
    # according to installation size.
    # Use the following table to approximate the SGA size needed for the
    # three scenarious provided in this file:
    # -------Installation/Database Size------
    # SMALL MEDIUM LARGE
    # Block 2K 4500K 6800K 17000K
    # Size 4K 5500K 8800K 21000K
    # To set up a database that multiple instances will be using, place
    # all instance-specific parameters in one file, and then have all
    # of these files point to a master file using the IFILE command.
    # This way, when you change a public
    # parameter, it will automatically change on all instances. This is
    # necessary, since all instances must run with the same value for many
    # parameters. For example, if you choose to use private rollback segments,
    # these must be specified in different files, but since all gc_*
    # parameters must be the same on all instances, they should be in one file.
    # INSTRUCTIONS: Edit this file and the other INIT files it calls for
    # your site, either by using the values provided here or by providing
    # your own. Then place an IFILE= line into each instance-specific
    # INIT file that points at this file.
    # NOTE: Parameter values suggested in this file are based on conservative
    # estimates for computer memory availability. You should adjust values upward
    # for modern machines.
    # You may also consider using Database Configuration Assistant tool (DBCA)
    # to create INIT file and to size your initial set of tablespaces based
    # on the user input.
    # replace DEFAULT with your database name
    db_name=DEFAULT
    db_files = 80 # SMALL
    # db_files = 400 # MEDIUM
    # db_files = 1500 # LARGE
    db_file_multiblock_read_count = 8 # SMALL
    # db_file_multiblock_read_count = 16 # MEDIUM
    # db_file_multiblock_read_count = 32 # LARGE
    db_block_buffers = 100 # SMALL
    # db_block_buffers = 550 # MEDIUM
    # db_block_buffers = 3200 # LARGE
    shared_pool_size = 3500000 # SMALL
    # shared_pool_size = 5000000 # MEDIUM
    # shared_pool_size = 9000000 # LARGE
    log_checkpoint_interval = 10000
    processes = 50 # SMALL
    # processes = 100 # MEDIUM
    # processes = 200 # LARGE
    parallel_max_servers = 5 # SMALL
    # parallel_max_servers = 4 x (number of CPUs) # MEDIUM
    # parallel_max_servers = 4 x (number of CPUs) # LARGE
    log_buffer = 32768 # SMALL
    # log_buffer = 32768 # MEDIUM
    # log_buffer = 163840 # LARGE
    # audit_trail = true # if you want auditing
    # timed_statistics = true # if you want timed statistics
    max_dump_file_size = 10240 # limit trace file size to 5 Meg each
    # Uncommenting the line below will cause automatic archiving if archiving has
    # been enabled using ALTER DATABASE ARCHIVELOG.
    # log_archive_start = true
    # log_archive_dest = disk$rdbms:[oracle.archive]
    # log_archive_format = "T%TS%S.ARC"
    # If using private rollback segments, place lines of the following
    # form in each of your instance-specific init.ora files:
    # rollback_segments = (name1, name2)
    # If using public rollback segments, define how many
    # rollback segments each instance will pick up, using the formula
    # # of rollback segments = transactions / transactions_per_rollback_segment
    # In this example each instance will grab 40/5 = 8:
    # transactions = 40
    # transactions_per_rollback_segment = 5
    # Global Naming -- enforce that a dblink has same name as the db it connects to
    global_names = TRUE
    # Edit and uncomment the following line to provide the suffix that will be
    # appended to the db_name parameter (separated with a dot) and stored as the
    # global database name when a database is created. If your site uses
    # Internet Domain names for e-mail, then the part of your e-mail address after
    # the '@' is a good candidate for this parameter value.
    # db_domain = us.acme.com      # global database name is db_name.db_domain
    # FOR DEVELOPMENT ONLY, ALWAYS TRY TO USE SYSTEM BACKING STORE
    # vms_sga_use_gblpagfil = TRUE
    # FOR BETA RELEASE ONLY. Enable debugging modes. Note that these can
    # adversely affect performance. On some non-VMS ports the db_block_cache_*
    # debugging modes have a severe effect on performance.
    #_db_block_cache_protect = true # memory protect buffers
    #event = "10210 trace name context forever, level 2" # data block checking
    #event = "10211 trace name context forever, level 2" # index block checking
    #event = "10235 trace name context forever, level 1" # memory heap checking
    #event = "10049 trace name context forever, level 2" # memory protect cursors
    # define parallel server (multi-instance) parameters
    #ifile = ora_system:initps.ora
    # define two control files by default
    control_files = (ora_control1, ora_control2)
    # Uncomment the following line if you wish to enable the Oracle Trace product
    # to trace server activity. This enables scheduling of server collections
    # from the Oracle Enterprise Manager Console.
    # Also, if the oracle_trace_collection_name parameter is non-null,
    # every session will write to the named collection, as well as enabling you
    # to schedule future collections from the console.
    # oracle_trace_enable = TRUE
    # Uncomment the following line, if you want to use some of the new 8.1
    # features. Please remember that using them may require some downgrade
    # actions if you later decide to move back to 8.0.
    #compatible = 8.1.0
    Thanks.
    Srini

  • Ssis - import data from flat file to table (sql server 2012)

    i have create a ssis for importing data from flat file to table in sql server 2012.
    but i got the below error for some column in data flow task.
    error: cannot processed because than one code page (950 and 1252)
    anyone helps~

    Hi,
    The issue occurs because the source flat file uses ANSI/OEM – Tranditional Chinese Big5 encoding. When processing the source file, the flat file connection manager uses code page 950 for the columns. Because SQL Server uses code page to perform conversions
    between non-Unicode data and Unicode data, the data in the code page 950 based input columns cannot be loaded to code page 1252 based destination columns. To resolve the issue, you need to load the data into a SQL Server destination table that includes Unicode
    columns (nchar or nvarchar), and convert the input columns to Unicode columns via Data Conversion or the Advanced Editor for the Flat File Source at the same time.
    Another option that may not be that practical is to create a new database based on the Chinese_Taiwan_Stroke_BIN collation, and load the data to non-Unicode columns directly.
    Reference:
    http://social.technet.microsoft.com/Forums/windows/en-US/f939e3ba-a47e-43b9-88c3-c94bdfb7da58/forum-faq-how-to-fix-the-error-the-column-xx-cannot-be-processed-because-more-than-one-code-page?forum=sqlintegrationservices 
    Regards,
    Mike Yin
    TechNet Community Support

Maybe you are looking for