SSIS best practice on importing External text files

Hi -
I am a fairly seasoned SSIS/ETL developer and I am struggling with the best architecure on how to import vendor files into a shared database.  I see there being 2 methods with in importing files, and I'm really wanting input from senior
level SSIS developers on their thoughts of importing vendor files, I see there being 2 methods:
- Set up the ETL to match the format of the file, and if the format is invalid, the entire ETL is unable to process the file, various error handling will be set up to log that the file import failed, but at this point, the ETL can no longer continue
- Set up the ETL to just take a text file, NOT looking at the format within the file, bring all of the data into ONE column, and then parse through the data, using the given file delimter and log any issues at that point
I have done both methods and I think there are advantages and disadvantages to both.  Hopefully I explained that well enough. Can anyone give me their thoughts, suggestions, experience, etc. on these 2 approaches of importing a file?  Any input
is greatly appreciated.
Thanks!
Jenna G

It depends on how much control you've on the source end. If you can ,best thing always would be to prefix the metadata and create your package based on that. Any violation should be flagged as an error and reported back for correction with row/column details
etc.
If you've no way to fix the source, then only go for single column approach. But this can still prove to be a nightmare if the source format keeps on changing. Giving that flexibility to users can prove to be costly sometimes and you may have to spend quite
a bit of time trying to fix inconsistencies in the source.
So in my opinion best would be the former approach. Only thing is the format (matadata) has to be fixed only after sufficient discussions with all involved and exception handling also has to be agreed upon.
Please Mark This As Answer if it helps to solve the issue Visakh ---------------------------- http://visakhm.blogspot.com/ https://www.facebook.com/VmBlogs

Similar Messages

  • Import external text file into table in database using web form

    I whant to import data from text delimited file located in os
    into oracle table using web form.
    I am using ORACLE DATABASE 8i, APPLICATION SERVER 9i
    and ORACLE FORMS DEVELOPER AND FORMS SERVER 6i (Patch 2)
    Is there anybody who know how can i do this?
    Thank you!

    WebUtil uploads the files as Binary - so yes you could have some issues if you have a Unix host - however, that would only mean that there is an extra character to trim off of the end of the line read by Text_io.

  • I'm creating an epk and I need to add a bio to the menu.  Can I import the text file or do I need to retype the bio somewhere?  What is the best way to proceed?

    I'm creating an epk and I need to add a bio to the menu.  Can I import the text file or do I need to retype the bio somewhere?  What is the best way to proceed?

    Electronic Press Kit.
    Usually in the form of video outtakes, movie trailers, cast and crew interviews. Basically, the stuff that gets bunged onto commercial DVDs as extras or ends up on late night TV "Making Of..." shows.
    To answer the OP, you want a button on the menu that leads to a still image, or series of them, set up as a slide show. You can copy/paste the text in DVD Studio and then add formatting via the Text menu. (Highlight the text and right click to access formatting controls). Or do it in a graphics application using the appropriate video image preset.

  • Importing a text file with variable values into web reporting

    Hello,
    I'm looking for a possibility to import values from a text file into a variable selection in Web reporting.
    With BEx reporting in Excel this is possible, by clicking on the multiple selection button in the variable popup screen, and subsequently selecting the "Import from Text file" button.
    However, this function seems not to be available for web-reporting...
    It would be great if someone could help me out with this!
    Thanks & regards,
    Arvid

    Hi,
    we could resolve this issue, so i thought it may also be helpful for others:
    In our example we used a file with numbers for materials. This file is stored somewhere in a directory, to which SAP BI must have authorization to read.
    The file looks something like this:
    4711
    4712
    4713
    4714
    The file is named "import.txt" and lays in directory "/usr/sap/EC6/files/bi/"
    *&  Include           ZXRSRU01
    * global variables
    Data: intern_range LIKE LINE OF i_t_var_range,
          l_s_range    TYPE rsr_s_rangesid,
          line(100)    TYPE c,
          p_file(128)  TYPE c,
          length_rangelow  type i,
          tmp_rangelow     like l_s_range-low.
    * internal tables for selection-transfer from transaction
    * Data: BEGIN OF it_file occurs 0,
    *        it_p_file(128) TYPE c,
    *      END of it_file.
    IF i_step = 1.
    ** variables can be changed or set before pop-up appears
      CASE i_vnam.
    * take material from external file to selection-list
         WHEN 'ZSD_UPMA'.
    ** call of transaction, with which the path can be set
    *CALL TRANSACTION 'ZBW_VARIABLE' using it_file
    *MODE 'A'      " call should be visible, so that variable can be set
    *UPDATE 'S'.   " first transaction, then processing
    ** Der Pfad, der in dem Selektionsbild eingegeben wird, wird an die Variable übergeben
    ** Der Set-Parameter ist in Report ZSD_SELECT_VARIABLE
    *  get parameter id 'VAR' field p_file.
    p_file = '/usr/sap/EC6/files/bi/import.txt'.
    * further handling of variable in BI
          OPEN DATASET p_file FOR INPUT IN TEXT MODE ENCODING DEFAULT.
          IF sy-subrc = 0.
            READ DATASET p_file INTO line.
            WHILE sy-subrc = 0.
              IF line(2) <> '//'.
                l_s_range-sign = 'I'.
                l_s_range-opt  = 'EQ'.
                l_s_range-low  = line.
    * fill with leading Zeros
    * in variable tmp_rangelow the value from l_s_range-low is taken
        tmp_rangelow    = l_s_range-low.
    * read the length
        length_rangelow = strlen( tmp_rangelow ).
    * in our case: material has 18 characters
        while length_rangelow lt 18.
          CONCATENATE '0' tmp_rangelow INTO tmp_rangelow.
          length_rangelow = length_rangelow + 1.
        endwhile.
    * initialize l_s_range-low
        clear l_s_range-low.
    * set with filled values
        l_s_range-low = tmp_rangelow.
    * transfer to structure
                APPEND l_s_range TO e_t_range.
              ENDIF.
              READ DATASET p_file INTO line.
            ENDWHILE.
          ENDIF.
          CLOSE DATASET p_file.
        ENDCASE.
    ELSEIF i_step = 2.
    ** in step 2 all variable values from pop-up input can be processed or
    ** User Exit variables can be derived
    * UserExit Ende.
    ENDIF.
    Hope i could help!
    Best regards,
    Tobias

  • Scroll bar with external text file

    Hello all,
    I need to make a flash movie with a graphic background with
    text from an external file.
    Ideally:
    1) The text file would NOT have to be imported into the flash
    movie. It would simply be in the same web folder as the flash file.
    2) Even better the text file would have a separate css file
    so I can change font size etc, while I tweak the movie, and get
    client approval.
    If 1 and 2 are not possible, then I can live with importing
    the text file. But CSS would sure be a nice addition!
    BTW I have Flaxh MX Professional 2004.
    Thanks
    Rowby

    Hey, I had to do a VERY similar thing for a student project a
    few months ago. If you go to
    Flashkit.com, you can find
    tutorials on how to import XML and CSS or XSL. I was able to do it
    and I don't know a lot of actionscript. If you want to see how I
    used it, go to
    the University of Scranton's MIT
    website.
    Hope that helps!

  • Accented Characters Support for External Text Files

    Hi there! I created a site which imports external text from a
    TXT file. The text is in Spanish, so I need accented vowels and
    Ñs. I've tried using simple text and HTML text... but when I
    run my movie, it doesn't show these international characters.
    Instead, I get some weird symbols or question marks... What should
    I do? Thanks in advance!

    Hi,
    Try this link it may have what your looking for it worked for
    me!
    http://www.adobe.com/cfusion/knowledgebase/index.cfm?id=tn_14143
    Danielle

  • Problem in bulk import of text files in SQL Server

    Hi,
    I am using sql server 2012 x 64 and trying to import some text files (tab/ comma separated) and ms access files in sql server tempdb
    I am trying bulkinsert on text file but it is inserting all data in a single line in sql while in text file I have 1000K lines,
    Then I tried importing other data using openrowset from access file but I am getting following error
    Cannot create an instance of OLE DB provider "MSDASC" for linked server "(null)".
    can any one please provide some assistance . If I am able to upload the data from flat text file then it will be enough as I am getting data in tsv or csv format
    Nitin

    If this is a one-off, I think you should try the import/export wizard in SQL Server Management Studio instead. This may be easier to get working.
    If you want to this import on a regular basis with BULK INSERT, we need to see the CREATE TABLE statement for your table and a sample file. Best if you can upload the file on Skydrive, Dropbox or similar, as the exact bytes matters.
    To connect to an Access database, you use the ACE provider. Never heard of MSDASC before.
    Erland Sommarskog, SQL Server MVP, [email protected]

  • Importing External HMTL files

    Is it possible to import external HMTL files to iWeb then Upload to .Mac?
    We have uploaded these external HMTL files directly to .Mac but find we do not have editing capabilities.

    Is it possible to import external HMTL files to iWeb
    then Upload to .Mac?
    We have uploaded these external HMTL files directly
    to .Mac but find we do not have editing capabilities.
    Put them at
    iDisk/Sites
    Address will be
    http://homepage.mac.com/yourusername/pagename.html
    You can edit html files in any text or html editor. In TextEdit, to view the code in an HTML file, choose File > Open and select "Ignore rich text commands." Then locate the .html file and click Open.
    If you want to use your existing html files with an iWeb site, check this out:
    http://www.geocities.com/[email protected]/widgets.html

  • Find Change through external text file

    Hello folks
    I am bit pretty in InDesign scripting so could you please look into this.
    How can i change any particular text field in Indesign CS3 document from text file.
    I do have find change script but for each InDesign document specific text file is assigned.
    So each time i have to modify find change GREP property that is also repetetive of work. Is there any way to get find change information should be extract from external text file.
    Many Tanks in advance

    In the FindChangeByList script, you could customize the function myFindFile(myFilePath) {...} as to search the FindChangeList text file in the document location rather than the script location. That's an example. The question is: given a document, where will you have the corresponding FindChangeList?
    @+
    Marc

  • Add one character at  a time from External Text file

    Hi,
    I'm using Flash 8, and I have created a standard Dynamic
    field that pulls the text from an external text file using the
    LoadVars function and it works fine.
    To get the correct effect for the design, I would like to be
    able to create the illusion that the characters are being typed out
    when the page loads as if someone has started typing and also hear
    the "Typewriter Clack" each time a character appears.
    The only place I have found an example is on the Syphon
    Filter Game website which seems to have created the illusion
    perfectly:
    http://www.syphonfilterdarkmirror-thegame.com/en_IE/
    (Check out the bottom right section)
    I could create a Movieclip and add a character each frame
    but, I would like to be able to edit the text when required via a
    text file or even ASP or PHP if neccessary.
    Any help would be grateful.
    Cheers

    or here:
    http://www.actionscripts.org/tutorials/beginner/Scripted_Typerwriter/index.shtml

  • Error while importing a text file for a list of values

    Hi Experts,
    While importing a text file for a list of values in designer, I am getting error message as below:
    "Invalid setting in Excel key of the Engines section of the Windows Registry. (3433)".
    I am using BO XI R3. Universe Designer version : 12.1.0.882
    Any suggestion ?
    Regards

    plzzzzzzzzzzzzzzzzz.............sme1 help me out

  • What is the best practice in securing deployed source files

    hi guys,
    Just yesterday, I developed a simple image cropper using ajax
    and flash. After compiling the package, I notice the
    package/installer delivers the same exact source files as in
    developed to the installed folder.
    This doesnt concern me much at first, but coming to think of
    it. This question keeps coming out of my head.
    "What is the best practice in securing deployed source
    files?"
    How do we secure application installed source files from
    being tampered. Especially, when it comes to tampering of the
    source files after it's been installed. E.g. modifying spraydata.js
    files for example can be done easily with an editor.

    Hi,
    You could compute a SHA or MD5 hash of your source files on
    first run and save these hashes to EncryptedLocalStore.
    On startup, recompute and verify. (This, of course, fails to
    address when the main app's swf / swc / html itself is
    decompiled)

  • "Import from Text File" & "Upload from Clipboard" icon not appearng in F110

    Hi Experts,
    While making the payment to the vendors through TN F110, in "PARAMETER" screen just next to 'STATUS' screen, there is a vendor field for selection in range and just ahead of this there is an icon for multiple vendor selection ( Bold arrow mark ). If I click on that multiple vender selection icon ( on bold arrow mark ), an another screen opens for multiple selection for vendors. In this screen 'Enter, Copy, Delete etc' icons are appearing at the bottom of the screen but "Import from Text File" & "Upload from Clipboard" icon are not appearing there.
    Can you please let me know how to bring that icons there?
    Points would be awarded.
    Rgds,
    Shailesh

    Hi Ravi,
    I have not yet created anything and have just entered the parameter. In the multiple screen for vendors there are 5 fields for individual and 5 for range selection. Even you can put more than five range selections or individual selections by clicking on the page button but there I am not finding these two icons.
    Can you please check at your end and let me know if you are getting or what to do to bring those two icons in F110.
    Rgds,
    Shailesh

  • When I import a text file(comma separated )into a numbers spread sheet all the data goes into one column. Why does the text not go into separate columns based on the commas.

    When I import a text file(comma separated) into a numbers spreadsheet all the data goes into one column instead of individual columns based on the comma separators.  Excel allows you to do this during the import..  Is there a way to accomplish this in numbers without opening it in Excel and the importing into Numbers.

    Your user info says iPad. This is the OS X Numbers forum. Assuming you are using OS X… Be sure the file is named with a .csv suffix.
    (I don't have an iPad, so I don't know the iOS answer.)

  • How to load external text file into a Form?

    Hi,
    I made a menu with 1-5 in a Form and 5 external text files. I want the user who is able to view the text content of the text file when he chooses one of them from the menu. And the text file screen has a "Back" command button to let him go back.
    How can I do it and can it support other languages such as Chinese and Japanese?
    Thanks for help.
    gogo

    Sorry, I made the mistake about the subject, it should be loading local file, not external file through http.
    I wrote a method but it throwed an exception when the midlet was run.
    private void loadText()
    try {
    InputStream is = this.getClass().getResourceAsStream("/text.txt");
    StringBuffer sb = new StringBuffer();
    int chr;
    while ((chr = is.read()) != -1)
    sb.append((char) chr);
    is.close();
    catch (Exception e)
    System.out.println("Error occurs while reading file");
    I put the text.txt file in the same folder with the main file (extends midlet). How can I load the text content and display it on StringItem?
    Thanks for any help.
    gogo

Maybe you are looking for