SSIS best practice on importing External text files
Hi -
I am a fairly seasoned SSIS/ETL developer and I am struggling with the best architecure on how to import vendor files into a shared database. I see there being 2 methods with in importing files, and I'm really wanting input from senior
level SSIS developers on their thoughts of importing vendor files, I see there being 2 methods:
- Set up the ETL to match the format of the file, and if the format is invalid, the entire ETL is unable to process the file, various error handling will be set up to log that the file import failed, but at this point, the ETL can no longer continue
- Set up the ETL to just take a text file, NOT looking at the format within the file, bring all of the data into ONE column, and then parse through the data, using the given file delimter and log any issues at that point
I have done both methods and I think there are advantages and disadvantages to both. Hopefully I explained that well enough. Can anyone give me their thoughts, suggestions, experience, etc. on these 2 approaches of importing a file? Any input
is greatly appreciated.
Thanks!
Jenna G
It depends on how much control you've on the source end. If you can ,best thing always would be to prefix the metadata and create your package based on that. Any violation should be flagged as an error and reported back for correction with row/column details
etc.
If you've no way to fix the source, then only go for single column approach. But this can still prove to be a nightmare if the source format keeps on changing. Giving that flexibility to users can prove to be costly sometimes and you may have to spend quite
a bit of time trying to fix inconsistencies in the source.
So in my opinion best would be the former approach. Only thing is the format (matadata) has to be fixed only after sufficient discussions with all involved and exception handling also has to be agreed upon.
Please Mark This As Answer if it helps to solve the issue Visakh ---------------------------- http://visakhm.blogspot.com/ https://www.facebook.com/VmBlogs
Similar Messages
-
Import external text file into table in database using web form
I whant to import data from text delimited file located in os
into oracle table using web form.
I am using ORACLE DATABASE 8i, APPLICATION SERVER 9i
and ORACLE FORMS DEVELOPER AND FORMS SERVER 6i (Patch 2)
Is there anybody who know how can i do this?
Thank you!WebUtil uploads the files as Binary - so yes you could have some issues if you have a Unix host - however, that would only mean that there is an extra character to trim off of the end of the line read by Text_io.
-
I'm creating an epk and I need to add a bio to the menu. Can I import the text file or do I need to retype the bio somewhere? What is the best way to proceed?
Electronic Press Kit.
Usually in the form of video outtakes, movie trailers, cast and crew interviews. Basically, the stuff that gets bunged onto commercial DVDs as extras or ends up on late night TV "Making Of..." shows.
To answer the OP, you want a button on the menu that leads to a still image, or series of them, set up as a slide show. You can copy/paste the text in DVD Studio and then add formatting via the Text menu. (Highlight the text and right click to access formatting controls). Or do it in a graphics application using the appropriate video image preset. -
Importing a text file with variable values into web reporting
Hello,
I'm looking for a possibility to import values from a text file into a variable selection in Web reporting.
With BEx reporting in Excel this is possible, by clicking on the multiple selection button in the variable popup screen, and subsequently selecting the "Import from Text file" button.
However, this function seems not to be available for web-reporting...
It would be great if someone could help me out with this!
Thanks & regards,
ArvidHi,
we could resolve this issue, so i thought it may also be helpful for others:
In our example we used a file with numbers for materials. This file is stored somewhere in a directory, to which SAP BI must have authorization to read.
The file looks something like this:
4711
4712
4713
4714
The file is named "import.txt" and lays in directory "/usr/sap/EC6/files/bi/"
*& Include ZXRSRU01
* global variables
Data: intern_range LIKE LINE OF i_t_var_range,
l_s_range TYPE rsr_s_rangesid,
line(100) TYPE c,
p_file(128) TYPE c,
length_rangelow type i,
tmp_rangelow like l_s_range-low.
* internal tables for selection-transfer from transaction
* Data: BEGIN OF it_file occurs 0,
* it_p_file(128) TYPE c,
* END of it_file.
IF i_step = 1.
** variables can be changed or set before pop-up appears
CASE i_vnam.
* take material from external file to selection-list
WHEN 'ZSD_UPMA'.
** call of transaction, with which the path can be set
*CALL TRANSACTION 'ZBW_VARIABLE' using it_file
*MODE 'A' " call should be visible, so that variable can be set
*UPDATE 'S'. " first transaction, then processing
** Der Pfad, der in dem Selektionsbild eingegeben wird, wird an die Variable übergeben
** Der Set-Parameter ist in Report ZSD_SELECT_VARIABLE
* get parameter id 'VAR' field p_file.
p_file = '/usr/sap/EC6/files/bi/import.txt'.
* further handling of variable in BI
OPEN DATASET p_file FOR INPUT IN TEXT MODE ENCODING DEFAULT.
IF sy-subrc = 0.
READ DATASET p_file INTO line.
WHILE sy-subrc = 0.
IF line(2) <> '//'.
l_s_range-sign = 'I'.
l_s_range-opt = 'EQ'.
l_s_range-low = line.
* fill with leading Zeros
* in variable tmp_rangelow the value from l_s_range-low is taken
tmp_rangelow = l_s_range-low.
* read the length
length_rangelow = strlen( tmp_rangelow ).
* in our case: material has 18 characters
while length_rangelow lt 18.
CONCATENATE '0' tmp_rangelow INTO tmp_rangelow.
length_rangelow = length_rangelow + 1.
endwhile.
* initialize l_s_range-low
clear l_s_range-low.
* set with filled values
l_s_range-low = tmp_rangelow.
* transfer to structure
APPEND l_s_range TO e_t_range.
ENDIF.
READ DATASET p_file INTO line.
ENDWHILE.
ENDIF.
CLOSE DATASET p_file.
ENDCASE.
ELSEIF i_step = 2.
** in step 2 all variable values from pop-up input can be processed or
** User Exit variables can be derived
* UserExit Ende.
ENDIF.
Hope i could help!
Best regards,
Tobias -
Scroll bar with external text file
Hello all,
I need to make a flash movie with a graphic background with
text from an external file.
Ideally:
1) The text file would NOT have to be imported into the flash
movie. It would simply be in the same web folder as the flash file.
2) Even better the text file would have a separate css file
so I can change font size etc, while I tweak the movie, and get
client approval.
If 1 and 2 are not possible, then I can live with importing
the text file. But CSS would sure be a nice addition!
BTW I have Flaxh MX Professional 2004.
Thanks
RowbyHey, I had to do a VERY similar thing for a student project a
few months ago. If you go to
Flashkit.com, you can find
tutorials on how to import XML and CSS or XSL. I was able to do it
and I don't know a lot of actionscript. If you want to see how I
used it, go to
the University of Scranton's MIT
website.
Hope that helps! -
Accented Characters Support for External Text Files
Hi there! I created a site which imports external text from a
TXT file. The text is in Spanish, so I need accented vowels and
Ñs. I've tried using simple text and HTML text... but when I
run my movie, it doesn't show these international characters.
Instead, I get some weird symbols or question marks... What should
I do? Thanks in advance!Hi,
Try this link it may have what your looking for it worked for
me!
http://www.adobe.com/cfusion/knowledgebase/index.cfm?id=tn_14143
Danielle -
Problem in bulk import of text files in SQL Server
Hi,
I am using sql server 2012 x 64 and trying to import some text files (tab/ comma separated) and ms access files in sql server tempdb
I am trying bulkinsert on text file but it is inserting all data in a single line in sql while in text file I have 1000K lines,
Then I tried importing other data using openrowset from access file but I am getting following error
Cannot create an instance of OLE DB provider "MSDASC" for linked server "(null)".
can any one please provide some assistance . If I am able to upload the data from flat text file then it will be enough as I am getting data in tsv or csv format
NitinIf this is a one-off, I think you should try the import/export wizard in SQL Server Management Studio instead. This may be easier to get working.
If you want to this import on a regular basis with BULK INSERT, we need to see the CREATE TABLE statement for your table and a sample file. Best if you can upload the file on Skydrive, Dropbox or similar, as the exact bytes matters.
To connect to an Access database, you use the ACE provider. Never heard of MSDASC before.
Erland Sommarskog, SQL Server MVP, [email protected] -
Is it possible to import external HMTL files to iWeb then Upload to .Mac?
We have uploaded these external HMTL files directly to .Mac but find we do not have editing capabilities.Is it possible to import external HMTL files to iWeb
then Upload to .Mac?
We have uploaded these external HMTL files directly
to .Mac but find we do not have editing capabilities.
Put them at
iDisk/Sites
Address will be
http://homepage.mac.com/yourusername/pagename.html
You can edit html files in any text or html editor. In TextEdit, to view the code in an HTML file, choose File > Open and select "Ignore rich text commands." Then locate the .html file and click Open.
If you want to use your existing html files with an iWeb site, check this out:
http://www.geocities.com/[email protected]/widgets.html -
Find Change through external text file
Hello folks
I am bit pretty in InDesign scripting so could you please look into this.
How can i change any particular text field in Indesign CS3 document from text file.
I do have find change script but for each InDesign document specific text file is assigned.
So each time i have to modify find change GREP property that is also repetetive of work. Is there any way to get find change information should be extract from external text file.
Many Tanks in advanceIn the FindChangeByList script, you could customize the function myFindFile(myFilePath) {...} as to search the FindChangeList text file in the document location rather than the script location. That's an example. The question is: given a document, where will you have the corresponding FindChangeList?
@+
Marc -
Add one character at a time from External Text file
Hi,
I'm using Flash 8, and I have created a standard Dynamic
field that pulls the text from an external text file using the
LoadVars function and it works fine.
To get the correct effect for the design, I would like to be
able to create the illusion that the characters are being typed out
when the page loads as if someone has started typing and also hear
the "Typewriter Clack" each time a character appears.
The only place I have found an example is on the Syphon
Filter Game website which seems to have created the illusion
perfectly:
http://www.syphonfilterdarkmirror-thegame.com/en_IE/
(Check out the bottom right section)
I could create a Movieclip and add a character each frame
but, I would like to be able to edit the text when required via a
text file or even ASP or PHP if neccessary.
Any help would be grateful.
Cheersor here:
http://www.actionscripts.org/tutorials/beginner/Scripted_Typerwriter/index.shtml -
Error while importing a text file for a list of values
Hi Experts,
While importing a text file for a list of values in designer, I am getting error message as below:
"Invalid setting in Excel key of the Engines section of the Windows Registry. (3433)".
I am using BO XI R3. Universe Designer version : 12.1.0.882
Any suggestion ?
Regardsplzzzzzzzzzzzzzzzzz.............sme1 help me out
-
What is the best practice in securing deployed source files
hi guys,
Just yesterday, I developed a simple image cropper using ajax
and flash. After compiling the package, I notice the
package/installer delivers the same exact source files as in
developed to the installed folder.
This doesnt concern me much at first, but coming to think of
it. This question keeps coming out of my head.
"What is the best practice in securing deployed source
files?"
How do we secure application installed source files from
being tampered. Especially, when it comes to tampering of the
source files after it's been installed. E.g. modifying spraydata.js
files for example can be done easily with an editor.Hi,
You could compute a SHA or MD5 hash of your source files on
first run and save these hashes to EncryptedLocalStore.
On startup, recompute and verify. (This, of course, fails to
address when the main app's swf / swc / html itself is
decompiled) -
"Import from Text File" & "Upload from Clipboard" icon not appearng in F110
Hi Experts,
While making the payment to the vendors through TN F110, in "PARAMETER" screen just next to 'STATUS' screen, there is a vendor field for selection in range and just ahead of this there is an icon for multiple vendor selection ( Bold arrow mark ). If I click on that multiple vender selection icon ( on bold arrow mark ), an another screen opens for multiple selection for vendors. In this screen 'Enter, Copy, Delete etc' icons are appearing at the bottom of the screen but "Import from Text File" & "Upload from Clipboard" icon are not appearing there.
Can you please let me know how to bring that icons there?
Points would be awarded.
Rgds,
ShaileshHi Ravi,
I have not yet created anything and have just entered the parameter. In the multiple screen for vendors there are 5 fields for individual and 5 for range selection. Even you can put more than five range selections or individual selections by clicking on the page button but there I am not finding these two icons.
Can you please check at your end and let me know if you are getting or what to do to bring those two icons in F110.
Rgds,
Shailesh -
When I import a text file(comma separated) into a numbers spreadsheet all the data goes into one column instead of individual columns based on the comma separators. Excel allows you to do this during the import.. Is there a way to accomplish this in numbers without opening it in Excel and the importing into Numbers.
Your user info says iPad. This is the OS X Numbers forum. Assuming you are using OS X… Be sure the file is named with a .csv suffix.
(I don't have an iPad, so I don't know the iOS answer.) -
How to load external text file into a Form?
Hi,
I made a menu with 1-5 in a Form and 5 external text files. I want the user who is able to view the text content of the text file when he chooses one of them from the menu. And the text file screen has a "Back" command button to let him go back.
How can I do it and can it support other languages such as Chinese and Japanese?
Thanks for help.
gogoSorry, I made the mistake about the subject, it should be loading local file, not external file through http.
I wrote a method but it throwed an exception when the midlet was run.
private void loadText()
try {
InputStream is = this.getClass().getResourceAsStream("/text.txt");
StringBuffer sb = new StringBuffer();
int chr;
while ((chr = is.read()) != -1)
sb.append((char) chr);
is.close();
catch (Exception e)
System.out.println("Error occurs while reading file");
I put the text.txt file in the same folder with the main file (extends midlet). How can I load the text content and display it on StringItem?
Thanks for any help.
gogo
Maybe you are looking for
-
Can I get my laptop to connect via HDMI to mini display with iMac midi2011
I want to use as display my iMac( 27" midi 2011), connecting it with laptop HDMI port
-
Two iTunes accounts each on their own iPads. How do we share the Mac with both?
My husband has an iPhone and iPad with his iTunes set up on them. I have an iPhone and iPad with my own iTunes set up. We have just bought a MacBook Air, is there any way of linking the Mac to both of our devices and separate iTunes accounts?
-
Acrobat XI Pro for Mac saved file not seen in a AFP NAS
After I save a document through Acrobat XI Pro to NAS in Mac OS 10.8, I can't see the document in that directory by Finder through AFP-protocol , however I can see it through SAMBA-protocol. This never happened before, only after I saved the file thr
-
Hi, We recently upgraded a site from SP 2010 to SP 2013. For moving to production, I copied over the content database to production and used the following command to to make the content database accessible in production: Mount-SPContentDatabase -Name
-
Interactive Report with Null Columns
I've got a user who wants to export an interactive report, and he needs certain columns to appear on the export for him to fill the data in later (this is an intermediate step, while we work on getting all the data he needs in to the database). I've