What is the best way to load and convert data from a flat file?

Hi,
I want to load data from a flat file, convert dates, numbers and some fields with custom logic (e.g. 0,1 into N,Y) to the correct format.
The rows where all to_number, to_date and custom conversions succeed should go into table STG_OK. If some conversion fails (due to an illegal format in the flat file), those rows (where the conversion raises some exception) should go into table STG_ERR.
What is the best and easiest way to archive this?
Thanks,
Carsten.

Hi,
thanks for your answers so far!
I gave them a thought and came up with two different alternatives:
Alternative 1
I load the data from the flat file into a staging table using sqlldr. I convert the data to the target format using sqlldr expressions.
The columns of the staging table have the target format (date, number).
The rows that cannot be loaded go into a bad file. I manually load the data from the bad file (without any conversion) into the error table.
Alternative 2
The columns of the staging table are all of type varchar2 regardless of the target format.
I define data rules for all columns that require a later conversion.
I load the data from the flat file into the staging table using external table or sqlldr without any data conversion.
The rows that cannot be loaded go automatically into the error table.
When I read the data from the staging table, I can safely convert it since it is already checked by the rules.
What I dislike in alternative 1 is that I manually have to create a second file and a second mapping (ok, I can automate this using OMB*Plus).
Further, I would prefer using expressions in the mapping for converting the data.
What I dislike in alternative 2 is that I have to create a data rule and a conversion expression and then keep the data rule and the conversion expression in sync (in case of changes of the file format).
I also would prefer to have the data in the staging table in the target format. Well, I might load it into a second staging table with columns having the target format. But that's another mapping and a lot of i/o.
As far as I know I need the data quality option for using data rules, is that true?
Is there another alternative without any of these drawbacks?
Otherwise I think I will go for alternative 1.
Thanks,
Carsten.

Similar Messages

  • What is the Best way To Copy and paste data from 1 book to another

     I have 18 sheets in 5 different books that I want to extract data from specific cells.  What is the best way to do this?  Example:  1 sheet is called Numbers E-O1 data in 13:WXYZ. The data updates and moves up 1 row every time I enter
    a new number. So let's say I enter the number 12. Through a lot of calculations the output goes in 13:WXYZ. To what I call a counter which is a 4 digit number.  Anyways, how can I send that 4 digit number to a totally different sheet?  To bullet
    what I'm talking about
    data in cells Row 13:WXYZ in book called Numbers sheet E-O1
    send data to book called "Vortex Numbers" Sheet E-O row 2001:CDEF
    What formula or Macro can I use to make this work?
    thank you!

    Hello Larbec,
    Syntax:
    '[BookName]SheetName'!Range
    Enter in cell  2001:CDEF:
    ='[Numbers]E-O1'!13:WXYZ
    This assumes that the file is open in Excel. Otherwise you need to add the path:
    'ThePath[BookName]SheetName'!Range
    Best regards George

  • What is the best way to import and export footage from the 5D Mark II?

    Hello,
    I've just finished shooting what I am considering to be my directorial masterpiece.  Shot it on the Canon 5D (1080p, 24fps), and the footage looks amazing.  Now I am ready to start editing and have been using premiere lately, but I have yet to figure out the proper pipeline.  I want to know the best way to retain resolution before I delve into this project.
    My questions:
    1)  What is the best way to start a new project and import the footage without having to render whilst editing, so as to retain all resolution and originality of the source footage?
    2)  What is the best way/ codec/ format to export this same footage once editing is complete so as to retain that crisp 1080p for which the 5D is so recognized?
    3)  What is the best way/ codec/ format to import and export/ render between premiere and after effects?  I am speaking mostly of vfx and color correction.  I also have some 30fps footage that I intend to slow down in AE and then import into premiere.
    I know this is pretty broad, but as a solo filmmaker I really need someone's guidance.  I rarely ever finish my films with the same, crisp look as the footage.  I need pipeline help, and really appreciate it!

    ascreenwriter wrote:
    Hello,
    I've just finished shooting what I am considering to be my directorial masterpiece.  Shot it on the Canon 5D (1080p, 24fps), and the footage looks amazing.  Now I am ready to start editing and have been using premiere lately, but I have yet to figure out the proper pipeline.  I want to know the best way to retain resolution before I delve into this project.
    My questions:
    1)  What is the best way to start a new project and import the footage without having to render whilst editing, so as to retain all resolution and originality of the source footage?
    2)  What is the best way/ codec/ format to export this same footage once editing is complete so as to retain that crisp 1080p for which the 5D is so recognized?
    3)  What is the best way/ codec/ format to import and export/ render between premiere and after effects?  I am speaking mostly of vfx and color correction.  I also have some 30fps footage that I intend to slow down in AE and then import into premiere.
    I know this is pretty broad, but as a solo filmmaker I really need someone's guidance.  I rarely ever finish my films with the same, crisp look as the footage.  I need pipeline help, and really appreciate it!
    1. Follow the advice above. Also use the Media Browser to import the footage in case you have spanned media files. Import files with the Media Browser.
    2. It largely depends on what you wish to ouput to: Blu-ray, web, etc. This FAQ gives the best answer: What are the best export settings?
    3. Use the Replace with Adobe After Effects Composition function.

  • What's the best way to load FieldPoint measurement data into PI System?

    I am finding the best way to load data collected by NI Field Point (FP2220) into the PI system of our power plant.
    I found pieces of information about FieldPoint OPC server in NI.com. Not sure if it comes with Field Point Hardware, sold by NI as a separate product or it is actually non-standard NI products. Anyway, I know that there exists a thing called FieldPoint OPC server.
    The PI system I mentioned has a OPC client software called PI-OPC interface. It is able to communicate with standard OPC DA server. If that FieldPoint OPC server is a standard OPC DA server provide data collected by Field Point complying to OPC standard, than that's perfect.
    Anyone familar with PI system and NI product, please help if the above is going to work or if there is a better way to put Fieldpoint data into PI.

    Hi Eric,
    This information really helpful, thanks. Regarding to the NI OPC server for NI FieldPoint, I have the other query.
    In my setup, there are two sets of FieldPoint located in two different locations on my ethernet network. They are going to be controlled by a single PC. If I am going to connect both my FieldPoint sets with OPC standard, how many NI OPC server for FieldPoint do I need to connect to? Are there two NI OPC servers each serves one FieldPoint set? Or there is only NI OPC server which serves both FieldPoint sets?
    I am concerning about the number of NI OPC server instances running, because the number of OPC client license I need to purchase depends on how many OPC server I need to connect to. If one NI OPC server serves both my FieldPoint sets, I only need to buy one OPC client license; otherwise, I need to purchase two. In the future, I am going to have another two sets of FieldPoint sets, so the answer of my query determines how many OPC clients I need to purchase eventually - One or four. A huge price difference.
    Looking forward to your reply.
    Regards,
    Roger

  • What is the best way to geotag photos with data from GPS Tracker Log (GPX format)?

    I just returned from a 4 month trip to Southeast Asia. Before I left, I set the time on my camera, a Canon 7D, to match the GPS time on my GPS tracker, a QStarz BT-Q1000X. Both times were set to UTC, also known as GMT.
    I have imported the data from the GPS log into Aperture as a GPX file. Now I would like to sync the location metadata on my photos with the data from the GPS log. How can I do this using Aperture 3?
    I seem to be having two problems:
    Aperture wants to shift the times based on which part of the track I drag the images onto. Since the camera and gps tracker times were sync'd, I do not want any shift.
    The time shifts that are displayed are 12 hours and a few minutes, rather than 0 hours and a few minutes.
    A few additional notes:
    When I imported the GPS track, Aperture automatically changed the time zone of the track to Asia/Ho Chi Minh City, which is correct (GMT+7).
    I have batch changed the images from Camera Time: UTC to Actual Time: GMT+7
    The time on my computer is US/Eastern Daylight Time (GMT-4), or Eastern Standard Time (GMT-5) when the photos where taken.
    Since I was gone for so long, and my GPS tracker memory is limited, I set the tracker to record a trackpoint once every 60 seconds.
    Is there a solution to my problems using Aperture 3, or should I start looking for third party applications to geotag the photos before importing them into Aperture?
    Thank you.

    I think the 12 hour discrepancy is more likely a result of the 12 hour difference between GMT+5 and GMT-7. The dates and times of the photos and the GPS track are displayed properly in Aperture. See the images below.

  • What is the best way to kill/stop a data load?

    Hi.
    What is the best way to kill/stop a data load?
    I have a data load from my QA R/3 system that is extracting 115.000.000+ records. The problem is that the selection in the function module used in the data source does not work, and the problem was not detected because of the nature of the data on the development system.
    I could kill processes owned by my background user (on both R/3 and BW) but I risk killing other loads, and sometimes the job seems to restart if I just try to kill processes. If I remove transactional RFCs in SM58 the load does not terminate; I only skip one or more datapackages. I have also tried to change the QM-status in the monitor to red, but that does not stop the load either...
    So isn't there a nice fool-proof way of stopping a dataload?
    Best regards,
    Christian Frier

    Hi,
    There r 2 ways to kill the job.
    One is using transation RSMO locate the job and display the status tab double click on the yellow light that is shown on the line total, a pop will come 'set overall status ' is displayed select the desired status that is red and save it. Then return to the monitor page and select the header tab double ckick on the data target right click and then goto 'manage',there should be request sitting there probably with yellow lights , highlight the line with the faulty request click the delete button then click refresh button.
    Second is goto SM37 and click on the active selection and enter the jobname and then click excute the particulr job should appear highlight the jobname then click on the stop iconthat appears on the taskbar( 3 rd from left)
    hope it is clear.
    Regards-
    Siddhu

  • What's The Best Way to Load a Replacement iPod Touch?

    I just received a replacement iPod Touch 3G for my daughter. Her iTunes library for the Touch that we returned to Apple is on her MacBook Pro. What is the best way to load the new Touch? She plans on using the same name for it. Should I restore from a previous backup or just plug it in and let in sync? Do I need to set anything on iTunes for disk management?
    Thank you,
    Bruce

    If she wants it like her old one the previous backup, if there is anything she wants to change though, now would be a perfect time to just resync it and start over. It really depends on what she wants.

  • What is the best way to store and search 2D data

    Hi,
    There is a set of data (~10k records ) in 2D dimension.
    like this :
    Col 1, Col 2, Col3....
    What is the best way to store and search those records ?
    Thanks in advance
    Wilson

    Hi,
    Either userObjet[][] if you know how much data you have, and the data size is fixed, or use a list of lists. E.g. A Vector of Vectors (some will probably say that you should use an ArrayList instead, and that could be the case, but it sounds like you would want to display the data later on, and a DefaultTableModel (for JTables) uses a Vector as data holder).
    Kaj

  • What is the best way to scan and sort old photos in iPhoto

    What is the best way to scan and sort old photos in iPhoto?  They do not have digital dates.

    Hey Chicago Sue,
    Once you scan them and have them on your desktop. You should use Automator and assign the common IPTC tags to the images, so that when you do import them into iPhoto, they get recorded.
    Here is an example of an action in Automator:

  • HT1364 I just bought a new PC and now have ample space on my C drive to house my music Library which is currenlty installed on a external drive.  What is the best way to install and move the itunes library to my C Drive?

    I just bought a new PC and now have ample space on my C drive to house my music Library which is currenlty installed on a external drive.  What is the best way to install and move the itunes library to my C Drive?

    If the entire library is on the external drive then simply copy the iTunes folder into <User's Music> on the new computer, then install iTunes. If you've already installed iTunes you will want to remove the empty iTunes folder in <User's Music> first.
    If it turns out you only have the media folder on the external drive then take a look at this post...
    tt2

  • What is the best way to secure and harden a Macbook Pro against unwanted surveillance?

    What is the best way to secure and harden a Macbook Pro against unwanted surveillance? Tor, VPN, Little Snitch, etc. This would be for that latest version of Mavericks.

    djbabybokchoy wrote:
    Nothing specific, just speaking in general. Ex-wives, governments, bad guys...anyone really. I'm just looking to make my Mac a bit more private and secure, especially when on public networks.
    Governments and ex's will/may have recourse to the legal process (or in the case of the Gov they can choose to ignore the legal system if they feel like it) when they want to see something of yours, good luck hardening your Mac against that. The best way to avoid the possibility of snooping over public networks is to avoid them but if you can't then Kappy's suggestion will help.
    Strong passwords (everywhere) and don't use the same password in multiple locations.
    If you really want to secure your home wireless use Mac address connection authentication, do not allow unknown Mac addresses to connect. It's much stronger than a WPA password alone.

  • There are way too many photos on my internal hard drive. I have older libraries, and newer libraires in iPhoto and Aperture 2.  What is the best way to find and reduce the number of duplicate photos/libraries  before upgrading to Aperture 3?

    There are way too many photos on my internal hard drive. I have older libraries, and newer libraires in iPhoto and Aperture 2.  What is the best way to find and reduce the number of duplicate photos/libraries  before upgrading to Aperture 3?

    Sharon-
    Good idea.
    Back up first.
    I would probably wait for the merge function of the latest version before merging. Be sure to verify every merge.
    Merge Libraries, then from within Aperture move images to referenced on external hard drives.
    HTH
    -Allen

  • What is the best way to create a database schema from XML

    What is the best way to create a database schema from XML?
    i have  a complex XML file that I want to create a database from and consistently import new XML files of the same schema type. Currently I have started off by mapping the XSD into Excel and using Mysql for Excel to push into MySQL.
    There must be a more .net microsoft solution for this but I cannot locate the topic and tools by searching. What are the best tools and way to manage this?
    Taking my C# further

    Hi Saythj,
    When mentioning "a database schema from XML", do you mean the
    XML Schema Collections? If that is what you mean, when trying to import XML files of the same schema type, you may take the below approach.
    Create an XML Schema Collection basing on your complex XML, you can find
    many generating tools online to do that.
    Create a Table with the above created schema typed XML column as below.
    CREATE TABLE youTable( Col1 int, Col2 xml (yourXMLSchemaCollection))
    Load your XML files and try to insert the xml content into the table above from C# or some other approaches. The XMLs that can't pass the validation fail inserting into that table.
    If you have any question, feel free to let me know.
    Eric Zhang
    TechNet Community Support

  • What is the best way to transfer my iphoto library from old imac to new imac. both have iphoto 9.2.1

    What is the best way to transfer my iphoto library from current imac to my new imac.  Both imacs have iphoto 9.2.1
    os on old imac is 10.6.8.   os on new imac is 10.7.2.   I want the albums  and events to stay intact.  I have 4500 photos and 79 videos. 14.9 gb. looking for some advice here. Thanks so much.

    Connect the two Macs together (network, firewire target mode, etc) and drag the iPhoto library intact as a single entity from the old Mac to the pictures folder of the new Mac - launch iPhoto on the new mac and it will open the library and convert it as needed and you will be ready move forward.
    LN

  • What is the best way of compressing a large 3 hour final cut file

    What is the best way of compressing a large 3 hour final cut file. I shot the play and it is in final cut and I rndered it so now I have a 22gb file that I need to put on a dvd . Any suggestions
    Thanks
    Macbook Pro
    2.3 GHz Intel Core i7
    with Final Cut Pro 7.0.3
    using Lion as operating system

    Presuming your menus aren't complicated(include audio or video) the total size for the MPG-2 and AC3 files should probably be under 8GB for a dual layer disk. The inspector will tell you the estimated size. 2-pass varible bit rate would be recommended.
    Trying to fit 3 hours on a DVD-5 will only bring very noticable quality hits. Compressor will let you change the average bit rate so that you can fit 174 minutes but trade-off isn't worth it in my opinion.
    Be aware that dual-layer -R and +R media may not play well for everyone everywhere.
    I presume you are not making 1000 or more copies? If you were replication could solve this.
    One other alternative would be to break-up the show into two parts and spread it across two DVD-5s.

Maybe you are looking for

  • Can i use iphone 4 in europe

    I am wondering if people have found it more cost effective to use the iphone 4 in europe (am going to denmark) or to try to buy some cheap other kind of used phone and put a sim card i buy over there in it. I am new to using forums, so if this q has

  • OCA material needed plzzz help me

    HI I want to get OCA certification and i needed OCA material. Anybody help me in this regards?

  • HT5312 How do I get to change my security questions?

    How can I change my security questions?

  • Error: "Java.lang.ClassNotFoundException:InterfaceApplet.class "

    I am trying to execute an applet but the foolowing error occurs: "Java.lang.ClassNotFoundException: InterfaceApplet.class Caused by: java.io.IOException: open HTTP connection failed " My applet is another computer, I execute the connection for networ

  • Hardware for an iCal Server?

    Ok this might turn into a bragging contest but mostly I'm looking into an idea of what sort of computing power we need to run our iCal server. Currently it's this: Mac Mini 1.5GHz Intel Core Solo 2GB RAM OS X 10.5.4 Server w 10 clients I have 52 user