Comparison of Data Loading techniques - Sql Loader & External Tables

Below are 2 techniques using which the data can be loaded from Flat files to oracle tables.
1)     SQL Loader:
a.     Place the flat file( .txt or .csv) on the desired Location.
b.     Create a control file
Load Data
Infile "Mytextfile.txt" (-- file containing table data , specify paths correctly, it could be .csv as well)
Append or Truncate (-- based on requirement) into oracle tablename
Separated by "," (or the delimiter we use in input file) optionally enclosed by
(Field1, field2, field3 etc)
c.     Now run sqlldr utility of oracle on sql command prompt as
sqlldr username/password .CTL filename
d.     The data can be verified by selecting the data from the table.
Select * from oracle_table;
2)     External Table:
a.     Place the flat file (.txt or .csv) on the desired location.
abc.csv
1,one,first
2,two,second
3,three,third
4,four,fourth
b.     Create a directory
create or replace directory ext_dir as '/home/rene/ext_dir'; -- path where the source file is kept
c.     After granting appropriate permissions to the user, we can create external table like below.
create table ext_table_csv (
i Number,
n Varchar2(20),
m Varchar2(20)
organization external (
type oracle_loader
default directory ext_dir
access parameters (
records delimited by newline
fields terminated by ','
missing field values are null
location ('file.csv')
reject limit unlimited;
d.     Verify data by selecting it from the external table now
select * from ext_table_csv;
External tables feature is a complement to existing SQL*Loader functionality.
It allows you to –
•     Access data in external sources as if it were in a table in the database.
•     Merge a flat file with an existing table in one statement.
•     Sort a flat file on the way into a table you want compressed nicely
•     Do a parallel direct path load -- without splitting up the input file, writing
Shortcomings:
•     External tables are read-only.
•     No data manipulation language (DML) operations or index creation is allowed on an external table.
Using Sql Loader You can –
•     Load the data from a stored procedure or trigger (insert is not sqlldr)
•     Do multi-table inserts
•     Flow the data through a pipelined plsql function for cleansing/transformation
Comparison for data loading
To make the loading operation faster, the degree of parallelism can be set to any number, e.g 4
So, when you created the external table, the database will divide the file to be read by four processes running in parallel. This parallelism happens automatically, with no additional effort on your part, and is really quite convenient. To parallelize this load using SQL*Loader, you would have had to manually divide your input file into multiple smaller files.
Conclusion:
SQL*Loader may be the better choice in data loading situations that require additional indexing of the staging table. However, we can always copy the data from external tables to Oracle Tables using DB links.

Please let me know your views on this.

Similar Messages

  • Check data before loading through SQL *Loader

    Hi all,
    I have a temp table which is loaded through SQL*Loader.This table is used by a procedure for inserting data into another table.
    I get error of 0RA-01722 frequently during procdures execution.
    I have decided to check for the error data through the control file itself.
    I have few doubts about SQL Loader.
    Will a record containing character data for column declared as INTEGER EXTERNAL in ctrl file get discarded?
    Does declaring column as INTERGER EXTERNAL take care of NULL values?
    Does a whole record gets discarded if one of the column data is misplaced in the record in input file?
    Control File is of following format:
    LOAD DATA
    APPEND INTO TABLE Temp
    FIELDS TERMINATED BY "|" optionally enclosed by "'"
    trailing nullcols
    FILEDATE DATE 'DD/MM/YYYY',
    ACC_NUM INTEGER EXTERNAL,
    REC_TYPE ,
    LOGO , (Data:Numeric Declared:VARCHAR)
    CARD_NUM INTEGER EXTERNAL,
    ACTION_DATE DATE 'DD/MM/YYYY',
    EFFECTIVE_DATE DATE 'DD/MM/YYYY',
    ACTION_AMOUNT , (Data:Numeric Declared:NUMBER)
    ACTION_STORE , (Data:Numeric Declared:VARCHAR)
    ACTION_AUTH_NUM ,
    ACTION_SKU_NUM ,
    ACTION_CASE_NUM )
    What changes do I need to make in this file regarding above questions?

    Is there any online document for this?<br>
    Here it is

  • Need faster data loading (using sql-loader)

    i am trying to load approx. 230 million records (around 60-bytes per record) from a flat file into a single table. i have tried sql-loader (conventional load) and i'm seeing performance degrade as the file is being processed. i am avoiding direct path sql-loading because i need to maintain uniqueness using my primary key index during the load. so the degradation of the load performance doesn't shock me. the source data file contains duplicate records and may contain records that are duplicates of those that are already in the table (i am appending during sql-loader).
    my other option is to unload the entire table to a flat file, concatenate the new file onto it, run it through a unique sort, and then direct-path load it.
    has anyone had a similar experience? any cool solutions available that are quick?
    thanks,
    jeff

    It would be faster to suck into a Oracle table, call it a temporary table and then make a final move into the final table.
    This way you could direct load into an oracle table then you could
    INSERT /*+ APPEND */ INTO Final_Table
        SELECT DISTINCT *
        FROM   Temp_Table
        ORDER BY ID;This would do a 'direct load' type move from your temp teable to the final table, which automatically merging the duplicate records;
    So
    1) Direct Load from SQL*Loader into temp table.
    2) Place index (non-unique) on temp table column ID.
    3) Direct load INSERT into the final table.
    Step 2 may make this process faster or slower, only testing will tell.
    Good Luck,
    Eric Kamradt

  • SQL Loader error: SQL*Loader-926. Please help

    Hi,
    While loading some files to my database table, I am getting the following error. I am using 'Truncate' option while loading the file:
    Error:
    ====
    SQL*Loader-926: OCI error while executing delete/truncate (due to REPLACE/TRUNCATE keyword) for table LOS_STAGE_DS4
    ORA-01426: numeric overflow
    Here's the loader properties(excerpts from load log)
    ================================
    SQL*Loader: Release 11.1.0.6.0 - Production on Fri Nov 26 04:54:18 2010
    Copyright (c) 1982, 2007, Oracle. All rights reserved.
    Control File: d:\Prod\rent_Load\Bin\rent_Load.ctl
    Data File: d:\Prod\rent_Load\Data\rent.704
    Bad File: d:\Prod\rent_Load\Bad\rent.704
    Discard File: none specified
    (Allow all discards)
    Number to load: ALL
    Number to skip: 0
    Errors allowed: 1000000000
    Bind array: 64 rows, maximum of 256000 bytes
    Continuation: none specified
    Path used: Conventional
    Table LS_STAGE, loaded from every logical record.
    Insert option in effect for this table: TRUNCATE
    Column Name Position Len Term Encl Datatype
    Could someone please help and advise what is the root cause of this error?
    Thanks,

    The root cause is in the error ora-1426, which you can look up in the online error documentation at http://tahiti.oracle.com . No one knows every error message by heart. This means it is expected you look up the error prior to posting, and you don't expect any volunteer in this forum to look up the error on your behalf.
    Also this is a typical candidate for being a known problem, and known problems can be found on My Oracle Support.
    Sybrand Bakker
    Senior Oracle DBA

  • Track flat files that failed loading in sql loader

    Hi,
    Can anyone please suggest me any way to track the flat files which failed while loading in sql loader. And pass this failed flat file name to any column of a database table
    Thanks in advance.
    Edited by: 806821 on Nov 2, 2010 10:22 AM

    Hi Morgan thnannks for ur reply.
    Define failed. 1 row not loaded ... no rows not loaded ... what operating system ... what version of the Oracle database ... track in a table, send an email?
    Your inquiry is long on generalities and short on specifics: Fill in all the blanks ... not just the ones I've listed above.
    even if 1 row is not loaded even then it should be considered failed to load and the name of the that particular flat file should be fetched.
    Operating system is unix
    Oracle database we are using is R12
    track in a table , yeah we want to send an email notificaiton whenever the flat files fails to load.
    Thanks once again...!!

  • Why no exclusive lock for conventional path loading for SQL*Loader?

    why no exclusive lock for conventional path loading for SQL*Loader?
    it use insert statement so it should use exclusive lock right?
    thanks

    ok, so only update statement would put a lock but not for insert statement?
    because I have seen a situation where a user update rows in a sesssion (without commit) prevent another user update the rows.
    thanks

  • *Urgent*How to insert data from MS SQL to the table that create at the adobe form?

    Hi,
    I'm using Adobe life cycle designer 8 to do my interactive form. I would like to ask how to insert data from MS SQL to the table that i have created in my adobe interactive form?
    I really need the information ASAP as i need to hand in my project by next week... i really appreciate any one who reply this post.
    Thanks

    Tou need to do a couple of things
    1. On the Essbase server, set up an odbc system connection to your MySQL database
    2. In the load rule , go to the file menu and select open SQL data source and in the data source put in your SQL statement . A couple of hints. Where it says Select, don't put in the word select and where it say from don't put in from. The system adds them for you. The easiest way ti enter a SQL statement is to do it all in the select area So if your SQL would normanlly say select * from mytable just enter the code as * from mytable in the select area
    The click ol/retrieve and enter in your connection info. Itshould bring data back into the load rule. Save the load rule and use it

  • Can we cleanse and transform data at flat file or external table level?

    Hi,
    I have some data that I want to cleanse and transform. I don't want to cleanse it after i populate the external table, I want to get done with it at flat file level or while populating the external table. Can we cleanse and transform data at flat file or external table level through Oracle or OWB 11.2? Is it possible to run a conditional load (i.e. having a where clause or if-else-then) for an external table? Can we call oracle functions for an external table at the time of creation?
    Thanks in advance.
    Regards,
    Ann.

    Hi Oleg,
    Thanks a lot for the clarification. :)
    So is there a way that I can clease the data within the text file through Oracle or OWB? I have datatype mismatches in the data and most of my data is getting rejected because of that. The way I can think of, for solving this problem, is to create the external table with all fields with datatype varchar and then cleansing the data. But it doesn't seem very effecient plus it will get very complicated because I have almost 80-90 fields.
    Any help?
    Thanks and regards,
    Ann.

  • How to write data to text file using external tables

    can anybody tell how to write data to text file using external tables concept?

    Hi,
    Using external table u can load the data in your local table in database,
    then using your local db table and UTL_FILE pacakge u can wrrite data to text file
    external table
    ~~~~~~~~~~~
    http://download-east.oracle.com/docs/cd/B19306_01/server.102/b14200/statements_7002.htm#i2153251
    UTL_FILE
    ~~~~~~~~~
    http://download-east.oracle.com/docs/cd/B19306_01/appdev.102/b14258/u_file.htm#sthref14093
    Message was edited by:
    Nicloei W
    Message was edited by:
    Nicloei W

  • How to validate data is in specific list while loading from SQL*Loader

    I have a sample data file like below
    1,name1,05/02/2012 10:00:00,blue
    2,name2,06/02/2012 10:00:00,red
    3,name3,07/02/2012 10:00:00,yellow
    4,name4,08/02/2012 10:00:00,white
    I would like to validate 4Th column to be a valid color (ie) All color should be in a specific list, if it is not in the lis then the record should do to bad/discard file
    How can do that while loading Data From SQL*Loader?

    user8860934 wrote:
    I have a sample data file like below
    1,name1,05/02/2012 10:00:00,blue
    2,name2,06/02/2012 10:00:00,red
    3,name3,07/02/2012 10:00:00,yellow
    4,name4,08/02/2012 10:00:00,white
    I would like to validate 4Th column to be a valid color (ie) All color should be in a specific list, if it is not in the lis then the record should do to bad/discard file
    How can do that while loading Data From SQL*Loader?Probably a lot easier with an EXTERNAL TABLE (they're much more flexible).
    Is SQL Loader a mandatory requirement for some reason?

  • Dual Language Data upload using SQL*Loader

    I have to load data consisting of English and Traditional Chinese characters into 8.1.5 from external files. These external files are in MS Excel.
    Is there any way that I can user SQL*Loader for this ? If yes, how ? If not, is there any other way out.
    The NLS in the database is currently AMERICAN.
    null

    ORA-00604 error occurred at recursive SQL level string
    Cause: An error occurred while processing a recursive SQL statement (a statement applying to internal dictionary tables).
    Action: If the situation described in the next error on the stack can be corrected, do so; otherwise contact Oracle Support Services
    This kind of error occurs when data dictionary is
    query a lot.
    Joel P�rez

  • Export data ready for sql loader

    Hi,
    I need to export some data from my database ready to be used by sql loader ... what are tools can I use? Using sqlplus (i.e. spool file.csv select c1||";"||c2||";" from table; spool off) is the only tool I can use?

    Hello,
    You have multiple choices depending upon your oracle version on both source and target.
    *1. Conventional export/import (no need to generate csv)*
    *2. Datapump (10g)*
    *3. Extracting .csv file using sql and loading using sqlldr*
    *4. External table unloading data using datapump.*
    *5. External table using .csv file generated via sql script.*
    Regards

  • How to handle Multiple date formats for the same date field in SQL*Loader

    Dear All,
    I got a requirement where I need to get data from a text file and insert the same into oracle table.
    I am using SQL*Loader to populate the data from the text file into my table.
    The file has one field where I am expecting date date data in multiple formats, like dd/mon/yyyy, yyyy/dd/mon, yyyy/mon/dd, ,mm/dd/yyyy, mon/dd/yyyy.
    While using SQL*Loader, I can see Loading is failing for records where we have formats like yyyy/dd/mon, yyyy/mon/dd, mon/dd/yyyy.
    Is there any way in SQL*Loader where we can mention all these date formats so that this date data should go smoothly into the underlying date column in the table.
    Appreciate your response on this.
    Thanks,
    Madhu K.

    The point being made was, are you sure that you can uniquely identify a date format from the value you receieve? Are you sure that the data stored is only of a particular limited set of formats?
    e.g. if you had a value of '07/08/03' how do you know what format that is?
    It could be...
    7th August 2003 (thus assuming it's DD/MM/RR format)
    or
    8th July 2003 (thus assuming it's MM/DD/RR format)
    or
    3rd August 2007 (thus assuming it's RR/MM/DD format)
    or
    8th March 2007 (thus assuming it's RR/DD/MM format)
    or even more obscurely...
    3rd July 2008 (MM/RR/DD)
    or
    7th March 2008 (DD/RR/MM)
    Do you have any information to tell you what formats are valid that would allow you to be specific and know what date format is meant?
    This is a classic example of why dates should be stored on the database using DATE datatype and not VARCHAR2. It can lead to corruption of data, especially if the date can be entered in any format a user wishes.

  • Need to Skip the fields duaring the data upload by SQL*Loader.

    Hi all,
    I am not able to get how to skip some fileds in the file while uploading it thru SQL*Loader. Let us have a look the below scenario.
    The table has 4 columns. but in the input file came up with 6 fields. Let us assume the four fields came as first in the order. Then, we can populate the data into the 4 columns of the table. At the same time we want to skip the remaining fields. Even those fields/columns are no longer available in the database table.
    For example:
    create table data_temp(sno number,name varchar2(10 char),sex char(1),loc varchar2(20 char));
    Data file
    sno,name,sex,loc,organization,contact
    1,ram,M,India,HP,998976789
    2,Shesha,M,India,IBM,7890808098
    Control_file
    OPTIONS(SKIP=1)
    LOAD DATA
    INFILE *
    APPEND INTO TABLE data_temp
    FIELDS TERMINATED BY ',' optionally enclosed by '"'
    TRAILING NULLCOLS
    sno "trim(:sno)",
    name "SUBSTR(trim(:name),1,20)" ,
    sex "SUBSTR(trim(:sex),1,1)",
    loc "SUBSTR(trim(:loc),1,20)" ,
    Please suggest me how to implement the above scenario in the control file.
    Thanks in Advance!!
    Regards,
    Vissu.....

    Use FILLER. Control file:
    OPTIONS(SKIP=1)
    LOAD DATA
    INFILE *
    APPEND INTO TABLE data_temp
    FIELDS TERMINATED BY ',' optionally enclosed by '"'
    TRAILING NULLCOLS
    sno "trim(:sno)",
    name "SUBSTR(trim(:name),1,20)" ,
    sex "SUBSTR(trim(:sex),1,1)",
    loc "SUBSTR(trim(:loc),1,20)" ,
    organization filler,
    contact filler
    begindata
    sno,name,***,loc,organization,contact
    1,ram,M,India,HP,998976789
    2,Shesha,M,India,IBM,7890808098
    {code}
    Now:
    {code}
    SQL> create table data_temp(sno number,name varchar2(10 char),sex char(1),loc varchar2(20 char));
    Table created.
    SQL> host sqlldr scott/tiger control=c:\temp\vissu.ctl log=c:\temp\vissu.log
    SQL> select * from data_temp
      2  /
           SNO NAME       S LOC
             1 ram        M India
             2 Shesha     M India
    SQL>
    {code}
    SY.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               

  • Load Xml Data in oracle (SQL LOADER)

    i have written a sql loader script given below....
    load data
    infile 'contact1.xml' "str '</CONTACT>'"
    truncate
    into table XmlTest
    dummy filler terminated by ">",
    Location_id enclosed by "<Location_id>" and "</Location_id>",
    fname enclosed by "<fname>" and "</fname>",
    lname enclosed by "<lname>" and "</lname>"
    the content of the contact1.xml is given below.....
    <CONTACT>
    <Location_id>1</Location_id>
    <fname>Robert</fname>
    <lname>Jones</lname>
    </CONTACT>
    <CONTACT>
    <Location_id>1</Location_id>
    <fname>Robert</fname>
    <lname>Jones</lname>
    </CONTACT>
    the script works fine for the given xml... but if one of the tag in the xml is changed to below syntax....
    <CONTACT Location_id="1">
    <fname>Robert</fname>
    <lname>Jones</lname>
    </CONTACT>
    <CONTACT Location_id="1">
    <fname>Robert</fname>
    <lname>Jones</lname>
    </CONTACT>
    what modification is required in the script to read such xml...Kindly Help!!

    And in what way does this relate to SQL and PL/SQL?
    There is a demo of loading XML in Morgan's Library under SQL*Loader.
    But as your question is off-topic here ... if the demo does not help you please continue this by starting a thread in the correct forum: Database General.
    Thanks.

Maybe you are looking for

  • E72 Mail for Exchange calendar sync

    I have problems with repeating calendar events. Example: I create event 1.6.2011, repeating every week for one year. I have set in phone to remove calendar events older than 3 week. After 3 week from 1.6.2011 all events from that repeating meeting ar

  • How to upgrade Stellent 7.5.1 to Oracle Content Server 10gR3

    Hi, I need to upgrade Stellent 7.5.1 to OCS 10gR3. Can someone provide the steps. Do I need to install OCS 10gR3 and then migrate the folders and content how do I migrate all the folder, content to 10gR3. Is there any documentation, blogs, steps that

  • ASCII to EBCDIC conversion

    Hi there! I am working on an Oracle data extract project. My output file will send to a DB2 database and a mainframe application. In my file, some fields' type is COMP-3. I use Oracle build-in function convert () to convert to EBCDIC. This works fine

  • Implement History, Back & Forward in custom code

    Hi All We have our custom developed Header Code, where we have controling Personalize, Logout, Favorite & Top level navigation code. I want to implement the SAP oup of box history, Back Forward functionality in custom developed code. Please help me h

  • Macbook Pro Won't Run Past 1GHz Without Its Battery

    Hello, I've been using a Macbook Pro 2.16GHz for half a year. I just realized that the computer will never get beyond 1GHz, both in Windows and Mac OS (I checked with Sisoft, CPU-Z, and CoreDuoTemp) when I remove the battery and run on AC Adapter. Ho