SQL Loader versus External Table

If anyone has worked on external tables please let me know your views.

user637544 wrote:
for sqlldr i follow the following approach
1. truncate table
2. call sqlldr and load records in staging table
3. check for bad records
how do i tuncate, call external table and check bad records for external table.As part of the SQL*Loader control file you can tell it to truncate the table as part of the process before it loads in the data.
The key differences between SQL*Loader and External Tables are:
1. SQL*Loader is an external utility run from the o/s command line, whereas External tables are usable from within SQL and PL/SQL.
2. SQL*Loader can only load data up into the database, whereas External tables can allow you to read files and write files (10g onwards)
3. SQL*Loader has limited control on skipping rows etc. whereas External tables can be queried using SQL and has all the flexibility that SQL can offer.
4. SQL*Loader requires connection information to the database, whereas External tables are already part of the database
5. SQL*Loader can only change the source filename through dynamic o/s level scripts, whereas External tables can have the filename changed using an ALTER TABLE ... LOCATION ... command.
If you want to truncate your staging table before loading it with data from the external table, then you will just have to do that as a seperate step, because the external table is not associated with other tables. Simply truncate the other table and then query the data from the external table into the staging table.
External tables offer a lot more flexibility than using SQL*Loader.

Similar Messages

SQL *Loader and External Table

Hi,
Can anyone tell me the difference between SQL* Loader and External table?
What are the conditions under we can use SQL * Loader and External Table.
Thanx

External tables are accessible from SQL, which generally simplifies life if the data files are physically located on the database server since you don't have to coordinate a call to an external SQL*Loader script with other PL/SQL processing. Under the covers, external tables are normally just invoking SQL*Loader.
SQL*Loader is more appropriate if the data files are on a different server or if it is easier to call an executable rather than calling PL/SQL (i.e. if you have a batch file that runs on a server other than the database server that wants to FTP a data file from a FTP server and then load the data into Oracle).
Justin

Sql loader or external table

Hi all good morning.
can we write a stored procedure that loads a file into the table using sql loader?
can Java call the above stored procedure?
regards
raj
Edited by: user10887630 on Apr 23, 2009 6:18 AM

Are you saying the files themselves can't be located on the server? or just that the process for loading the files can't be located on the server.
If the files themselves cannot reside on the server then you won't be able to use SQL*Loader or External tables from within stored procedures. It would require some clever Java code type stuff to get across to a.n.other machine where the files are stored and get the data.
Typically it is normal for such files to be located on the server. What reasons do the DBA's give for not wanting them there?
DBA's say that application specific files cannot be maintained on the Database serverSo what are all the database data files then? They hold data from the applications. ;)

Sql Loader Vs External table? Which one is preffered?

Hello guru,
We trying to load the data into about 160 tables. Here are the following things we are doing currently,
1. We get the flat file in .txt format
2. We are writing the control file for each flat file for thier respective tables
3.And we are using sql loader command to load this data into the table.
I was wondering if this process is cumbersome, and i m not sure if external table could be simple in loading the tables compared to what we are doing ?And i have not used external tables, so wanted to know which on eis better ? So any idea is greatly appriciated !
FYI.. Version :- Oracle 11g
Thank you so much!

Thanks for you reply justin !
I found the below example for loading the data into external table...
CREATE OR REPLACE DIRECTORY dat_dir AS 'C:\Oradata\Data';
CREATE OR REPLACE DIRECTORY log_dir AS 'C:\Oradata\Log';
CREATE OR REPLACE DIRECTORY bad_dir AS 'C:\Oradata\Bad';
GRANT READ ON DIRECTORY dat_dir TO scott;
GRANT WRITE ON DIRECTORY log_dir TO scott;
GRANT WRITE ON DIRECTORY bad_dir TO scott;
CREATE TABLE revext (person      VARCHAR2(20),
                     rev_jan     NUMBER(4),
                     rev_feb     NUMBER(4),
                     rev_mar     NUMBER(4),
                     rev_apr     NUMBER(4),
                     rev_mai     NUMBER(4),
                     rev_jun     NUMBER(4),
                     rev_jul     NUMBER(4),
                     rev_aug     NUMBER(4),
                     rev_sep     NUMBER(4),
                     rev_oct     NUMBER(4),
                     rev_nov     NUMBER(4),
                     rev_dez     NUMBER(4))
ORGANIZATION EXTERNAL
   TYPE ORACLE_LOADER
   DEFAULT DIRECTORY dat_dir
   ACCESS PARAMETERS
     records delimited by newline
     badfile bad_dir:'revext%a_%p.bad'
     logfile log_dir:'revext%a_%p.log'
     fields terminated by ','
     missing field values are null
     ( person,
       rev_jan,
       rev_feb,
       rev_mar,
       rev_apr,
       rev_mai,
       rev_jun,
       rev_jul,
       rev_aug,
       rev_sep,
       rev_oct,
       rev_nov,
       rev_dez
   LOCATION ('revext.dat')
PARALLEL 4
REJECT LIMIT UNLIMITED;
CREATE TABLE revenue (
    person       VARCHAR2(20),
    month        VARCHAR2(3),
    revenue      NUMBER,
    CONSTRAINT revenue_pk PRIMARY KEY (person,month));
INSERT INTO revenue (person,month,revenue)
   SELECT person,'Jan',rev_jan
   FROM revext--but currently we are using sql loader, our data looks like this
1119Smith      01/01/1982AXYZ corporation xyz corp
1111collen      01/01/1990AABC corporation abc corp
     and control file is like this
INTO TABLE "XYZ_tbl"
   ID                     POSITION(01:05)        CHAR                  "DECODE(RTRIM(:ID), NULL, 'NA', :ID)"       ,
   Name                POSITION(06:15)        CHAR                  "DECODE(RTRIM(:NAME), NULL, 'NA', :Name)"   ,
   Act_dt              POSITION(16:25)        DATE                  "MM/DD/YYYY" NULLIF ACT_DT=BLANKS
My question is, can i use the options like NULLIF/ DECODE or changing datatypes when loading or functions like REPLACE / TO_DATE in External tables ? Any idea? Any example code or SQL will great
Thank you so much!

SQL*Loader or external table for load a MSG (email) file

Hi there!
I'm looking for a way to load an email in a Oracle DB.
I mean, not all the email's body in a column, but to "parse" it in a multi column/table fashion.
Is it possible to do with a sql*loader script or an external table?
I think it is not possible, and that I must switch to XML DB.
Any idea?
Thanks,
Antonio

Hello,
Why don't you just load the entire MSG (email) as clob into one email_body column or whatever column name you want to use.
To load data upto 32k, you can use varchar2(32656) but its not a good idea to load clob in that manner because it's very inconsistent as length can
vary resulting in string literal too long. So you have 2 choices now, first you have to use either procedure or anonymous block to load clob data.
First Method -- I loaded alert.log successfully and you can imagine how big this file can be (5MB in my test case)
CREATE OR REPLACE DIRECTORY DIR AS '/mydirectory/logs';
DECLARE
   clob_data   CLOB;
   clob_file   BFILE;
BEGIN
   INSERT INTO t1clob
   VALUES (EMPTY_CLOB ())
   RETURNING clob_text INTO clob_data;
   clob_file   := BFILENAME ('DIR', 'wwalert_dss.log');
   DBMS_LOB.fileopen (clob_file);
   DBMS_LOB.loadfromfile (clob_data,
                          clob_file,
                          DBMS_LOB.getlength (clob_file)
   DBMS_LOB.fileclose (clob_file);
   COMMIT;
END;Second Method: Use of Sqlldr
Example of controlfile
LOAD DATA
INFILE alert.log "STR '|\n'"
REPLACE INTO table t1clob
   clob_text char(30000000)
)Hope this helps

SQL loader verses External table

hi
Can anybody guide me the advantages of External table over SQL Loader in data warehousing environment.
Or and any URL where I can find suitabel knowledge.
Thanks in advance.
Moloy

here it is.
http://download-uk.oracle.com/docs/cd/B19306_01/server.102/b14215/et_concepts.htm#sthref1672

Sql loader utl_file & external table

can any one let me know the differences between.
1.sql loader
2.utl_file
3.external table
Regards.
Asif.

To expand on Aron's answer....
SQL*Loader - An operating system utility which uses control files (which you create) to load data files onto database tables.
UTL_FILE - A database package which can be used for reading and writing files in any format you care to design programmatically.
External Table - The latest thing which can be used instead of SQL*Loader. This is done from the database end, by creating a table as an external table and pointing it at the source file on the operating system. It also allows information similar to that put in the SQL*Loader control files to be specified against the table. By querying against the table you are in fact querying against the source file. There are some limitation compared to regular database tables such as there is no ability to write to the external table.
;)

SQL Loader AND Temporary Tables

Hi :
Have a Couple of Questions..
(1) Is SQL Loader not part of the ORACLE DB 9.X?? Coz I could not find that exe.
(2) Can we have the Data file on the Client Machine?? OR is it mandatory that the data file should be on the Server to load the data?

SQL loader is part of Oracle9i and you can use its driver to create external table.Moreover for external tables datafiles should be on server.

SQL LOADER USING EXTRNAL TABLE

I have .csv file having around 70k records
in which fields are delimited by tab and
enclosed in double quotes but double quotes may be part of data.
and records are delimited by newline.
After creating external table when I issue SELECT statment
select count(*) from proTxt ;
ERROR at line 1:
ORA-29913: error in executing ODCIEXTTABLEOPEN callout
ORA-29400: data cartridge error
KUP-04020: found record longer than buffer size supported, 524288, in C:\Program Files\Apache Software Foundation\Tomcat
5.5\webapps\tmTest\upload\product\Data\output09_1.txt
ORA-06512: at "SYS.ORACLE_LOADER", line 19
Following is the create table statement:
CREATE TABLE proTxt (PRO_CODE VARCHAR2(30),
PRO_DESC VARCHAR2(500),
PUR_PRICE VARCHAR2(20),
SALE_PRICE VARCHAR2(20)
ORGANIZATION EXTERNAL
TYPE ORACLE_LOADER
DEFAULT DIRECTORY DAT_DIR
ACCESS PARAMETERS
records delimited by NEWLINE SKIP 1
badfile BAD_DIR:'proTxt%a_%p.bad'
logfile LOG_DIR:'proTxt%a_%p.log'
fields terminated by X'9' OPTIONALLY ENCLOSED BY '"' AND '"'
missing field values are null
( PRO_CODE,
PRO_DESC,
PUR_PRICE,
SALE_PRICE
LOCATION ('output09_1.txt')
PARALLEL 4
REJECT LIMIT UNLIMITED;
record size is not large.
Log file :
LOG file opened at 12/05/12 20:25:40
KUP-04020: found record longer than buffer size supported, 524288, in C:\Program Files\Apache Software Foundation\Tomcat 5.5\webapps\tmTest\upload\product\Data\output09_1.txt
KUP-04053: record number 2
data file
PRO_CODE     PRO_DESC     PUR_PRICE     SALE_PRICE
"0000336658"     "BEARING"     "Rs.0.00"     "Rs.0.00"
"0000790028"     "SEAL"     "Rs.76.00"     "Rs.90.00"
"0000790118"     "SPRING"     "Rs.24.00"     "Rs.28.00"
"0000792284"     "F.BRK.CAL.W/O PA"     "Rs.2,627.00"     "Rs.3,100.00"
"0000792285"     "F.BRK.CAL.W/O PA"     "Rs.2,627.00"     "Rs.3,100.00"
"0005896322"     "PISTON, RING"     "Rs.5,000.00"     "Rs.5,900.00"
"0005896323"     "PISTONS, RINGS AND P"     "Rs.17,755.00"     "Rs.20,951.00"
"0005896559"     "PISTON, RINGS AND PI"     "Rs.5,000.00"     "Rs.5,900.00"

Hi,
when i used
records delimited by *'\r'*
then 4226 record written to table
but enclosed charcter double quotes["] were also written and
there is some space between charcters
" 0 0 0 0 3 3 6 6 5 8 " " B E A R I N G " " R s . 0 . 0 0 " " R s . 0 . 0 0 "
" 0 0 0 0 8 5 6 7 0 7 " " P L U G " " R s . 0 . 0 0 " " R s . 0 . 0 0 "
Definitely this is "External Table with Flatfile Moved Across Platforms" issue.
when I opened .csv file in excel and saved as tab delimited it works fine.
But I do not know plateform of data file.
How to know the CHARACTERSET of data file
Log file
Field Definitions for table PROTXT
Record format DELIMITED, delimited by
Data in file has same endianness as the platform
Rows with all null fields are accepted
Fields in Data Source:
PRO_CODE CHAR (255)
Terminated by "9"
Enclosed by """ and """
Trim whitespace same as SQL Loader
PRO_DESC CHAR (255)
Terminated by "9"
Enclosed by """ and """
Trim whitespace same as SQL Loader
PUR_PRICE CHAR (255)
Terminated by "9"
Enclosed by """ and """
Trim whitespace same as SQL Loader
SALE_PRICE CHAR (255)
Terminated by "9"
Enclosed by """ and """
Trim whitespace same as SQL Loader

Error when loading from External Tables in OWB 11g

Hi,
I face a strange problem while loading data from flat file into the External Tables.
ORA-12899: value too large for column EXPIRED (actual: 4, maximum: 1)
error processing column EXPIRED in row 9680 for datafile <data file location>/filename.dat
In a total of 9771 records nearly 70 records are rejected due to the above mentioned error. The column (EXPIRED) where the error being reported doesn't have value greater than 1 at all. I suspect it to be a different problem.
Example: One such record that got rejected is as follows:
C|234|Littérature commentée|*N*|2354|123
highlightened in Bold is the EXPIRED Column.
When I tried to insert this record into the External Table using UTL_FILE Utility it got loaded successfully. But when I try with the file already existing in the file directory it again fails with the above error, and I would like to mention that all the records which have been loaded are not Ok, please have a look at the DESCRIPTION Column which is highlightened. The original information in the data file looks like:
C|325|*Revue Générale*|N|2445|132
In the External Table the Description Value is replaced by the inverted '?' as follows:
Reue G¿rale
Please help.
Thanks,
JL.

user1130292 wrote:
Hi,
I face a strange problem while loading data from flat file into the External Tables.
ORA-12899: value too large for column EXPIRED (actual: 4, maximum: 1)
error processing column EXPIRED in row 9680 for datafile <data file location>/filename.dat
In a total of 9771 records nearly 70 records are rejected due to the above mentioned error. The column (EXPIRED) where the error being reported doesn't have value greater than 1 at all. I suspect it to be a different problem.
Example: One such record that got rejected is as follows:
C|234|Littérature commentée|*N*|2354|123
highlightened in Bold is the EXPIRED Column.
When I tried to insert this record into the External Table using UTL_FILE Utility it got loaded successfully. But when I try with the file already existing in the file directory it again fails with the above error, and I would like to mention that all the records which have been loaded are not Ok, please have a look at the DESCRIPTION Column which is highlightened. The original information in the data file looks like:
C|325|*Revue Générale*|N|2445|132
In the External Table the Description Value is replaced by the inverted '?' as follows:
Reue G¿rale
Please help.
Thanks,
JL.sorry, couldnt see the highlighted test.could you plesae enclsoe it in tags
also post the table definition with attributes. BTW..Whats your NLS_LANGUAGE SET TO?

Sql*loader and nested tables

I'm having trouble loading a nested table via sqlldr in Oracle 10g and was hoping someone could point me in the right direction. I keep getting the following error:
SQL*Loader-403: Referenced column not present in table mynamespace.mytable
Here's an overview of my type and table definitions:
create type mynamespace.myinfo as object
i_name varchar2(64),
i_desc varchar2(255)
create TYPE mynamespace.myinfotbl as TABLE of mynamespace.myinfo;
create table mynamespace.mytable
Info mynamespace.myinfotbl,
note varchar2(255)
NESTED TABLE Info STORE AS mytable_nested_tab;
My control file looks like this:
load data
infile 'mydatafile.csv'
insert into table mynamespace.mytable
fields terminated by ',' optionally enclosed by '"'
TRAILING NULLCOLS
Info nested table count(6)
Info column object
i_name char(64),
i_desc char(255)
note
Example mydatafile.csv would be something like:
lvl1,des1,lvl2,des2,lvl3,des3,lvl4,des4,lvl5,des5,lvl6,des6,a test data set
I can't figure out why sqlldr keeps rejecting this control file. I'm using 'direct=false' in my .par file.
Any hints?

I just noticed that my email is wrong. If you can help, plese send email to [email protected]
thanks.

Selective load in external table

Hi all,
I need to load data into my external table based on conditions. If i use the 'load when' im able to load the data into the table but my requirement is to load data into different columns of the same table based on the conditional "when" clause.
This is equivalent to wht we do in .ctl file using loader..Th scenario being:
LOAD DATA
INFILE '/m018/applmgr/ERP1/modap/1.0.0/bin/BNKSTMT260508.txt'
APPEND
INTO TABLE
MODAP_IN_BNK_RECON_TBL
WHEN record_type = '01'
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
TRAILING NULLCOLS
(RECORD_TYPE POSITION(1)
,SENDER_IDENTI
,RECEIVER_IDENTI
     ,FILE_CREATION_DATE DATE "YYMMDD" NULLIF FILE_CREATION_DATE=BLANKS
     ,FILE_CREATION_TIME
     ,FILE_SEQ_NO
     ,RECORD_LENGTH
     ,BLOCKING_FACTOR
     ,VERSION_NO
INTO TABLE
MODAP_IN_BNK_RECON_TBL
WHEN record_type = '02'
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
TRAILING NULLCOLS
(RECORD_TYPE POSITION(1) "TRIM(:RECORD_TYPE)"
,RECEIVER_IDENTI "TRIM(:RECEIVER_IDENTI)"
,ORIGINATOR_IDENTI "TRIM(:ORIGINATOR_IDENTI)"
     ,GROUP_STATUS "TRIM(:GROUP_STATUS)"
     ,AS_OF_DATE DATE "YYMMDD" NULLIF AS_OF_DATE=BLANKS
     ,AS_OF_TIME "TRIM(:AS_OF_TIME)"
     ,CURRENCY_CODE "TRIM(:CURRENCY_CODE)"
I want to replicate the same in external table. Please suggest me a suitable solution for this at the earliest.
Regards,
Balaji V

Here is a sample I used:
DROP TABLE myschema.mytable;
CREATE TABLE myschema.mytable (
"ID" VARCHAR2(12),
"RESP_DOB" DATE,
"RESP_FAM_REL" VARCHAR2(5),
"PET_FIRST_NAME" VARCHAR2(11),
"PET_MIDDLE_NAME" VARCHAR2(11),
"PET_LAST_NAME" VARCHAR2(17),
"PET_SUFFIX" VARCHAR2(1)
ORGANIZATION EXTERNAL (
TYPE ORACLE_LOADER
DEFAULT DIRECTORY "MY_DIRECTORY"
ACCESS PARAMETERS(
RECORDS DELIMITED BY NEWLINE
BADFILE 'the_BAD.TXT'
DISCARDFILE 'the_DISCARD.TXT'
LOGFILE 'the_LOG.TXT'
FIELDS
MISSING FIELD VALUES ARE NULL
ID POSITION(1:12),
RESP_DOB POSITION(98:105) DATE MASK "mmddyyyy",
RESP_FAM_REL POSITION(106:110) NULLIF RESP_FAM_REL=".",
PET_FIRST_NAME POSITION(111:121) NULLIF PET_FIRST_NAME=".",
PET_MIDDLE_NAME POSITION(122:132) NULLIF PET_MIDDLE_NAME=".",
PET_LAST_NAME POSITION(133:149) NULLIF PET_LAST_NAME=".",
PET_SUFFIX POSITION(150:150) NULLIF PET_SUFFIX="."
LOCATION ('the_file.txt')
);

Loading millions of rows using SQL*loader to a table with constraints

I have a table with constraints and I need to load millions of rows in it using SQL*Loader.
What is the best way to do this, means what SQL*Loader options to use, for getting the best loading performance and how to deal with constraints?
Regards

- check if your table has check constraints (like column not null)
if you trust the data in the file you have to load you can disable this constrainst and after the loader enable this constrainst.
- Check if you can modify the table and place it in nologging mode (generate less redo but ONLY is SOME Conditions)
Hope it helps
Rui Madaleno

Sql*loader and relate tables

I have a file formated like the following:
111|1,2,3|
222|7,8|
Oracle relate table looks like:
tab1 (id1 number, id2 number)
Can sql*loader handle this kind of data? I should have 5 records in tab1 once the load is finished.
tab1 should have:
111,1
111,2
111,3
222,7
222,8

NO
It will not work like that.
You have to have data file like
111|1
111|2
111|3
222|7
222|8
Prashant

Sql loader master detail tables

Hi
I am trying to load master detail tables using sql loader. I have a sequence in the master table as primary key and have to load primary key in detail table. is there any way i can query up the primary key by name in the control file. please whats the best way to do this.
can i query up the primary key in the master table by the name in detail table which is jan09.
e.g
master data
jan09,description
detail data
jan09,'test1','100,'y'
LOAD DATA
INFILE 'test.data'
insert
INTO TABLE master_table
fields terminated by ',' optionally enclosed by '"' trailing nullcols
(id position(1:2) "myseq.nextval",
code char,
creation_date date sysdate,
last_update_date date sysdate,
into TABLE BDETAILS
fields terminated by ',' optionally enclosed by '"' trailing nullcols
(id number "select id from master_table where code=" || :code
name char,
amt decimal,
flag char
)

Hey Here is what you need.
You will have to understand and test this properly.
It took me quiet some time before i could get to this stage
Here is the test data with all the commands.
As far the ID in the detail table is concerned you have no option but to update that column once the loading is complete.
1) create Table
CREATE TABLE master_table(ID NUMBER,code VARCHAR2(50),creation DATE,last_update DATE);
CREATE TABLE bdetails(ID NUMBER,NAME VARCHAR2(100),amt NUMBER,flag VARCHAR2(10));
ALTER TABLE master_table ADD (chc VARCHAR2(10));2) Create Sequence
CREATE SEQUENCE testseq INCREMENT BY 1;3) CTL FILE
LOAD DATA
INFILE 'c:\sqlldr\test.txt' /*path of data file */
append
INTO TABLE master_table
when (1) ='M' /*to check if the record in the data file is a master record */
fields terminated by ',' optionally enclosed by '"' trailing nullcols
(id expression "testseq.nextval", /*select next value from sequence */
mcol1 filler, /*FILLER keyword is sused to skip first column from data file */
chc,
code ,
creation "sysdate",
last_update "sysdate"
Into TABLE BDETAILS
when (1) ='D' /*to check if the record in the data file is a Detail record */
fields terminated by ',' optionally enclosed by '"' trailing nullcols
(id expression "testseq.currval", /* UNFORTUNATELY THIS PICKS UP THE LAST ID OF THE MASTER RECORD SESSION STUFF GOING ON HERE*/
col1 filler position(1:2), /* specifying this POSITION(1:2) is very very important as you need to reset the cursor to start of the line for the next record */
name ,
amt ,
flag )4) SQLLDR Command
sqlldr scott/tiger data=c:\sqlldr\test.txt control=c:\sqlldr\1.ctl
5) data file
M,jan09,description
D,test1,100,y
M,feb09,description1
D,test2,200,x
M,mar09,description2
D,test3,200,xHope this helps you and give you a lead how to proceed.
Cheers!!!
Bhushan

SQL Loader versus External Table

Similar Messages

Maybe you are looking for