Sql loader and bulk data
hi,
I want to insert 100,000 records daily in a table for the first month and then in next month these records are going to be replaced by new updated records.
there might be few addition and deletion in the previous records also.
actually its consumer data so there might be few consumer who have withdrawn the utility and there will be some more consumer added in the database.
but almost 99% of the previous month data have to be updated/replaced with the fresh month data.
For instance, what i have in my mind is that i will use sql loader to load data for the first month and then i will delete the previous data using sqlPlus and load the fresh month data using sql loader again.
1. Is this ok ? or there is some better solution to this.
2. I have heard of external files, are they feasible in my scenario?
3. I have planned that i will make scripts for sqlPlus and Loader and use them in batch files. (OS windows 2003 server, Oracle 9i database). is there some better choice to make all the procedure automatic?
looking for your suggestions
nadeem ameer
I would suggest u use External tables since its more flexible then
sqlloader & is a better option.
For using external tables
1)u will have to create a directory first
2)Generally creation od directory is done by sys,hence after creating the directory
privileges read & write to be provided to user .
3)Creation of external tables.
4) Now use the table as a normal table to insert ,update delete in
ur table.
U can get more information from
http://www.oracle-base.com/articles/9i/SQLNewFeatures9i.php#ExternalTables
Create Directory <directory_name> as <Directory path where file be present>
Grant read,write on directory <directory_name> to <username>
CREATE TABLE <table_name>
(<column names>)
ORGANIZATION EXTERNAL
TYPE ORACLE_LOADER
DEFAULT DIRECTORY ,directory_name>
ACCESS PARAMETERS
RECORDS DELIMITED BY NEWLINE
FIELDS TERMINATED BY ','
LOCATION (<filename>)
PARALLEL 5
REJECT LIMIT 200;
Hope this helps.
Similar Messages
-
Problem with SQL*Loader and different date formats in the same file
DB: Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bi
System: AIX 5.3.0.0
Hello,
I'm using SQL*Loader to import semi-colon separated values into a table. The files are delivered to us by a data provider who concatenates data from different sources and this results in us having different date formats within the same file. For example:
...;2010-12-31;22/11/1932;...
I load this data using the following lines in the control file:
EXECUTIONDATE1 TIMESTAMP NULLIF EXECUTIONDATE1=BLANKS "TO_DATE(:EXECUTIONDATE1, 'YYYY-MM-DD')",
DELDOB TIMESTAMP NULLIF DELDOB=BLANKS "TO_DATE(:DELDOB, 'DD/MM/YYYY')",
The relevant NLS parameters:
NLS_LANGUAGE=FRENCH
NLS_DATE_FORMAT=DD/MM/RR
NLS_DATE_LANGUAGE=FRENCH
If I load this file as is the values loaded into the table are 31 dec 2010 and 22 nov *2032*, aven though the years are on 4 digits. If I change the NLS_DATE_FORMAT to DD/MM/YYYY then the second date value will be loaded correctly, but the first value will be loaded as 31 dec *2020* !!
How can I get both date values to load correctly?
Thanks!
SylvainThis is very strange, after running a few tests I realized that if the year is 19XX then it will get loaded as 2019, and if it is 20XX then it will be 2020. I'm guessing it may have something to do with certain env variables that aren't set up properly because I'm fairly sure my SQL*Loader control file is correct... I'll run more tests :-(
-
SQL*Loader and binary data
i have a C routine that builds SQL*Loader input files. the input files contain multiple records, with a couple of integer columns and a raw(1400) field. the control file specifies a record separator of '|', which seems weird (having text in the middle of raw data) but also is somewhat co-operative (see below).
so i basically write each integer to the file, then a short (2-byte) length value for the raw field, then the raw field. then the '|' separator.
i've noticed that if the size of raw field is 400 bytes or less, everything works fine, i get the correct number of records in the database.
unfortunately, with a size of 401 or more, SQL*Loader parses the thing into twice as many records as it should. so if i've written 3 records to my input data file, with each record's raw field at 400 bytes or less i get 3 records loaded. but any with a raw field of 401+, i get two records for each.
any ideas why? and how to correct this? also, any ideas on a better way to do this? all the examples of large data in the online doc and the o'reilly book favor showing examples with large character data, which does me not much good.
tia. johni have a C routine that builds SQL*Loader input files. the input files contain multiple records, with a couple of integer columns and a raw(1400) field. the control file specifies a record separator of '|', which seems weird (having text in the middle of raw data) but also is somewhat co-operative (see below).
so i basically write each integer to the file, then a short (2-byte) length value for the raw field, then the raw field. then the '|' separator.
i've noticed that if the size of raw field is 400 bytes or less, everything works fine, i get the correct number of records in the database.
unfortunately, with a size of 401 or more, SQL*Loader parses the thing into twice as many records as it should. so if i've written 3 records to my input data file, with each record's raw field at 400 bytes or less i get 3 records loaded. but any with a raw field of 401+, i get two records for each.
any ideas why? and how to correct this? also, any ideas on a better way to do this? all the examples of large data in the online doc and the o'reilly book favor showing examples with large character data, which does me not much good.
tia. john -
SQL Loader and formatting dates
I've got dates formatted like so: 2012-05-10T17:04:51-08:00
How can I get SQL Loader to load these into a Date column??
ThanksThanks for the reply-
SQL Loader is running on Win, Oracle 11 on Linux
The format is standard UTC
YYYY-MM-DDTHH24:MI:SS±HH:MM Where the ±HH:MM refers to time zone offset from GMT. The "T" is just a separator between date and time (and is always "T").
I looked at your references, and tried a few dozen variants without success.
Most of my attempts have been using something similar to this
"YYYY-MM-DDTHH24:MI:SSTZH:TZM"
or this:
"YYYY-MM-DD'T'HH24:MI:SSTZH:TZM"
or this:
'YYYY-MM-DD"T"HH24:MI:SSTZH:TZM'
Thanks -
Import and process larger data with SQL*Loader and Java resource
Hello,
I have a project to import data from a text file in a schedule. A lager data, with nearly 20,000 record/1 hours.
After that, we have to analysis the data, and export the results into a another database.
I research about SQL*Loader and Java resource to do these task. But I have no experiment about that.
I'm afraid of the huge data, Oracle could be slowdown or the session in Java Resource application could be timeout.
Please tell me some advice about the solution.
Thank you very much.With '?' mark i mean " How i can link this COL1 with column in csv file ? "
Attilio -
SQL Loader and Insert Into Performance Difference
Hello All,
Im in a situation to measure performance difference between SQL Loader and Insert into. Say there 10000 records in a flat file and I want to load it into a staging table.
I know that if I use PL/SQL UTL_FILE to do this job performance will degrade(dont ask me why im going for UTL_FILE instead of SQL Loader). But I dont know how much. Can anybody tell me the performance difference in % (like 20% will decrease) in case of 10000 records.
Thanks,
Kannan.Kannan B wrote:
Do not confuse the topic, as I told im not going to use External tables. This post is to speak the performance difference between SQL Loader and Simple Insert Statement.I don't think people are confusing the topic.
External tables are a superior means of reading a file as it doesn't require any command line calls or external control files to be set up. All that is needed is a single external table definition created in a similar way to creating any other table (just with the additional external table information obviously). It also eliminates the need to have a 'staging' table on the database to load the data into as the data can just be queried as needed directly from the file, and if the file changes, so does the data seen through the external table automatically without the need to re-run any SQL*Loader process again.
Who told you not to use External Tables? Do they know what they are talking about? Can they give a valid reason why external tables are not to be used?
IMO, if you're considering SQL*Loader, you should be considering External tables as a better alternative. -
SQL *Loader and External Table
Hi,
Can anyone tell me the difference between SQL* Loader and External table?
What are the conditions under we can use SQL * Loader and External Table.
ThanxExternal tables are accessible from SQL, which generally simplifies life if the data files are physically located on the database server since you don't have to coordinate a call to an external SQL*Loader script with other PL/SQL processing. Under the covers, external tables are normally just invoking SQL*Loader.
SQL*Loader is more appropriate if the data files are on a different server or if it is easier to call an executable rather than calling PL/SQL (i.e. if you have a batch file that runs on a server other than the database server that wants to FTP a data file from a FTP server and then load the data into Oracle).
Justin -
SQL*Loader and timestamp values
I'm loading timestamp values (among other data) from a text file into Oracle using SQL*Loader.
The data I load is formatted like the following:
2008/11/13 23:55:21.366
2008/11/13 23:55:22.782
2008/11/13 23:55:25.879
Hence my control file look like this:
TSTAMP TIMESTAMP "YYYY/MM/DD HH24:MI:SS.FF",
The timestamp data in the input file are in UTC, however I load into a column that is of type "TIMESTAMP(3)", i.e. in server time. This is on purpose. I do not want the data to be in UTC when I look at them.
Therefore after the load I have to manually do
UPDATE mytable
SET tstamp = tstamp + (tstamp - sys_extract_utc(tstamp));
This update-after-loading actually takes quite some time due to the amount of records. I would like to avoid it and instead do it as part of the load. Is this possible somehow?
I'm on Oracle 10.2.
Thanks.Hi
What about setting a special Time Zone in your database?
You have two options when setting which time zone the database belongs to. You can either qualify it as a displacement from GMT/UTC in the format of 'hh:mm' or you can specify it as a name that has an entry in the V$TIMEZONE table.
select tzname,tzabbrev from V$TIMEZONE_NAMES;
select DBTIMEZONE from dual;
ALTER database SET TIME_ZONE = 'Denmark/Copenhagen';
select SESSIONTIMEZONE from dual;
select CURRENT_TIMESTAMP from dual;
See that link
http://download.oracle.com/docs/cd/B19306_01/server.102/b14231/create.htm#sthref319
Edited by: Hub on Dec 7, 2008 3:58 PM -
Help in calling sql loader and an oracle procedure in a script
Hi Guru's
please help me in writing an unix script which will call sql loader and also an oracle procedure..
i wrote an script which is as follows.
!/bin/sh
clear
#export ORACLE_SID='HOBS2'
sqlldr USERID=load/ps94mfo16 CONTROL=test_nica.ctl LOG=test_nica.log
retcode=`echo $?`
case "$retcode" in
0) echo "SQL*Loader execution successful" ;;
1) echo "SQL*Loader execution exited with EX_FAIL, see logfile" ;;
2) echo "SQL*Loader execution exited with EX_WARN, see logfile" ;;
3) echo "SQL*Loader execution encountered a fatal error" ;;
*) echo "unknown return code";;
esac
sqlplus USERID=load/ps94mfo16 << EOF
EXEC DO_TEST_SHELL_SCRIPT
it is loading the data in to an oracle table
but the procedure is not executed..
any valuable suggestion is highly appriciated..
Cheersmultiple duplicate threads:
to call an oracle procedure and sql loader in an unix script
Re: Can some one help he sql loader issue. -
HELP: SQL*LOADER AND Ref Column
Hallo,
I have already posted and I really need help and don't come further with this
I have the following problem. I have 2 tables which I created the following way:
CREATE TYPE gemark_schluessel_t AS OBJECT(
gemark_id NUMBER(8),
gemark_schl NUMBER(4),
gemark_name VARCHAR2(45)
CREATE TABLE gemark_schluessel_tab OF gemark_schluessel_t(
constraint pk_gemark PRIMARY KEY(gemark_id)
CREATE TYPE flurstueck_t AS OBJECT(
flst_id NUMBER(8),
flst_nr_zaehler NUMBER(4),
flst_nr_nenner NUMBER(4),
zusatz VARCHAR2(2),
flur_nr NUMBER(2),
gemark_schluessel REF gemark_schluessel_t,
flaeche SDO_GEOMETRY
CREATE TABLE flurstuecke_tab OF flurstueck_t(
constraint pk_flst PRIMARY KEY(flst_id),
constraint uq_flst UNIQUE(flst_nr_zaehler,flst_nr_nenner,zusatz,flur_nr),
flst_nr_zaehler NOT NULL,
flur_nr NOT NULL,
gemark_schluessel REFERENCES gemark_schluessel_tab
Now I have data in the gemark_schluessel_tab which looks like this (a sample):
1 101 Borna
2 102 Draisdorf
Now I wanna load data in my flurstuecke_tab with SQL*Loader and there I have problems with my ref column gemark_schluessel.
One data record looks like this in my file (it is without geometry)
1|97|7||1|1|
If I wanna load my data record, it does not work. The reference (the system generated OID) should be taken from gemark_schluessel_tab.
LOAD DATA
INFILE *
TRUNCATE
CONTINUEIF NEXT(1:1) = '#'
INTO TABLE FLURSTUECKE_TAB
FIELDS TERMINATED BY '|'
TRAILING NULLCOLS (
flst_id,
flst_nr_zaehler,
flst_nr_nenner,
zusatz,
flur_nr,
gemark_schluessel REF(CONSTANT 'GEMARK_SCHLUESSEL_TAB',GEMARK_ID),
gemark_id FILLER
BEGINDATA
1|97|7||1|1|
Is there a error I made?
Thanks in advance
Tigmultiple duplicate threads:
to call an oracle procedure and sql loader in an unix script
Re: Can some one help he sql loader issue. -
Problem : Load PDF or similiar files( stored at operating system) into an oracle table using SQl*Loader .
and than Unload the files back from oracle tables to prevoius format.
I 've used SQL*LOADER .... " sqlldr " command as :
" sqlldr scott/[email protected] control=c:\sqlldr\control.ctl log=c:\any.txt "
Control file is written as :
LOAD DATA
INFILE 'c:\sqlldr\r_sqlldr.txt'
REPLACE
INTO table r_sqlldr
Fields terminated by ','
id sequence (max,1) ,
fname char(20),
data LOBFILE(fname) terminated by EOF )
It loads files ( Pdf, Image and more...) that are mentioned in file r_sqlldr.txt into oracle table r_sqlldr
Text file ( used as source ) is written as :
c:\kalam.pdf,
c:\CTSlogo1.bmp
c:\any1.txt
after this load ....i used UTL_FILE to unload data and write procedure like ...
CREATE OR REPLACE PROCEDURE R_UTL AS
l_file UTL_FILE.FILE_TYPE;
l_buffer RAW(32767);
l_amount BINARY_INTEGER ;
l_pos INTEGER := 1;
l_blob BLOB;
l_blob_len INTEGER;
BEGIN
SELECT data
INTO l_blob
FROM r_sqlldr
where id= 1;
l_blob_len := DBMS_LOB.GETLENGTH(l_blob);
DBMS_OUTPUT.PUT_LINE('blob length : ' || l_blob_len);
IF (l_blob_len < 32767) THEN
l_amount :=l_blob_len;
ELSE
l_amount := 32767;
END IF;
DBMS_LOB.OPEN(l_blob, DBMS_LOB.LOB_READONLY);
l_file := UTL_FILE.FOPEN('DBDIR1','Kalam_out.pdf','w', 32767);
DBMS_OUTPUT.PUT_LINE('File opened');
WHILE l_pos < l_blob_len LOOP
DBMS_LOB.READ (l_blob, l_amount, l_pos, l_buffer);
DBMS_OUTPUT.PUT_LINE('Blob read');
l_pos := l_pos + l_amount;
UTL_FILE.PUT_RAW(l_file, l_buffer, TRUE);
DBMS_OUTPUT.PUT_LINE('writing to file');
UTL_FILE.FFLUSH(l_file);
UTL_FILE.NEW_LINE(l_file);
END LOOP;
UTL_FILE.FFLUSH(l_file);
UTL_FILE.FCLOSE(l_file);
DBMS_OUTPUT.PUT_LINE('File closed');
DBMS_LOB.CLOSE(l_blob);
EXCEPTION
WHEN OTHERS THEN
IF UTL_FILE.IS_OPEN(l_file) THEN
UTL_FILE.FCLOSE(l_file);
END IF;
DBMS_OUTPUT.PUT_LINE('Its working at last');
END R_UTL;
This loads data from r_sqlldr table (BOLBS) to files on operating system ,,,
-> Same procedure with minor changes is used to unload other similar files like Images and text files.
In above example : Loading : 3 files 1) Kalam.pdf 2) CTSlogo1.bmp 3) any1.txt are loaded into oracle table r_sqlldr 's 3 rows respectively.
file names into fname column and corresponding data into data ( BLOB) column.
Unload : And than these files are loaded back into their previous format to operating system using UTL_FILE feature of oracle.
so PROBLEM IS : Actual capacity (size ) of these files is getting unloaded back but with quality decreased. And PDF file doesnt even view its data. means size is almot equal to source file but data are lost when i open it.....
and for images .... imgaes are getting loaded an unloaded but with colors changed ....
Also features ( like FFLUSH ) of Oracle 've been used but it never worked
ANY SUGGESTIONS OR aLTERNATE SOLUTION TO LOAD AND UNLOAD PDFs through Oracle ARE REQUESTED.
------------------------------------------------------------------------------------------------------------------------Thanks Justin ...for a quick response ...
well ... i am loading data into BLOB only and using SQL*Loader ...
I've never used dbms_lob.loadFromFile to do the loads ...
i 've opend a file on network and than used dbms_lob.read and
UTL_FILE.PUT_RAW to read and write data into target file.
actually ...my process is working fine with text files but not with PDF and IMAGES ...
and your doubt of ..."Is the data the proper length after reading it in?" ..m not getting wat r you asking ...but ... i think regarding data length ..there is no problem... except ... source PDF length is 90.4 kb ..and Target is 90.8 kb..
thats it...
So Request u to add some more help ......or should i provide some more details ?? -
SQL loader Field in data file exceeds maximum length for CLOB column
Hi all
I'm loading data from text file separated by TAB and i got the error below for some lines.
Event the column is CLOB data type is there a limitation of the size of a CLOB data type.
The error is:
Record 74: Rejected - Error on table _TEMP, column DEST.
Field in data file exceeds maximum length
I'm using SQL Loader and the database is oracle 11g r2 on linux Red hat 5
Here are the line causing the error fronm my data file and my table description for test:
create table TEMP
CODE VARCHAR2(100),
DESC VARCHAR2(500),
RATE FLOAT,
INCREASE VARCHAR2(20),
COUNTRY VARCHAR2(500),
DEST CLOB,
WEEK VARCHAR2(10),
IS_SAT VARCHAR2(50),
IS_SUN VARCHAR2(50)
CONTROL FILE:
LOAD DATA
INTO TABLE TEMP
APPEND
FIELDS TERMINATED BY X'9' TRAILING NULLCOLS
CODE,
DESC,
RATE,
INCREASE,
COUNTRY),
DEST,
WEEK,
IS_SAT,
IS_SUN
Data file:
BHS Mobile Bahamas - Mobile 0.1430 1 "242357, 242359, 242375, 242376, 242395, 242421, 242422, 242423, 242424, 242425, 242426, 242427, 242428, 242429, 242431, 242432, 242433, 242434, 242435, 242436, 242437, 242438, 242439, 242441, 242442, 242443, 242445, 242446, 242447, 242448, 242449, 242451, 242452, 242453, 242454, 242455, 242456, 242457, 242458, 242462, 242463, 242464, 242465, 242466, 242467, 242468, 24247, 242524, 242525, 242533, 242535, 242544, 242551, 242552, 242553, 242554, 242556, 242557, 242558, 242559, 242565, 242577, 242636, 242646, 242727"
BOL Mobile ENTEL Bolivia - Mobile Entel 0.0865 Increase 591 "67, 68, 71, 72, 73, 740, 7410, 7411, 7412, 7413, 7414, 7415, 7420, 7421, 7422, 7423, 7424, 7425, 7430, 7431, 7432, 7433, 7434, 7435, 7436, 7437, 7440, 7441, 7442, 7443, 7444, 7445, 7450, 7451, 7452, 7453, 7454, 7455, 746, 7470, 7471, 7472, 7475, 7476, 7477, 7480, 7481, 7482, 7483, 7484, 7485, 7486, 7490, 7491, 7492, 7493, 7494, 7495, 7496" Thank you.Hi
Thank you for youe help, I found the solution and here what i do in my Control file i added
char(40000) OPTIONALLY ENCLOSED BY '"' .
LOAD DATA
INTO TABLE TEMP
APPEND
FIELDS TERMINATED BY X'9' TRAILING NULLCOLS
CODE,
DESC,
RATE,
INCREASE,
COUNTRY,
DEST
char(40000) OPTIONALLY ENCLOSED BY '"',
WEEK,
IS_SAT,
IS_SUN
Thank you for your help. -
I'm using SQL Loader to import data from a file, but I'm having trouble with numeric types.
I'm trying to import an amount into a table. The amount field in the table is of type NUMBER(11,2).
I suppose that Double is the correct type to use in the ctl-file definition. How do I use this type?
I've tried to use:
AMOUNT DOUBLE TERMINATED BY ";" NULLIF BELOP=BLANKS,
but the I get an error message because I'm using terminated by
I've also tried using:
BELOP DOUBLE NULLIF BELOP=BLANKS,
but then it isn't able to find the delimiter.
I've tried to search the net for examples using Double, but I haven't found anything of current interest yet.
So, my question is:
1. Is Double the correct type to use in this case?
2. How do I use Double?I hardly ever specify a data type in a SQL*Loader control file; I just let Oracle handle it. If it is not the correct datatype for the column Oracle will reject it anyway. The main exception is DATE columns, but even then only because most flat files don't have dates in the default format so the data type specifcation is essentially just a format mask.
Regards,
Steve Rooney -
ORACLE 8I SQL*LOADER DATAFILE의 특정 FIELD DATA를 SKIP하고 LOADING하는 방법
제품 : ORACLE SERVER
작성날짜 : 2002-04-09
ORACLE 8I SQL*LOADER DATAFILE의 특정 FIELD DATA를 SKIP하고 LOADING하는 방법
===========================================================================
아래의 예제와 같이 가변 길이의 filed들이 ',', '|' 와 같은 구분자로
구분이 되고 있는 경우 oracle 8i부터 제공되는 'FILLER'라고 하는 필드
구분자를 사용하여 상태인식자로 표시하여 insert시 skip할 수 있다.
<Example>
TABLE : skiptab
===========================
col1 varchar2(20)
col2 varchar2(20)
col3 varchar2(20)
CONTROLFIEL : skip.ctl
load data
infile skip.dat
into table skiptab
fields terminated by ","
(col1 char,
col2 filler char,
col3 char)
DATAFILE : skip.dat
SMITH, DALLAS, RESEARCH
ALLEN, CHICAGO, SALES
WARD, CHICAGO, SALES
data loading :
$sqlldr scott/tiger control=skip.ctl
결과 :
COL1 COL3
SMITH RESEARCH
ALLEN SALES
WARD SALES -
Is there any difference in Oracle 9i SQL Loader and Oracle 10g SQL Loader
Hi
Can anyone tell me whether is there any difference in Oracle 9i SQL Loader and Oracle 10g SQL Loader?
I am upgrading the 9i db to 10g and wanted to run the 9i SQL Loader control files on upgraded 10g db. So please let me know is there any difference which I need to consider any modifications in the control files..
Thank you in advance
Adianswered
Maybe you are looking for
-
How to provide streaming in J2EE web application?
Hi Friends, I want to build web application which will provide streaming of data. This data may be in raw data, binary, files, audio, video etc. As we are using HTTP protocol for web application, then how to do streaming between server and client usi
-
Blackberry Desktop Manager Language
Why is it that when I download Blackberry Desktop Manager from the Offical Blackberry Website it comes in Arabic and doesnt give you the option to change the Language. Funilly enough I don't read Arabic so I cant exactly search for Setting/Language.
-
Hello! I'm trying to find a way to make my Macbook unable to boot without an external media to initialize the computer. Or at least a way to remove the harddrives from the listed boot options by hitting the option/alt key on bootup. I want to remove
-
How does create a server with multiple Clients ?
Any people can lead me . How does create a server with multiple Clients ? Thanks
-
How do I open mail attachment without having to save first
I have an iMac with Maverick 10.9.2 and mail version 7.2. I am trying to switch over from my laptop to using the iMac exclusively. The laptop mail version is 4.6. In that version, when I receive an email with an attachment, there's a little icon of