Upload records - SQL loader.
Dear All,
We are in the process of getting a big volume data upload project. The data is to be uploaded using SQLLDR in Oracle DB. However the no of records are huge in GB.
Max volume looks to be for the table: 153 million records x 197 (Max size of all data fields in one record)Unicode characters volume. If I assume 1 byte per Unicode char, it comes to 153,639,503 x 197 = 30,266,982,091 bytes; which is about 26.3 GB.
Kindly help to answer below queries –
a. Can you suggest approximately how much time SQLLDR process will take to upload the data of 26.3 GB ?
b. If it is 4 bytes per Unicode char, it will be 105.2GB. Is there a way to handle file sizes of that volume?
c. How can we enhance the data upload process for large volume data (performance and tuning)?
Thanks in advance.
deb
882134 wrote:
We are in the process of getting a big volume data upload project. The data is to be uploaded using SQLLDR in Oracle DB. However the no of records are huge in GB. Relatively small volumes. I am loading over 35 files every 60 seconds. Each CSV file is 21MB in size. 35 files can contain around 4,645,417 rows in total.
a. Can you suggest approximately how much time SQLLDR process will take to upload the data of 26.3 GB ?How long is a piece of string?
b. If it is 4 bytes per Unicode char, it will be 105.2GB. Is there a way to handle file sizes of that volume?Not relevant. It is not the size of the data - it is WHAT needs to be done using that data.
c. How can we enhance the data upload process for large volume data (performance and tuning)?SQL*Loader supports direct and parallel loads. Overheads of the data structure to load into can be reduced by eliminating table constraints, triggers and indexes. Etc.
Oracle is very capable of dealing with large volumes of data is a performant and scalable fashion. How well it performs and scales depends on how you use Oracle.
Similar Messages
-
Need to Skip the fields duaring the data upload by SQL*Loader.
Hi all,
I am not able to get how to skip some fileds in the file while uploading it thru SQL*Loader. Let us have a look the below scenario.
The table has 4 columns. but in the input file came up with 6 fields. Let us assume the four fields came as first in the order. Then, we can populate the data into the 4 columns of the table. At the same time we want to skip the remaining fields. Even those fields/columns are no longer available in the database table.
For example:
create table data_temp(sno number,name varchar2(10 char),sex char(1),loc varchar2(20 char));
Data file
sno,name,sex,loc,organization,contact
1,ram,M,India,HP,998976789
2,Shesha,M,India,IBM,7890808098
Control_file
OPTIONS(SKIP=1)
LOAD DATA
INFILE *
APPEND INTO TABLE data_temp
FIELDS TERMINATED BY ',' optionally enclosed by '"'
TRAILING NULLCOLS
sno "trim(:sno)",
name "SUBSTR(trim(:name),1,20)" ,
sex "SUBSTR(trim(:sex),1,1)",
loc "SUBSTR(trim(:loc),1,20)" ,
Please suggest me how to implement the above scenario in the control file.
Thanks in Advance!!
Regards,
Vissu.....Use FILLER. Control file:
OPTIONS(SKIP=1)
LOAD DATA
INFILE *
APPEND INTO TABLE data_temp
FIELDS TERMINATED BY ',' optionally enclosed by '"'
TRAILING NULLCOLS
sno "trim(:sno)",
name "SUBSTR(trim(:name),1,20)" ,
sex "SUBSTR(trim(:sex),1,1)",
loc "SUBSTR(trim(:loc),1,20)" ,
organization filler,
contact filler
begindata
sno,name,***,loc,organization,contact
1,ram,M,India,HP,998976789
2,Shesha,M,India,IBM,7890808098
{code}
Now:
{code}
SQL> create table data_temp(sno number,name varchar2(10 char),sex char(1),loc varchar2(20 char));
Table created.
SQL> host sqlldr scott/tiger control=c:\temp\vissu.ctl log=c:\temp\vissu.log
SQL> select * from data_temp
2 /
SNO NAME S LOC
1 ram M India
2 Shesha M India
SQL>
{code}
SY. -
Dual Language Data upload using SQL*Loader
I have to load data consisting of English and Traditional Chinese characters into 8.1.5 from external files. These external files are in MS Excel.
Is there any way that I can user SQL*Loader for this ? If yes, how ? If not, is there any other way out.
The NLS in the database is currently AMERICAN.
nullORA-00604 error occurred at recursive SQL level string
Cause: An error occurred while processing a recursive SQL statement (a statement applying to internal dictionary tables).
Action: If the situation described in the next error on the stack can be corrected, do so; otherwise contact Oracle Support Services
This kind of error occurs when data dictionary is
query a lot.
Joel P�rez -
Is there a way to load in spaces using Sql*Loader given that the table it is loading to has a not null column but has a default value. My data file has spaces for a number field instead of zeros. Since my table column is declared as not null, it's failing even if I have a default value of '0' in my column.
You can tackle this in the control file. Here is a snippet from Oracle OnLine books - hopefully this will help out! Cheers.
==========================================
Setting a Column to Null or Zero
If you want all inserted values for a given column to be null, omit the column's specifications entirely. To set a column's values
conditionally to null based on a test of some condition in the logical record, use the NULLIF clause; see NULLIF Keyword. To set a
numeric column to zero instead of NULL, use the DEFAULTIF clause, described next.
DEFAULTIF Clause
Using DEFAULTIF on numeric data sets the column to zero when the specified field condition is true. Using DEFAULTIF on
character (CHAR or DATE) data sets the column to null (compare with Numeric External Datatypes). See also Specifying Field
Conditions for details on the conditional tests.
DEFAULTIF field_condition
A column may have both a NULLIF clause and a DEFAULTIF clause, although this often would be redundant.
Note: The same effects can be achieved with the SQL string and the DECODE function. See Applying SQL Operators to
Fields
NULLIF Keyword
Use the NULLIF keyword after the datatype and optional delimiter specification, followed by a condition. The condition has the same
format as that specified for a WHEN clause. The column's value is set to null if the condition is true. Otherwise, the value remains
unchanged.
NULLIF field_condition
The NULLIF clause may refer to the column that contains it, as in the following example:
COLUMN1 POSITION(11:17) CHAR NULLIF (COLUMN1 = "unknown")
This specification may be useful if you want certain data values to be replaced by nulls. The value for a column is first determined from
the datafile. It is then set to null just before the insert takes place. Case 6: Loading Using the Direct Path Load Method provides
examples of the NULLIF clause.
Note: The same effect can be achieved with the SQL string and the NVL function. See Applying SQL Operators to Fields.
Null Columns at the End of a Record
When the control file specifies more fields for a record than are present in the record, SQL*Loader must determine whether the
remaining (specified) columns should be considered null or whether an error should be generated. The TRAILING NULLCOLS
clause, described in TRAILING NULLCOLS, explains how SQL*Loader proceeds in this case.
========================================== -
Hi,
I have similar issue where i have requirement to load 270 million record per day into single table(having No constraints & Indexes), where every CTL file contain 37000 records.
I have machine having 16 CPU and 2 thread per CPU is set.I am using the PARALLEL=TRUE, MULTITHREADING=TRUE,DIRECT=TRUE option in sql loader.
E.g:- OPTIONS( ERRORS=100000, SILENT=all, MULTITHREADING=TRUE, DIRECT=TRUE, PARALLEL=TRUE, SKIP_INDEX_MAINTENANCE=TRUE,streamsize=1048576, readsize=1048576, columnarrayrows=8000 )
Also enable the PARALLEL degree and set the value to 32.
When i am running sqlloader 4 session with above configuration it was taking total 4-5 seconds to load 4 CTL files having 37000 records per file. Well for initial 50 million records sql loader behave normally to load the CTL files in 4-5 seconds, but after 50 million records in table, time taken to process the 4 CTL files was gradually increasing to 40 to 70 seconds and it was still increasing as the number of records more and more in table.
I don't know why sqlloader behave like this after 50 million record in table.
Below is the parallel parameter set on the machine
SQL> show parameter parallel;
NAME TYPE VALUE
fast_start_parallel_rollback string LOW
parallel_adaptive_multi_user boolean TRUE
parallel_automatic_tuning boolean FALSE
parallel_degree_limit string CPU
parallel_degree_policy string MANUAL
parallel_execution_message_size integer 16384
parallel_force_local boolean FALSE
parallel_instance_group string
parallel_io_cap_enabled boolean FALSE
parallel_max_servers integer 80
parallel_min_percent integer 0
NAME TYPE VALUE
parallel_min_servers integer 0
parallel_min_time_threshold string AUTO
parallel_server boolean FALSE
parallel_server_instances integer 1
parallel_servers_target integer 32
parallel_threads_per_cpu integer 2
recovery_parallelism integer 0
Kindly repl on the above queryI've got a similar problem.
I'm running Informatica which runs SQL*Loader via OCI.
With direct & parallel my Indexes give me ORA-26002 - quite understandable.
As its all wrapped up, I cannot say SKIP_INDEX_MAINTENANCE directly.
1) Is there a way to give SQL*Loader some Default-Parameters? I read something about a File shrept.lst but I cannot find a reference in the Oracle documentation.
Something useful down this road?
2) when I omit parallel, what am I losing? I load one file into one partition of one table. I read the SQL*Loader Documentation but didn't get the message of the 'Intrasegment Parallel Loading' Paragraph. Is this using parallel? Or is it just enabling to use parallel (if there will be a second SQL*Loader)? If its all about 'enabling' I can easily omit this - I know there won't be a second SQL*Loader for this partition.
regards,
Kathrin -
Need help, Trouble in uploading records using sql loader in Forms 6i
Hi,
I am trying to develop a screen for uploading records to a table by using a ctl file, batch file and sql loader.
Env: Forms 6i, Oracle 8
Table to be updated is: shy_upload_table
My TSN entry looks similar to this,
TEST_AXA.CNB.COM =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = 11.23.11.123)(PORT = 1234))
(CONNECT_DATA =
(SID = axdabc)
My intention is whenever i press the upload_button, I should truncate the table and upload it with the contents of the file.
In the when-button-pressed event of the upload_button I have the following code. always I am able to truncate the table but am not able to upload it with the contents of the file. Can any of you help me fix this problem ?
declare
var_control varchar2(256);
VAR_DATA VARCHAR2(256);
VAR_OUTPUT VARCHAR2(500);
var_filename varchar2(256);
str varchar2(50);
cnt number;
begin
FORMS_DDL('TRUNCATE TABLE shy_upload_table ');
select count(*) into cnt from shy_upload_table;
message('count '||cnt);
MESSAGE('');
If NOT form_success Then
MESSAGE('Upload Failed');
MESSAGE('Upload Failed');
else
set_item_property('DISPLAY_PB',enabled,property_true);
--when ever i run, i am able to see the display_pb enabled. it means form_success is true.
end if;
var_filename := :txt_filename;
--I have tried with each of the below option,
--sqlldr userid/[email protected] control=F:\ERP\file_upload.ctl
--sqlldr userid/password@axdabc control=F:\ERP\file_upload.ctl
--sqlldr userid/password@TEST_AXA.CNB.COM control=F:\ERP\file_upload.ctl
VAR_DATA :='data=' || var_filename ;
VAR_OUTPUT := var_control|| ' ' ||VAR_DATA;
host('F:\a.bat');
end;
batch file contents...
# I have tried with each of the below options
sqlldr userid/[email protected] control=F:\ERP\file_upload.ctl data=F:\ERP\sample.txt log=F:\ERP\x.log bad=F:\ERP\x.bad
#sqlldr userid/password@axdabc control=F:\ERP\file_upload.ctl data=F:\ERP\sample.txt log=F:\ERP\x.log bad=F:\ERP\x.bad
#sqlldr userid/password@TEST_AXA.CNB.COM control=F:\ERP\file_upload.ctl data=F:\ERP\sample.txt log=F:\ERP\x.log bad=F:\ERP\x.bad
pause
Thanks
vishHi Francois,
Thanks for responding, I am not very sure of what you want me to try out.
When I double click the batch file containing the below, the record gets inserted in the table. Only when using my form and trying to upload, it fails to insert the record.
batch file contents...
#sqlldr userid/password@TEST_AXA.CNB.COM control=F:\ERP\file_upload.ctl data=F:\ERP\sample.txt log=F:\ERP\x.log bad=F:\ERP\x.bad
pause
Thanks
Vish -
SQL*Loader- Records Rejected - Error on table ORA-01722: invalid number
Getting the following errors :
Please tell me where I am going wrong?
Attached is the log file and snippets of datafile along with the control file !!
Also please direct me how can i upload 4900 records at one go?
SQL*Loader: Release 11.1.0.7.0 - Production on Fri Oct 14 03:06:06 2011
Copyright (c) 1982, 2007, Oracle. All rights reserved.
Control File: sample.ctl
Data File: Cities.csv
Bad File: Cities.bad
Discard File: none specified
(Allow all discards)
Number to load: ALL
Number to skip: 0
Errors allowed: 50
Bind array: 64 rows, maximum of 256000 bytes
Continuation: none specified
Path used: Conventional
Table CITY, loaded from every logical record.
Insert option in effect for this table: INSERT
Column Name Position Len Term Encl Datatype
ID FIRST * , CHARACTER
NAME NEXT 35 , ' CHARACTER
COUNTRYCODE NEXT 3 , ' CHARACTER
POPULATION NEXT * WHT CHARACTER
Record 1: Rejected - Error on table CITY, column POPULATION.
ORA-01722: invalid number
Record 2: Rejected - Error on table CITY, column POPULATION.
ORA-01722: invalid number
Record 3: Rejected - Error on table CITY, column POPULATION.
ORA-01722: invalid number
Record 4: Rejected - Error on table CITY, column POPULATION.
ORA-01722: invalid number
Record 5: Rejected - Error on table CITY, column POPULATION.
ORA-01722: invalid number
Record 6: Rejected - Error on table CITY, column POPULATION.
ORA-01722: invalid number
Record 7: Rejected - Error on table CITY, column POPULATION.
ORA-01722: invalid number
Record 8: Rejected - Error on table CITY, column POPULATION.
ORA-01722: invalid number
Record 9: Rejected - Error on table CITY, column POPULATION.
ORA-01722: invalid number
Record 10: Rejected - Error on table CITY, column POPULATION.
ORA-01722: invalid number
Record 11: Rejected - Error on table CITY, column POPULATION.
ORA-01722: invalid number
Record 12: Rejected - Error on table CITY, column POPULATION.
ORA-01722: invalid number
Record 13: Rejected - Error on table CITY, column POPULATION.
ORA-01722: invalid number
Record 14: Rejected - Error on table CITY, column POPULATION.
ORA-01722: invalid number
Record 15: Rejected - Error on table CITY, column POPULATION.
ORA-01722: invalid number
Record 16: Rejected - Error on table CITY, column POPULATION.
ORA-01722: invalid number
Record 17: Rejected - Error on table CITY, column POPULATION.
ORA-01722: invalid number
Record 18: Rejected - Error on table CITY, column POPULATION.
ORA-01722: invalid number
Record 19: Rejected - Error on table CITY, column POPULATION.
ORA-01722: invalid number
Record 20: Rejected - Error on table CITY, column POPULATION.
ORA-01722: invalid number
Record 21: Rejected - Error on table CITY, column POPULATION.
ORA-01722: invalid number
Record 22: Rejected - Error on table CITY, column POPULATION.
ORA-01722: invalid number
Record 23: Rejected - Error on table CITY, column POPULATION.
ORA-01722: invalid number
Record 24: Rejected - Error on table CITY, column POPULATION.
ORA-01722: invalid number
Record 25: Rejected - Error on table CITY, column POPULATION.
ORA-01722: invalid number
Record 26: Rejected - Error on table CITY, column POPULATION.
ORA-01722: invalid number
Record 27: Rejected - Error on table CITY, column POPULATION.
ORA-01722: invalid number
Record 28: Rejected - Error on table CITY, column POPULATION.
ORA-01722: invalid number
Record 29: Rejected - Error on table CITY, column POPULATION.
ORA-01722: invalid number
Record 30: Rejected - Error on table CITY, column POPULATION.
ORA-01722: invalid number
Record 31: Rejected - Error on table CITY, column POPULATION.
ORA-01722: invalid number
Record 32: Rejected - Error on table CITY, column POPULATION.
ORA-01722: invalid number
Record 33: Rejected - Error on table CITY, column POPULATION.
ORA-01722: invalid number
Record 34: Rejected - Error on table CITY, column POPULATION.
ORA-01722: invalid number
Record 35: Rejected - Error on table CITY, column POPULATION.
ORA-01722: invalid number
Record 36: Rejected - Error on table CITY, column POPULATION.
ORA-01722: invalid number
Record 37: Rejected - Error on table CITY, column POPULATION.
ORA-01722: invalid number
Record 38: Rejected - Error on table CITY, column POPULATION.
ORA-01722: invalid number
Record 39: Rejected - Error on table CITY, column POPULATION.
ORA-01722: invalid number
Record 40: Rejected - Error on table CITY, column POPULATION.
ORA-01722: invalid number
Record 41: Rejected - Error on table CITY, column POPULATION.
ORA-01722: invalid number
Record 42: Rejected - Error on table CITY, column POPULATION.
ORA-01722: invalid number
Record 43: Rejected - Error on table CITY, column POPULATION.
ORA-01722: invalid number
Record 44: Rejected - Error on table CITY, column POPULATION.
ORA-01722: invalid number
Record 45: Rejected - Error on table CITY, column POPULATION.
ORA-01722: invalid number
Record 46: Rejected - Error on table CITY, column POPULATION.
ORA-01722: invalid number
Record 47: Rejected - Error on table CITY, column POPULATION.
ORA-01722: invalid number
Record 48: Rejected - Error on table CITY, column POPULATION.
ORA-01722: invalid number
Record 49: Rejected - Error on table CITY, column POPULATION.
ORA-01722: invalid number
Record 50: Rejected - Error on table CITY, column POPULATION.
ORA-01722: invalid number
Record 51: Rejected - Error on table CITY, column POPULATION.
ORA-01722: invalid number
MAXIMUM ERROR COUNT EXCEEDED - Above statistics reflect partial run.
Table CITY:
0 Rows successfully loaded.
51 Rows not loaded due to data errors.
0 Rows not loaded because all WHEN clauses were failed.
0 Rows not loaded because all fields were null.
Space allocated for bind array: 35840 bytes(64 rows)
Read buffer bytes: 1048576
Total logical records skipped: 0
Total logical records read: 64
Total logical records rejected: 51
Total logical records discarded: 0
Run began on Fri Oct 14 03:06:06 2011
Run ended on Fri Oct 14 03:06:12 2011
Elapsed time was: 00:00:06.18
CPU time was: 00:00:00.03
my control file (sample.ctl):
load data infile 'Cities.csv'
into table city
fields terminated by ','
(id integer external,
name char(35) enclosed by "'",
countrycode char(3) enclosed by "'",
population integer external terminated by '\n'
my datafile (Cities.csv) (it contains 4900 records, but I am showing here just 4 records for ease)
3830,'Virginia Beach','USA',425257
3831,'Atlanta','USA',416474
3832,'Sacramento','USA',407018
3833,'Oakland','USA',399484
Thanks in advance!!Look that when I change a little bit your datafile as follows
1,'Kabul','AFG',1780000
2,'Qandahar','AFG','237500'
3,'Herat','AFG','186800' I got the same error (2 last rows rejected for the same error invalid number)
mhouri > select * from cities;
ID NAME COU POPULATION
1 Kabul AFG 1780000
SQL*Loader: Release 10.2.0.3.0 - Production on Fri Oct 14 10:38:06 2011
Copyright (c) 1982, 2005, Oracle. All rights reserved.
Control File: cities.ctl
Data File: cities.dat
Bad File: cities.bad
Discard File: none specified
(Allow all discards)
Number to load: ALL
Number to skip: 0
Errors allowed: 50
Bind array: 64 rows, maximum of 256000 bytes
Continuation: none specified
Path used: Conventional
Table CITIES, loaded from every logical record.
Insert option in effect for this table: INSERT
Column Name Position Len Term Encl Datatype
ID FIRST * , CHARACTER
NAME NEXT 35 , ' CHARACTER
COUNTRYCODE NEXT 3 , ' CHARACTER
POPULATION NEXT * WHT CHARACTER
Record 4: Rejected - Error on table CITIES, column ID.
Column not found before end of logical record (use TRAILING NULLCOLS)
Record 2: Rejected - Error on table CITIES, column POPULATION.
ORA-01722: invalid number
Record 3: Rejected - Error on table CITIES, column POPULATION.
ORA-01722: invalid number
Table CITIES:
1 Row successfully loaded.
3 Rows not loaded due to data errors.
0 Rows not loaded because all WHEN clauses were failed.
0 Rows not loaded because all fields were null.
Space allocated for bind array: 35840 bytes(64 rows)
Read buffer bytes: 1048576
Total logical records skipped: 0
Total logical records read: 4
Total logical records rejected: 3
Total logical records discarded: 0
Run began on Fri Oct 14 10:38:06 2011
Run ended on Fri Oct 14 10:38:06 2011
Elapsed time was: 00:00:00.23
CPU time was: 00:00:00.09Population value within the data file should be a number
Best regards
Mohamed Houri -
SQL Loader Oracle 10g problem in upload date with time data -- Very urgent.
Hi
I am trying to upload data using SQL loader. There are three columns in the table
defined as DATE. When I tried upload a data like this '2007-02-15 15:10:20', it is not loading time part. The date stored as 02/15/2008' only. There is not time on that. I tried with many different format nothing work. Can please help me ?
I have also tried with to_date --> to_timestamp it did not work.
The application is going to be in production, I cannot change DATE to TIME STAMP. This is very urgent.
LASTWRITTEN "decode(:LASTWRITTEN,'null',Null, to_date(:LASTWRITTEN,'YYYY-MM-DD HH24:Mi:SS'))",
CREATEDON "decode(:CREATEDON,'null',Null, to_date(:CREATEDON,'YYYY-MM-DD HH24:Mi:SS'))",
LASTUPDATEDON(21) "decode(:LASTUPDATEDON,'null',Null, to_date(:LASTUPDATEDON(21),'DD/MM/YYYY HH24:MI:SS'))"Your problem is most likely in decode - the return type in your expression will be character based on first search value ('null'), so it will be implicitly converted to character and then again implicitly converted to date by loading into date column. At some of this conversions you probably are loosing your time part. You can try instead use cast:
SQL> desc t
Name Null? Type
LASTWRITTEN DATE
CREATEDON DATE
LASTUPDATEDON DATE
SQL> select * from t;
no rows selected
SQL> !cat t.ctl
LOAD DATA
INFILE *
INTO TABLE T
TRUNCATE
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' TRAILING NULLCOLS
LASTWRITTEN
"decode(:LASTWRITTEN,'null',cast(Null as date),
to_date(:LASTWRITTEN,'YYYY-MM-DD HH24:MI:SS'))",
CREATEDON
"decode(:CREATEDON,'null',cast(Null as date),
to_date(:CREATEDON,'YYYY-MM-DD HH24:MI:SS'))",
LASTUPDATEDON
"decode(:LASTUPDATEDON,'null',cast(Null as date),
to_date(:LASTUPDATEDON,'DD/MM/YYYY HH24:MI:SS'))"
BEGINDATA
2007-02-15 15:10:20,null,null
null,2007-02-15 15:10:20,null
null,null,15/02/2007 15:10:20
SQL> !sqlldr userid=scott/tiger control=t.ctl log=t.log
SQL*Loader: Release 10.2.0.3.0 - Production on Fri Feb 29 00:20:07 2008
Copyright (c) 1982, 2005, Oracle. All rights reserved.
Commit point reached - logical record count 3
SQL> select * from t;
LASTWRITTEN CREATEDON LASTUPDATEDON
15.02.2007 15:10:20
15.02.2007 15:10:20
15.02.2007 15:10:20Best regards
Maxim -
How do I skip footer records in Data file through control file of sql*loade
hi,
I am using sql*loader to load data from data file and i have written control file for it. How do i skip last '5' records of data file or the footer records to be skiped to read.
For first '5' records to be skiped we can use "skip" to achieve it but how do i acheive for last '5' records.
2)
Can I mention two data files in one control file if so what is the syntax(like we give INFILE Where we mention the path of data file can i mention two data file in same control file)
3)
If i have datafile with variable length (ie 1st record with 200 charcter, 2nd with 150 character and 3rd with 180 character) then how do i load data into table, i mean what will be the syntax for it in control file.
4)if i want to insert sysdate into table through control file how do i do it.
5) If i have variable length records in data file and i have first name then white space between then and then last name, how do i insert this value which includes first name and last name into single column of the table.( i mean how do you handle the white space in between first name and last name in data file)
Thanks in advance
ramYou should read the documentation about SQL*Loader.
-
SQL Loader-704: Internal error: Maximum record length must be = [10000000]
Hi,
running SQL*Loader (Release 8.1.7.2.1) causes an error "SQL*Loader-704: Internal error: Maximum record length must be <= [10000000]". This error occurs when SQLLoader is trying to load several thousand records into a database table. Each record is less than 250 bytes in length.
Any idea what could cause the problem?
Thanks in advance!
Ingo
And here's an extract from the log file generated by SQLLoader :
Number to load: ALL
Number to skip: 0
Errors allowed: 50
Bind array: 1360 rows, maximum of 10485760 bytes
Continuation: none specified
Path used: Conventional
Table "SYSTEM"."BASICPROFILE$1", loaded from every logical record.
Insert option in effect for this table: APPEND
TRAILING NULLCOLS option in effect
Column Name Position Len Term Encl Datatype
UUID FIRST * O(X07) CHARACTER
DOMAINID NEXT * O(X07) CHARACTER
LASTMODIFIED NEXT * O(X07) DATE DD/MM/YYYY HH24:MI:SS
ANNIVERSARY NEXT * O(X07) CHARACTER
BIRTHDAY NEXT * O(X07) CHARACTER
COMPANYNAME NEXT * O(X07) CHARACTER
DESCRIPTION NEXT * O(X07) CHARACTER
FIRSTNAME NEXT * O(X07) CHARACTER
COMPANYNAMETRANSCRIPTION NEXT * O(X07) CHARACTER
FIRSTNAMETRANSCRIPTION NEXT * O(X07) CHARACTER
GENDER NEXT * O(X07) CHARACTER
HOBBIES NEXT * O(X07) CHARACTER
HONORIFIC NEXT * O(X07) CHARACTER
JOBTITLE NEXT * O(X07) CHARACTER
KEYWORDS NEXT * O(X07) CHARACTER
LASTNAME NEXT * O(X07) CHARACTER
LASTNAMETRANSCRIPTION NEXT * O(X07) CHARACTER
NICKNAME NEXT * O(X07) CHARACTER
PREFERREDLOCALE NEXT * O(X07) CHARACTER
PREFERREDCURRENCY NEXT * O(X07) CHARACTER
PROFESSION NEXT * O(X07) CHARACTER
SECONDLASTNAME NEXT * O(X07) CHARACTER
SECONDNAME NEXT * O(X07) CHARACTER
SUFFIX NEXT * O(X07) CHARACTER
TITLE NEXT * O(X07) CHARACTER
CONFIRMATION NEXT * O(X07) CHARACTER
DEFAULTADDRESSID NEXT * O(X07) CHARACTER
BUSINESSPARTNERNO NEXT * O(X07) CHARACTER
TYPECODE NEXT * O(X07) CHARACTER
OCA NEXT * O(X07) CHARACTER
SQL*Loader-704: Internal error: Maximum record length must be <= [10000000]As a second guess, the terminator changes or goes missing at some point in the data file. If you are running on *NIX, try wc -l data_file_name. This will give a count of the number of lines (delimited by CHR(10) ) that are in the file. If this is not close to the number you expected, then that is your problem.
You could also try gradually working through the data file loading 100 records, then 200, then 300 etc. to see where it starts to fail.
HTH
John -
SQL Loader problem while loading records from txt file to database table.
I am getting following error while loading records from flat txt file into database table with the help of 'sqlldr' command. I have executed catldr.sql from RDBMS folder but it is still showing same error. I am setting DIRECT = TRUE while issuing sqlldr command. If I try with DIRECT = FALSE then it works fine. Database is Oracle 8i.
SQL*Loader-951: Error calling once/load initialization
ORA-24329: invalid character set identifier
F1 Please.Hello,
Direct path load, can only be used with SQL*Loader and Database have the same version.
Care to tell the database version and sql*loader version you are using.
-Sri -
SQL*Loader and unzipping with uploaded file
hi, everybody
i was trying to integrate some functionalities, today separated. i made a simple Apex page that uploads a file, and now i need to process that file.
is there any way to process that uploaded file and load it onto a table, just like i'd do with Sql*Loader?
in the same way, can i unzip an uploaded zip file, just like Oracle Portal 3.0.9 used to do in its contect areas with zip itens?
thanksHi, Ivo
about ult_compress, ok. i'll check it out.
talking about uploaded files, the only thing i've found were:
a) download an uploaded file
b) use an uploaded image to integrate in a web application.
unfortunately, i didn't find anything that could help me process an uploaded text file the way i do with SQL*Loader.
i'll keep on trying
thanx -
Load multiple records in 1 table from 1 line in text file w/sql loader
hi guys,
quick question, perhaps someone can help. searched around and didn't see this question asked before, and can't find any answer in SQL Loader faqs or docs.
i know i can extract multiple logical records from a single physical record in the input file as two logical records, and then use two into tables clauses to load the data into the table. see oracle 9i sql loader control file reference chapter 5 to see what i am talking about.
but my question follows:
cust_id amount1_val amount1_qual amount2_val amount2_qual amount3_val amount3_qual
123 1500.35 TA 230.34 VZ 3045.50 TW
basically i want to use one sql loader statement to load these 3 records into 1 table. the issue for me is that i need to re-use the cust_id for all 3 records as the key, along with the qualifier code. the example in the Oracle docs only works for data where the logical records are completely separate -- no shared column values.
i'm sure this is possible, perhaps using some :cust_id type parameter for the 2nd and 3rd records, or something, but i just don't have enough knowledge/experience with sql loader to know what to do. appreciate any help.
wayneHi wayne,
I found an example on what exactly you were looking for from an SQL*Loader documentation. Please see if it of some help to you
EXAMPLE
The control file is ULCASE5.CTL.
1234 BAKER 10 9999 101
1234 JOKER 10 9999 777
2664 YOUNG 20 2893 425
5321 OTOOLE 10 9999 321
2134 FARMER 20 4555 236
2414 LITTLE 20 5634 236
6542 LEE 10 4532 102
2849 EDDS xx 4555
4532 PERKINS 10 9999 40
1244 HUNT 11 3452 665
123 DOOLITTLE 12 9940
1453 MACDONALD 25 5532
In the above datafile
Column1 - Empno
Column2 - ENAME
Column3 - Depno.
Column4 - MGR
Column5 - Proj no.
-- Loads EMP records from first 23 characters
-- Creates and loads PROJ records for each PROJNO listed
-- for each employee
LOAD DATA
INFILE 'ulcase5.dat'
BADFILE 'ulcase5.bad'
DISCARDFILE 'ulcase5.dsc'
1) REPLACE
2) INTO TABLE emp
(empno POSITION(1:4) INTEGER EXTERNAL,
ename POSITION(6:15) CHAR,
deptno POSITION(17:18) CHAR,
mgr POSITION(20:23) INTEGER EXTERNAL)
2) INTO TABLE proj
(empno POSITION(1:4) INTEGER EXTERNAL,
3) projno POSITION(25:27) INTEGER EXTERNAL) -- 1st proj
Notes:
REPLACE specifies that if there is data in the tables to be loaded (EMP and PROJ), SQL*loader should delete the data before loading new rows.
Multiple INTO clauses load two tables, EMP and PROJ. The same set of records is processed three times, using different combinations of columns each time to load table PROJ.
Regards,
Murali Mohan -
SQL Loader loads duplicate records even when there is PK defined
Hi,
I have created table with pk on one of the column.Loaded the table using sql loader.the flat file has duplicate record.It loaded all the records without any error but now the index became unusable.
The requirement is to fail the process if there are any duplicate.Please help me in understaing why this is happening.
Below is the ctl file
OPTIONS(DIRECT=TRUE, ERRORS=0)
UNRECOVERABLE
load data
infile 'test.txt'
into table abcdedfg
replace
fields terminated by ',' optionally enclosed by '"'
col1 ,
col2
i defined pk on col1Check out..
http://download-east.oracle.com/docs/cd/B19306_01/server.102/b14215/ldr_modes.htm#sthref1457
It states...
During a direct path load, some integrity constraints are automatically disabled. Others are not. For a description of the constraints, see the information about maintaining data integrity in the Oracle Database Application Developer's Guide - Fundamentals.
Enabled Constraints
The constraints that remain in force are:
NOT NULL
UNIQUE
PRIMARY KEY (unique-constraints on not-null columns)Since OP has the primary key in place before starting the DIRECT path load, this is contradictory to what the documentation says or probably a bug? -
Sql loader - Data loading issue with no fixed record length
Hi All,
I am trying to load the following data through sql loader. However the records # 1, 3 & 4 are only loading succesfully into the table and rest of the records showing as BAD. What is missing in my syntax?
.ctl file:
LOAD DATA
INFILE 'C:\data.txt'
BADFILE 'c:\data.BAD'
DISCARDFILE 'c:\data.DSC' DISCARDMAX 50000
INTO TABLE icap_gcims
TRAILING NULLCOLS
CUST_NBR_MAIN POSITION(1:9) CHAR NULLIF (CUST_NBR_MAIN=BLANKS),
CONTACT_TYPE POSITION(10:11) CHAR NULLIF (CONTACT_TYPE=BLANKS),
INQUIRY_TYPE POSITION(12:13) CHAR NULLIF (INQUIRY_TYPE=BLANKS),
INQUIRY_MODEL POSITION(14:20) CHAR NULLIF (INQUIRY_MODEL=BLANKS),
INQUIRY_COMMENTS POSITION(21:60) CHAR NULLIF (INQUIRY_COMMENTS=BLANKS),
OTHER_COLOUR POSITION(61:75) CHAR NULLIF (OTHER_COLOUR=BLANKS),
OTHER_MAKE POSITION(76:89) CHAR NULLIF (OTHER_MAKE=BLANKS),
OTHER_MODEL_DESCRIPTION POSITION(90:109) CHAR NULLIF (OTHER_MODEL_DESCRIPTION=BLANKS),
OTHER_MODEL_YEAR POSITION(110:111) CHAR NULLIF (OTHER_MODEL_YEAR=BLANKS)
data.txt file:
000000831KHAN
000000900UHFA WANTS NEW WARRANTY ID 000001017OHAL
000001110KHAP
000001812NHDE231291COST OF SERVICE INSPECTIONS TOO HIGH MAXIMA 92 MK
000002015TPFA910115CUST UPSET WITH AIRPORT DLR. $200 FOR PLUGS,OIL,FILTER CHANGE. FW
Thanks,Hi,
Better if you have given the table structure, I check your script it was fine
11:39:01 pavan_Real>create table test1(
11:39:02 2 CUST_NBR_MAIN varchar2(50),
11:39:02 3 CONTACT_TYPE varchar2(50),
11:39:02 4 INQUIRY_TYPE varchar2(50),
11:39:02 5 INQUIRY_MODEL varchar2(50),
11:39:02 6 INQUIRY_COMMENTS varchar2(50),
11:39:02 7 OTHER_COLOUR varchar2(50),
11:39:02 8 OTHER_MAKE varchar2(50),
11:39:02 9 OTHER_MODEL_DESCRIPTION varchar2(50),
11:39:02 10 OTHER_MODEL_YEAR varchar2(50)
11:39:02 11 );
Table created.
11:39:13 pavan_Real>select * from test1;
no rows selected
C:\Documents and Settings\ivy3905>sqlldr ara/ara@pavan_real
control = C:\control.ctl
SQL*Loader: Release 9.2.0.1.0 - Production on Sat Sep 12 11:41:27 2009
Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved.
Commit point reached - logical record count 5
11:42:20 pavan_Real>select count(*) from test1;
COUNT(*)
5 control.ctl
LOAD DATA
INFILE 'C:\data.txt'
BADFILE 'c:\data.BAD'
DISCARDFILE 'c:\data.DSC' DISCARDMAX 50000
INTO TABLE test1
TRAILING NULLCOLS
CUST_NBR_MAIN POSITION(1:9) CHAR NULLIF (CUST_NBR_MAIN=BLANKS),
CONTACT_TYPE POSITION(10:11) CHAR NULLIF (CONTACT_TYPE=BLANKS),
INQUIRY_TYPE POSITION(12:13) CHAR NULLIF (INQUIRY_TYPE=BLANKS),
INQUIRY_MODEL POSITION(14:20) CHAR NULLIF (INQUIRY_MODEL=BLANKS),
INQUIRY_COMMENTS POSITION(21:60) CHAR NULLIF (INQUIRY_COMMENTS=BLANKS),
OTHER_COLOUR POSITION(61:75) CHAR NULLIF (OTHER_COLOUR=BLANKS),
OTHER_MAKE POSITION(76:89) CHAR NULLIF (OTHER_MAKE=BLANKS),
OTHER_MODEL_DESCRIPTION POSITION(90:109) CHAR NULLIF (OTHER_MODEL_DESCRIPTION=BLANKS),
OTHER_MODEL_YEAR POSITION(110:111) CHAR NULLIF (OTHER_MODEL_YEAR=BLANKS)
data.txt
000000831KHAN
000000900UHFA WANTS NEW WARRANTY ID 000001017OHAL
000001110KHAP
000001812NHDE231291COST OF SERVICE INSPECTIONS TOO HIGH MAXIMA 92 MK
000002015TPFA910115CUST UPSET WITH AIRPORT DLR. $200 FOR PLUGS,OIL,FILTER CHANGE. FW
CUST_NBR_MAIN CONTACT_TYPE INQUIRY_TYPE INQUIRY_MODEL INQUIRY_COMMENTS OTHER_COLOUR OTHER_MAKE OTHER_MODEL_DESCRIPTION OTHER_MODEL_YEAR
000000831 KH AN NULL NULL NULL NULL NULL NULL
000000900 UH FA WANTS NEW WARRANTY ID 000001017OHAL NULL NULL NULL NULL
000001110 KH AP NULL NULL NULL NULL NULL NULL
000001812 NH DE 231291C OST OF SERVICE INSPECTIONS TOO HIGH MAXI MA 92 MK NULL NULL NULL
000002015 TP FA 910115C UST UPSET WITH AIRPORT DLR. $200 FOR PLU GS,OIL,FILTER C HANGE. FW NULL NULL- Pavan Kumar N
Edited by: Pavan Kumar on Sep 12, 2009 11:46 AM
Maybe you are looking for
-
ITunes 9.2 update on windows 7 x64 failed What I did to fix it
Seems like everybody else had the same problems. In my case the upgrade originally failed with a 2324 error. It suggested trying again and if it continued that I should select tools download only. That is what I did. It will save the files to: C:\Use
-
How to obtain the Query Response Time of a query?
Given the Average Length of Row of tables and the number of rows in each table, is there a way we get the query response time of a query involving those tables. Query includes joins as well. For example, suppose there 3 tables t1, t2, t3. I wish to o
-
When I save out of Mac Mail often there is a very long delay before the save as window pops up. Same happens in Photoshop. Any bright ideas how to speed things up?
-
Camera Function after installing 2.3.3
Hi Installed 2.3.3. yesterday and have lost camera functionality. Image can be seen to be taken, but no image can be captured. Touch screen and normal all fail to capture an image. I have reset factory preferenes on camera but nothing. I also keen ge
-
As above (first time I've tried to use apple support. God it's a bloody palaver)