Bulk Load in SQLLDR

I have a huge dat file to load using sqlldr. I am told that there is a Bulk Load option that can be used. If true, how do I use it (syntax)?
Are there any other ways of loading large volume data on dat files into an Oracle DB?
A quick reply is appreciated

http://download-west.oracle.com/docs/cd/B10501_01/server.920/a96652/ch09.htm#1007453

Similar Messages

How to use RECNUM special field in a file bulk load interface (sqlldr)

Hi,
I'm trying to load an ordered set of full text lines from a flat file using Sql Loader 11.2 with ODI 11.1.1 bulk LKM (LKM File to Oracle - SQLLDR).
I have to keep track of each line number in a separate target table column NUM_SEQ and feed it with sqlldr RECNUM special field.
I haven't found any other way to do that but to tweak manually the generated sqlldr .ctl control file (bad but it works) :
NUM_SEQ RECNUM,
FULL_LINE CHAR(4000)
I've tried to map "RECNUM" as an expression in the map tab of the loading interface but the column itself gets discarded at .ctl generation.
I haven't found any mention of RECNUM in the whole ODI documentation, neither on this forum nor the Web.
Using an internal Oracle sequence in the subsequent steps of the ETL breaks the garantee of ordered lines.
Any hint ?

You will have to enhance the KM so that this clause gets added to the CTL file each time.
Add an Option to the KM in which you can specify name of the column that you want to act as line_number.
And then in the KM, change the "Generate CTL File" step and add
*<%=odiRef.getOption("RECNUM") %> RECNUM*
after the call to the <%=odiRef.getColList(" ") %> API
So, this will add the RECNUM column to the list of the columns generated.

Sqlldr bulk loading

i'm having a strange problem bulk loading a about 15k+ rows into an existing spatial table.
i use a perl script to 'watch' a directory intto which the usgs pushes a file. The first line in the file gets parsed and entered into a table (no problem here). For the rest of the file i generate a sqlldr control file that appends the data into my table. After the insert i do a transform of the lat, long columns and make an sdo_point like so:
$upstmt = $dbh->prepare("update shake_xyz set shape = dd832utm(lon,lat) where id = ?" );
$upstmt->bind_param(1,$id);
$upstmt->execute() or die "cant update! $DBI::errstr\n";
if a spatial index does not exist on the table, theres no problem.
If one does and i dont use a direct path load, theres no problem (except that this takes waaaaay longer than i'd like).
When i try to do a direct path load with an spatial index, i get the following rather distressing message
SQL*Loader-926: OCI error while uldlfca:OCIDirPathColArrayLoadStream for table SHAKE_XYZ
SQL*Loader-2026: the load was aborted because SQL Loader cannot continue.
SQL*Loader-925: Error while uldlgs: OCIStmtExecute (ptc_hp)
ORA-03114: not connected to ORACLE
SQL*Loader-925: Error while uldlgs: OCIStmtFetch (ptc_hp)
ORA-24338: statement handle not executed
SQL*Loader-925: Error while uldlgs: OCIStmtExecute (pts_hp)
ORA-03114: not connected to ORACLE
SQL*Loader-925: Error while uldlgs: OCIStmtFetch (pts_hp)
ORA-24338: statement handle not executed
SQL*Loader-925: Error while uldlgs: OCIStmtFetch (sbpt_hp)
ORA-24338: statement handle not executed
SQL*Loader-925: Error while uldlgs: OCIStmtExecute (sbpt_hp)
ORA-24338: statement handle not executed
SQL*Loader-925: Error while uldlgs: OCIStmtFetch (pts_hp)
ORA-24338: statement handle not executed
SQL*Loader-925: Error while uldlgs: OCIStmtFetch (sbpt_hp)
ORA-24338: statement handle not executed
ive truncated the message, it keeps repeating for another 50+ lines
here is what my input file looks like:
elvis[kxv4]{8}%     head grid-9.xyz
51111345 5.5 39.81 -120.64 AUG 10 2001 20:19:27 GMT -121.883 38.9833 -119.383 40.65 (Process time: Thu Sep 26 09:24:45 2002)
-121.8833 40.6500 0.9135 0.4790 3.0900 1.3490 0.5076 0.1056
-121.8666 40.6500 0.7209 0.3115 2.8600 1.0659 0.3301 0.0688
-121.8500 40.6500 0.7195 0.3110 2.8600 1.0650 0.3297 0.0689
-121.8333 40.6500 0.9087 0.4771 3.0800 1.3465 0.5059 0.1059
-121.8166 40.6500 0.7170 0.3102 2.8600 1.0638 0.3290 0.0690
-121.8000 40.6500 0.7161 0.3100 2.8600 1.0640 0.3287 0.0690
-121.7833 40.6500 0.7156 0.3098 2.8600 1.0649 0.3287 0.0690
-121.7666 40.6500 0.7156 0.3099 2.8600 1.0666 0.3288 0.0690
-121.7500 40.6500 0.7162 0.3103 2.8600 1.0693 0.3292 0.0690
here is what my control file looks like:
elvis[kxv4]{10}% head -20 51111345grid-9.xyz
LOAD DATA
INFILE *
INTO TABLE shake_xyz
APPEND
FIELDS TERMINATED BY ',' TRAILING NULLCOLS
( ID,
LON,
LAT,
PGA,
PGV,
MMI,
PSA1 NULLIF PSA1=BLANKS,
PSA2 NULLIF PSA2=BLANKS,
PSA3 NULLIF PSA3=BLANKS,
OBJECTID SEQUENCE(MAX,1)
BEGINDATA
51111345,-121.8833,40.6500,0.9135,0.4790,3.0900,1.3490,0.5076,0.1056
51111345,-121.8666,40.6500,0.7209,0.3115,2.8600,1.0659,0.3301,0.0688
for earthquakes with magnitude less than 5 the PSA% columns are blank ...
i'm woefully ignorant of sqlldr and how it works with indexes
any advice appreciated
thanks
--kassim

thanks dan,
thats a great idea (im using 9.2.0.4)
each event is assigned a unique id by usgs. But this number does not look like a sequence (higher numbers do not correspond to more recent events), but i also assign a sequence number to the event_id which i keep track of in an event table which has only one entry for each event instead of the 15k entries for the shake table.
time to read up on partitions...
thanks again; you rock!
--kassim

Bulk loading BLOBs using PL/SQL - is it possible?

Hi -
Does anyone have a good reference article or example of how I can bulk load BLOBs (videos, images, audio, office docs/pdf) into the database using PL/SQL?
Every example I've ever seen in PL/SQL for loading BLOBs does a commit; after each file loaded ... which doesn't seem very scalable.
Can we pass in an array of BLOBs from the application, into PL/SQL and loop through that array and then issue a commit after the loop terminates?
Any advice or help is appreciated. Thanks
LJ

It is easy enough to modify the example to commit every N files. If you are loading large amounts of media, I think that you will find that the time to load the media is far greater than the time spent in SQL statements doing inserts or retrieves. Thus, I would not expect to see any significant benefit to changing the example to use PL/SQL collection types in order to do bulk row operations.
If your goal is high performance bulk load of binary content then I would suggest that you look to use Sqlldr. A PL/SQL program loading from BFILEs is limited to loading files that are accessible from the database server file system. Sqlldr can do this but it can also load data from a remote client. Sqlldr has parameters to control batching of operations.
See section 7.3 of the Oracle Multimedia DICOM Developer's Guide for the example Loading DICOM Content Using the SQL*Loader Utility. You will need to adapt this example to the other Multimedia objects (ORDImage, ORDAudio .. etc) but the basic concepts are the same.
Once the binary content is loaded into the database, you will need a to write a program to loop over the new content and initialize the Multimedia objects (extract attributes). The example in 7.3 contains a sample program that does this for the ORDDicom object.

Bulk loading in 11.1.0.6

Hi,
I'm using bulk load to load about 200 million triples into one model in 11.1.0.6. The data is splitted into about 60 files with around 3 millions triples in each file. I have a script file which has
host sqlldr ...FILE1;
exec sem_apis.bulk_load_from_staging_table(...);
host sqlldr ...FILE2;
exec sem_apis.bulk_load_from_staging_table(...);
for every file to load.
When I run the script from command line, it looks that the time needed for the loading grows as more files are loaded. The first file took about 8 min to load, the second file took about 25 min,... It's now taking 2 and half hour to load one file after completing loading 14 files.
Is index rebuild causing this behavior? If that's the case is there any way to turn off the index during bulk loading? If the index rebuild is not the case what other parameters can we adjust to speed up the bulk loading?
Thanks,
Weihua

Bulk-append is slower than bulk-load because of incremental index maintenance. The uniqueness constraint enforcing index cannot be disabled. I'd suggest moving to 11.1.0.7 and then installing patch 7600122 to be able to make use of enhanced bulk-append that performs much better than in 11.1.0.6.
The best way to load 200 million rows in 11.1.0.6 would be to load into an empty RDF model via a single bulk-load. You can do it as follows (assuming the filenames are f1.nt thru f60.nt):
- [create a named pipe] mkfifo named_pipe.nt
- cat f*.nt > named_pipe.nt
on a different window:
- run sqlldr with named_pipe.nt as the data file to load all 200 million rows into a staging table (you could create staging table with COMPRESS option to keep the size down)
- next, run exec sem_apis.bulk_load_from_staging_table(...);
(I'd also suggest use of COMPRESS for the application table.)

Slow bulk load

I have a IOT table and am trying to bulk load 500M rows into it. Using sqlldr it looks like it will take a week while I was able to load MySQL in 3 hours. If anyone has any helpful advice to speed up the load it would be greatly appreciated.
My control file contains
OPTIONS (DIRECT=TRUE, ERRORS=5000000)
UNRECOVERABLE LOAD DATA
APPEND
into table applications
fields terminated by "|"
TRAILING NULLCOLS
( list of my 37 columns... )
And I run the command
sqlldr mydatabase control=loader.ctl skip_index_maintenance=true log=log.out data=mydata.dat
I tried using SORTED INDEXES in the control file however sqlldr would complain records were not in sorted order though they are. I read the error is caused by crossing extents. Should I just set the table to have a 200GB extent?

The following may be of assistance:
HW enqueue contention with LOB
As a workaround is it possible to reduce the number of concurrent updates to the LOBs (say reduce 8 to 4)? Also with a PCTVERSION of 0 there is a possibility you may get a snapshot too old error unless you've set RETENTION to some value (really depends on workload).
You've probably seen this although not much help for your problem:
LOB Performance Guidelines

SSRS 2005 report: Cannot bulk load Operating system error code 5(Access is denied.)

I built a SSRS 2005 report, which calls a stored proc on SQL Server 2005. The proc contains following code:
CREATE TABLE #promo (promo VARCHAR(1000))
BULK
INSERT #promo
FROM '\\aseposretail\c$\nz\promo_names.txt'
WITH
--FIELDTERMINATOR = '',
ROWTERMINATOR = '\n'
SELECT * from #promo
It's ok when I manually execute the proc in SSMS.
When I try to run the report from BIDS I got following error:
*Cannot bulk load because the file "\aseposretail\c$\nz\promo_names.txt" could not be opened. Operating system error code 5(Access is denied.).*
Note: I have gooled a bit and see many questions on this but they are not relevant because I CAN run the code no problem in SSMS. It's the SSRS having the issue. I know little about the security of SSRS.

I'm having the same type of issue. I can bulk load the same file into the same table on the same server using the same login on one workstation, but not on another. I get this error:
Msg 4861, Level 16, State 1, Line 1
Cannot bulk load because the file "\\xxx\abc.txt" could not be opened. Operating system error code 5(Access is denied.).
I've checked SQL client versions and they are the same, I've also set the client connection to TCP/IP only in the SQL Server Configuration Manager. Still this one workstation is getting the error. Since the same login is being used on both workstations and it works on one but not the other, the issue is not a permissions issue. I can also have another user login into the bad workstation and have the bulk load fail, but when they log into their regular workstation it works fine. Any ideas on what the client configuration issue is? These are the version numbers for Management Studio:
Microsoft SQL Server Management Studio 9.00.3042.00
Microsoft Analysis Services Client Tools 2005.090.3042.00
Microsoft Data Access Components (MDAC) 2000.085.1132.00 (xpsp.080413-0852)
Microsoft MSXML 2.6 3.0 5.0 6.0
Microsoft Internet Explorer 6.0.2900.5512
Microsoft .NET Framework 2.0.50727.1433
Operating System 5.1.2600
Thanks,
MWise

How to UPDATE a big table in Oracle via Bulk Load

Hi all,
in a datastore target as Oracle 11g, I have a big table having 300milions of record; the structure is One integer key + 10 columns attributes .
In IQ Source i have the same table with the same size ; the structure is One integer key + 1 column attributes .
What i need to do is to UPDATE that single field in Oracle from the values stored in IQ .
Any idea on how to organize efficiently the dataflow and the target writing mode ? bulk load ? api ?
thank you
Maurizio

Hi,
You cannot do bulk load when you need to UPDATE a field. Because all a bulk load does is add records to your table.
Since you have to UPDATE a field, i would suggest to go for SCD with
source > TC > MO > KG >target
Arun

Error while running bulk load utility for account data with CSV file

Hi All,
I'm trying to run the bulk load utility for account data using CSV but i'm getting following error...
ERROR ==> The number of CSV files provided as input does not match with the number of account tables.
Thanks in advance........

Please check your child table.
http://docs.oracle.com/cd/E28389_01/doc.1111/e14309/bulkload.htm#CHDCGGDA
-kuldeep

Bulk Load option doesn't work

Hi Experts,
I am trying to load data to HFM using Bulk load option but it doesnt work. When I Change the option to SQL insert, the loading is successful. The logs say that the temp file is missing. But when I go to the lspecified location , I see the control file and the tmp file. What am I missing to have bulk load working?Here's the log entry.
2009-08-19-18:48:29
User ID...........     kannan
Location..........     KTEST
Source File.......     \\Hyuisprd\Applications\FDM\CRHDATALD1\Inbox\OMG\HFM July2009.txt
Processing Codes:
BLANK............. Line is blank or empty.
ESD............... Excluded String Detected, SKIP Field value was found.
NN................ Non-Numeric, Amount field contains non numeric characters.
RFM............... Required Field Missing.
TC................ Type Conversion, Amount field could be converted to a number.
ZP................ Zero Suppress, Amount field contains a 0 value and zero suppress is ON.
Create Output File Start: [2009-08-19-18:48:29]
[TC] - [Amount=NN]     Batch Month File Created: 07/2009
[TC] - [Amount=NN]     Date File Created: 8/6/2009
[TC] - [Amount=NN]     Time File Created: 08:19:06
[Blank] -
Excluded Record Count.............. 3
Blank Record Count................. 1
Total Records Bypassed............. 4
Valid Records...................... 106093
Total Records Processed............ 106097
Begin Oracle (SQL-Loader) Process (106093): [2009-08-19-18:48:41]
[RDMS Bulk Load Error Begin]
     Message:      (53) - File not found
     See Bulk Load File:      C:\DOCUME~1\fdmuser\LOCALS~1\Temp\tWkannan30327607466.tmp
[RDMS Bulk Load Error End]
Thanks
Kannan.

Hi Experts,
I am facing the data import error while importing data from .csv file to FDM-HFM application.
2011-08-29 16:19:56
User ID...........     admin
Location..........     ALBA
Source File.......     C:\u10\epm\DEV\epm_home\EPMSystem11R1\products\FinancialDataQuality\FDMApplication\BMHCFDMHFM\Inbox\ALBA\BMHC_Alba_Dec_2011.csv
Processing Codes:
BLANK............. Line is blank or empty.
ESD............... Excluded String Detected, SKIP Field value was found.
NN................ Non-Numeric, Amount field contains non numeric characters.
RFM............... Required Field Missing.
TC................ Type Conversion, Amount field could be converted to a number.
ZP................ Zero Suppress, Amount field contains a 0 value and zero suppress is ON.
Create Output File Start: [2011-08-29 16:19:56]
[ESD] ( ) Inter Co,Cash and bank balances,A113000,Actual,Alba,Dec,2011,MOF,MOF,,YTD,Input_Default,[NONE],[NONE],[NONE],1
[ESD] ( ) Inter Co,"Trade receivable, prepayments and other assets",HFM128101,Actual,Alba,Dec,2011,MOF,MOF,,YTD,Input_Default,[NONE],[NONE],[NONE],35
[ESD] ( ) Inter Co,Inventories ,HFM170003,Actual,Alba,Dec,2011,MOF,MOF,,YTD,Input_Default,[NONE],[NONE],[NONE],69
[ESD] ( ) Inter Co,Financial assets carried at fair value through P&L,HFM241001,Actual,Alba,Dec,2011,MOF,MOF,,YTD,Input_Default,[NONE],[NONE],[NONE],103
[Blank] -
Excluded Record Count..............4
Blank Record Count.................1
Total Records Bypassed.............5
Valid Records......................0
Total Records Processed............5
Begin SQL Insert Load Process (0): [2011-08-29 16:19:56]
Processing Complete... [2011-08-29 16:19:56]
Please help me solve the issue.
Regards,
Sudhir Sinha

Bulk Insert Task Cannot bulk load because the file could not be opened.operating system error error code 3(The system cannot find the path specified.)

Following error i am getting after i chnaged the Path in Config File from
\\vs01\d$\\Deployment\Files\temp.txt
to
C:\Deployment\Files\temp.txt
[Bulk Insert Task] Error: An error occurred with the following error message: "Cannot bulk load because the file "C:\Deployment\Files\temp.txt" could not be opened. Operating system error code 3(The system cannot find the path specified.).".

I think i know whats going on. The Bulk Insert task runs by executing sql command (bulk insert) internally from the target sql server to load the file. This means that the SQL Server Agent of the target sql server should have permissions on the file you trying to load. This also means that you need to use UNC path instead to specify the file path (if the target server in on different machine)
Also from BOL (see section Usage Considerations - last bullet point)
http://msdn.microsoft.com/en-us/library/ms141239.aspx
* Only members of the sysadmin fixed server role can run a package that contains a Bulk Insert task.
Make sure you take care of this as well.
HTH
~Mukti
Mukti

Issue with Bulk Load Post Process Scheduled Task

Hello,
I successfully loaded users in OIM using the bulk load utility. I also have LDAP sync ON. The documentation says to run the Bulk Load Post Process scheduled task to push the loaded users in OIM into LDAP.
This works if we run the Bulk Load Post Process Scheduled Task right away after the run the bulk load.
If some time had passed and we go back to run the Bulk Load Post Process Scheduled Task, some of the users loaded through the bulk load utility are not created in our LDAP system. This created an off-sync situation between OIM and our LDAP.
I tried to use the usr_key as a parameter to the Bulk Load Post Process Scheduled Task without success.
Is there a way to force the re-evaluation of these users so they would get created in LDAP?
Thanks
Khanh

The scheduled task carries out post-processing activities on the users imported through the bulk load utility.

Issue with Bulk Load Post Process

Hi,
I ran bulk load command line utility to create users in OIM. I had 5 records in my csv file. Out of which 2 users were successfully created in OIM and for rest i got exception because users already existed. After that if i run bulk load post process for LDAP sync and generate the password and send notification. It is not working even for successfully created users. Ideally it should sync successfully created users. However if there is no exception i during bulk load command line utility then LDAP sync work fine through bulk load post process.Any idea how to resolve this issue and sync the user in OID which were successfully created. Urgent help would be appreciated.

The scheduled task carries out post-processing activities on the users imported through the bulk load utility.

Bulk load option Data Services Oracle

Hello,
I'm trying to use the option "Bulk load" but it doesn't work.
Error message : "DBS-070301Oracle <DB_INFO>> error message for operation <OCIDirPathPrepare>: <ORA-00942: table or view does not exist"
Is it necessary to affect dba grant for the oracle user that is declared in the Datastore?
The version is : Data Services XI 3.1
The database is : Oracle9i Enterprise Edition Release 9.2.0.7.0
Thx for your answer

this also happens in case there is a mismatch in Oracle Client and Server Version, like Oracle 9i client and 10g server
what is you Oracle Server version ?
you can also check similar post in DI BOB forum
http://www.forumtopics.com/busobj/viewtopic.php?t=122242

Notifications are not being sent when Bulk Load is done

Hi All,
I have OIM 11g setup on my machine. I use the bulk load utility for loading the user data. Now in my OIM setup, the notifications are being sent for all stuff like Reset Password. New account creation and other. However when I bulk load the users, notifications are not sent to their mail ids. I am running the scheduled job "Bulk load Post Process" which is necessary so that the users are synced to the LDAP repository. I have the LDAP Sync option checked and also the Notifications option set to yes in this scheduled job. Though the users are loaded successfully and are synced properly, the notifications are not sent. Can some one please guide me as to what could be the problem here?
Thanks,
$id

The code is probably only called in the Event method of the event handler that sends the notification. You can check the mds files and find the notification you are looking for and then use a code decompiler to find the class that is called. You can then use this code as a sample, or write your own notification code and create an event handler that runs in the BulkEvent.
And on another note there is also this System Configuration Variable: Recon.SEND_NOTIFICATION which is set to FALSE by default.
-Kevin

Bulk Load in SQLLDR

Similar Messages

Maybe you are looking for