Slow bulk load

I have a IOT table and am trying to bulk load 500M rows into it. Using sqlldr it looks like it will take a week while I was able to load MySQL in 3 hours. If anyone has any helpful advice to speed up the load it would be greatly appreciated.
My control file contains
OPTIONS (DIRECT=TRUE, ERRORS=5000000)
UNRECOVERABLE LOAD DATA
APPEND
into table applications
fields terminated by "|"
TRAILING NULLCOLS
( list of my 37 columns... )
And I run the command
sqlldr mydatabase control=loader.ctl skip_index_maintenance=true log=log.out data=mydata.dat
I tried using SORTED INDEXES in the control file however sqlldr would complain records were not in sorted order though they are. I read the error is caused by crossing extents. Should I just set the table to have a 200GB extent?

The following may be of assistance:
HW enqueue contention with LOB
As a workaround is it possible to reduce the number of concurrent updates to the LOBs (say reduce 8 to 4)? Also with a PCTVERSION of 0 there is a possibility you may get a snapshot too old error unless you've set RETENTION to some value (really depends on workload).
You've probably seen this although not much help for your problem:
LOB Performance Guidelines

Similar Messages

4.2.3/.4 Data load wizard - slow when loading large files

Hi,
I am using the data load wizard to load csv files into an existing table. It works fine with small files up to a few thousand rows. When loading 20k rows or more the loading process becomes very slow. The table has a single numeric column for primary key.
The primary key is declared at "shared components" -> logic -> "data load tables" and is recognized as "pk(number)" with "case sensitve" set to "No".
While loading data, these configuration leads to the execution of the following query for each row:
select 1 from "KLAUS"."PD_IF_CSV_ROW" where upper("PK") = upper(:uk_1)
which can be found in the v$sql view while loading.
It makes the loading process slow, because of the upper function no index can be used.
It seems that the setting of "case sensitive" is not evaluated.
Dropping the numeric index for the primary key and using a function based index does not help.
Explain plan shows an implicit "to_char" conversion:
UPPER(TO_CHAR(PK)=UPPER(:UK_1)
This is missing in the query but maybe it is necessary for the function based index to work.
Please provide a solution or workaround for the data load wizard to work with large files in an acceptable amount of time.
Best regards
Klaus

Nevertheless, a bulk loading process is what I really like to have as part of the wizard.
If all of the CSV files are identical:
use the Excel2Collection plugin ( - Process Type Plugin - EXCEL2COLLECTIONS )
create a VIEW on the collection (makes it easier elsewhere)
create a procedure (in a Package) to bulk process it.
The most important thing is to have, somewhere in the Package (ie your code that is not part of APEX), information that clearly states which columns in the Collection map to which columns in the table, view, and the variables (APEX_APPLICATION.g_fxx()) used for Tabular Forms.
MK

PL/SQL Bulk Loading

Hello,
I have one question regarding bulk loading. I did lot of bulk loading.
But my requirement is to call function which will do some DML operation and give ref key so that i can insert to fact table.
Because i can't use DML function in select statement. (which will give error). otherway is using autonomous transaction. which i tried working but performance is very slow.
How to call this function inside bulk loading process.
Help !!
xx_f is function which is using autonmous transction,
See my sample code
declare
cursor c1 is select a,b,c from xx;
type l_a is table of xx.a%type;
type l_b is table of xx.b%type;
type l_c is table of xx.c%type;
v_a l_a;
v_b l_b;
v_c l_c;
begin
open c1;
loop
fetch c1 bulk collect into v_a,v_b,v_c limit 1000;
exit when c1%notfound;
begin
forall i in 1..v_a.count
insert into xxyy
(a,b,c) values (xx_f(v_a(i),xx_f(v_b(i),xx_f(v_c(i));
commit;
end bulkload;
end loop;
close c1;
end;
I just want to call xx_f function without autonoumous transaction.
but with bulk loading. Please let me if you need more details
Thanks
yreddyr

Can you show the code for xx_f? Does it do DML, or just transformations on the columns?
Depending on what it does, an alternative could be something like:
DECLARE
   CURSOR c1 IS
      SELECT xx_f(a), xx_f(b), xx_f(c) FROM xx;
   TYPE l_a IS TABLE OF whatever xx_f returns;
   TYPE l_b IS TABLE OF whatever xx_f returns;
   TYPE l_c IS TABLE OF whatever xx_f returns;
   v_a l_a;
   v_b l_b;
   v_c l_c;
BEGIN
   OPEN c1;
   LOOP
      FETCH c1 BULK COLLECT INTO v_a, v_b, v_c LIMIT 1000;
      BEGIN
         FORALL i IN 1..v_a.COUNT
            INSERT INTO xxyy (a, b, c)
            VALUES (v_a(i), v_b(i), v_c(i));
      END;
      EXIT WHEN c1%NOTFOUND;
   END LOOP;
   CLOSE c1;
END;John

Bulk loading in 11.1.0.6

Hi,
I'm using bulk load to load about 200 million triples into one model in 11.1.0.6. The data is splitted into about 60 files with around 3 millions triples in each file. I have a script file which has
host sqlldr ...FILE1;
exec sem_apis.bulk_load_from_staging_table(...);
host sqlldr ...FILE2;
exec sem_apis.bulk_load_from_staging_table(...);
for every file to load.
When I run the script from command line, it looks that the time needed for the loading grows as more files are loaded. The first file took about 8 min to load, the second file took about 25 min,... It's now taking 2 and half hour to load one file after completing loading 14 files.
Is index rebuild causing this behavior? If that's the case is there any way to turn off the index during bulk loading? If the index rebuild is not the case what other parameters can we adjust to speed up the bulk loading?
Thanks,
Weihua

Bulk-append is slower than bulk-load because of incremental index maintenance. The uniqueness constraint enforcing index cannot be disabled. I'd suggest moving to 11.1.0.7 and then installing patch 7600122 to be able to make use of enhanced bulk-append that performs much better than in 11.1.0.6.
The best way to load 200 million rows in 11.1.0.6 would be to load into an empty RDF model via a single bulk-load. You can do it as follows (assuming the filenames are f1.nt thru f60.nt):
- [create a named pipe] mkfifo named_pipe.nt
- cat f*.nt > named_pipe.nt
on a different window:
- run sqlldr with named_pipe.nt as the data file to load all 200 million rows into a staging table (you could create staging table with COMPRESS option to keep the size down)
- next, run exec sem_apis.bulk_load_from_staging_table(...);
(I'd also suggest use of COMPRESS for the application table.)

SSRS 2005 report: Cannot bulk load Operating system error code 5(Access is denied.)

I built a SSRS 2005 report, which calls a stored proc on SQL Server 2005. The proc contains following code:
CREATE TABLE #promo (promo VARCHAR(1000))
BULK
INSERT #promo
FROM '\\aseposretail\c$\nz\promo_names.txt'
WITH
--FIELDTERMINATOR = '',
ROWTERMINATOR = '\n'
SELECT * from #promo
It's ok when I manually execute the proc in SSMS.
When I try to run the report from BIDS I got following error:
*Cannot bulk load because the file "\aseposretail\c$\nz\promo_names.txt" could not be opened. Operating system error code 5(Access is denied.).*
Note: I have gooled a bit and see many questions on this but they are not relevant because I CAN run the code no problem in SSMS. It's the SSRS having the issue. I know little about the security of SSRS.

I'm having the same type of issue. I can bulk load the same file into the same table on the same server using the same login on one workstation, but not on another. I get this error:
Msg 4861, Level 16, State 1, Line 1
Cannot bulk load because the file "\\xxx\abc.txt" could not be opened. Operating system error code 5(Access is denied.).
I've checked SQL client versions and they are the same, I've also set the client connection to TCP/IP only in the SQL Server Configuration Manager. Still this one workstation is getting the error. Since the same login is being used on both workstations and it works on one but not the other, the issue is not a permissions issue. I can also have another user login into the bad workstation and have the bulk load fail, but when they log into their regular workstation it works fine. Any ideas on what the client configuration issue is? These are the version numbers for Management Studio:
Microsoft SQL Server Management Studio 9.00.3042.00
Microsoft Analysis Services Client Tools 2005.090.3042.00
Microsoft Data Access Components (MDAC) 2000.085.1132.00 (xpsp.080413-0852)
Microsoft MSXML 2.6 3.0 5.0 6.0
Microsoft Internet Explorer 6.0.2900.5512
Microsoft .NET Framework 2.0.50727.1433
Operating System 5.1.2600
Thanks,
MWise

Web pages slow to load with IE11 and the cursor/mouse is very slow to work until after page loads

Web pages are very slow to load, I hve run Norton Eraser and found no issues that would cause this

New HP Pavilion three weeks ago. After several attempts by the 'experts' I finally got a straight answer from a tech in the Philipines yesterday, 01-29-2015. HP and Microsoft are working to resolve the issue of slow pages and a dead cursor until the next page finally loads. You may have discovered that Google Chrome DOES NOT have this issue. Comments about HP would not be allowed here so I will just say, use Chrome until the issue with IE 11 is resolved.

Firefox 4.0 keeps locking up and is slow to load pages

I just installed 4.0 this morning. Ever since, Hotmail is extremely slow to load, it keeps locking up or not responding.
I thought this was supposed to be faster and more streamlined?
Now, when I try to go back and install 3.6.13 it is no longer available. What gives? I want my old Firefox back.

http://www.mozilla.com/en-US/firefox/all-older.html

Since upgrading to ios8, pages very slow to load, and can no longer read items in my reading list when offline. ?????

SI since upgrading to ios8, Safari pages are very slow to load, Kindle crashes upon opening, and I can no longer read saved artices from my reading list when offline. Big disappointment. Is it possible to downgrade to previous ios?

I actually managed to fix it! But unfortunately I don't know what I was that I did that helped.
Anyways, this is kind of what I did.
1. I tried to reset form bakup via iTunes, no change.
2. I tried to reset as a new iPhone, problem still there and reading list also as bookmarks!? (several hundred)
3. Noticed that I could delete a bookmark, once in a while, the other times I got the 'Bookmarks are being synced. Please edit the bookmarks once syncing has completed.' message.
4. Because I figured that it was a memory problem, I thought it made sense to delete as many as possible, witch I did.
5. Noticed that it was more likely that I coould delete a bookmark if I restarted Safari.
6. Notice that it was even more likely that I could delete a bookmark if I turned the Safari sync with iCould off (with option detete information on iPhone) and then back on again.
7. When I was about half way through deleting the bookmark list I suddenly discovered that all but one of the items in my reading list was 'waiting' and one was downloading. I kind of was 'unstuck' now.
8. I started to delete items in my reading list, but every other time I got the 'Bookmarks are being synced. Please edit the bookmarks once syncing has completed.' message.
9. It took a while to delete all bookmarks and items in the reading list.
Hope you guys could use the info here to fix your problems too!

Mac seems slow to load web pages and email since mavericks upgrade

Since upgrade from Mountain Lion to Mavericks on my iMac, web pages seem a lot slower to load on screen. Also, when I open my mail inbox, the mail content (wording) takes a long time before appearing on the right hand side, then followed a little later with the display content. I have synchronised all my mail accounts and also emptied my cache in the safari browser, but this has not resolved the problem.

Go to Applications (you can do this by clicking on the desktop and hitting cmd-shift-A). Within the Utilities folder you should see a program called Terminal. Launch this program, which should just be a window that you can type into. Type the following (or just copy and paste into the window) and hit return:
     networksetup -setv6off Wi-fi
Then go back to Network Preferences and verify that IPv6 is set to off. If that doesn't help, enter these two commands in succession:
     cd /Library/Preferences/
     sudo mv com.apple.security-common.plist ~/Desktop/
Note that this moves a configuration file onto your desktop. If that doesn't fix things you may want to put it back:
     cd /Library/Preferences/
     sudo mv ~/Desktop/com.apple.security-common.plist .
Good luck!

Firefox is slow to load and view pages and repeated "Firefox is not responding" message

Firefox is quite slow to load and view pages with and I keep getting the "Firefox is not responding" message.
I have the latest version, plus the latest Adobe Flash Player, Shockwave Player, etc.

Note that Firefox 36 has been released officially.
*https://www.mozilla.org/en-US/firefox/36.0/releasenotes/
It is possible that your security software (firewall, anti-virus) blocks or restricts Firefox or the plugin-container process without informing you, possibly after detecting changes (update) to the Firefox program.
Remove all rules for Firefox and the plugin-container from the permissions list in the firewall and let your firewall ask again for permission to get full, unrestricted, access to install for Firefox and the plugin-container process and the updater process.
See:
*https://support.mozilla.org/kb/Server+not+found
*https://support.mozilla.org/kb/Firewalls
*https://support.mozilla.org/kb/fix-problems-connecting-websites-after-updating

Firefox 12 is very slow to load pages and if two tabs are opening it lags.

Firefox 12 is very slow to load pages and if two tabs are opening it lags. I've tried everything (e.g virus, defrag, I've created a new profile in "run", etc). I've had my router/line etc all checked and after 2 weeks of cleaning/testing etc its all pointing to Firefox as the problem. Please help as it's driving me nuts.

I have the exact same problem.
Not only does it load pages very slowly, sometimes the page will stop for a good 45 seconds and stop loading, only to finish it much later.
Something is wrong with Firefox 12, it shouldn't be this slow.
How is that each version is progressively worse than the one before it?.
Stop releasing so many versions Mozilla, release one that works and build from there.

Slow to load BB

Hello, Im new to BT but have BT phone and Broadband unlimited evening and weekends. My BB was switched on on 28th August and I realise it can take up to 10 days to stabilise from reading the info given. However it is very slow to load pages which often time out and its impossible to play games on XBOX live or watch youtube clips as it is still that bad. Will it improve at all?
Thank you
Solved!
Go to Solution.

Hi,
Sorry to hear that your having problems with your broadband.
Please see Keith's help guide here: Helping forum members to help you, it will go through some checks that are needed for us to help you.
A summary of the checks are:
1) you should be connected by the BT NTE5 master socket as extension sockets can reduce broadband performance
2) Can you please run a BT speed test (including IP Profile) http://speedtester.bt.com (not beta version)[Best done with a wired, ethernet, connection]
3) is there any noise on your line. dial 17070 option2 from landline. should be silent but slight hum normal on cordless phone.
4) please post adsl line statistics
ADSL Line Statistic Help:
If you have a BT Home Hub like the one below...
Then:
1) Go to http://192.168.1.254 or http://bthomehub.home
2) click Settings
3) Click Advanced Settings
4) Click Broadband
5) Click Connection or sometimes called ADSL (see picture Below)
The direct Address is http://bthomehub.home/index.cgi?active_page=9116 (for bthomehub3)
You will need to copy and past all the adsl line statistics ( Including HEC, CRC and FEC errors). You may need to click " More Details"
There are more useful links on Keith's website here: If you have an ADSL connection, please select this link
cheers
I'm no expert, so please correct me if I'm wrong

Pages are slow to load or stop loading

Once I'm connected to wi-fi, my pages are slow to load or stop loading and tell me 'I'm not connected".

If you are having the problem on more than one computer it would appear not to be a computer fault or your browser that is the problem.
The first thing I would do is a factory reset of the homehub by pressing a pin into the recess button on the rear for about 20 seconds. This can often sort out problems and will also give you a "clean sheet" to start some diagnostics from.
If this doesn't help please post what Infinity package you are on, I assume with 18Mbps it's Infinity 1 but please confirm, and do you have an Openreach modem and what homehub you have.
Also run this checker and post back a screen shot of the results
http://www.dslchecker.bt.com/adsl/adslchecker.welcome
and use this speed tester then carry out the further diagnostics and post back a screen shot of all the results including your IP profile for up and down. This test must be done with a wired connection.
http://speedtest.btwholesale.com/

PAGES slow to load

Does anyone else think "PAGES" is slow to load and what if anything can be done about it?

the rest of the Scrip: is htt
Script: http://l1.yimg.com/zz/combo?yui:3.5.1/build/yui/yui-min.js&os/mit/media/m/base/i mageloader-min-875847.js&os/mit/media/m/base/imageloader-bootstrap-min-815727.js &os/mit/media/m/base/viewport-loader-min-993847.js&os/mit/media/p/common/rmp-min -1063452.js:14

How to UPDATE a big table in Oracle via Bulk Load

Hi all,
in a datastore target as Oracle 11g, I have a big table having 300milions of record; the structure is One integer key + 10 columns attributes .
In IQ Source i have the same table with the same size ; the structure is One integer key + 1 column attributes .
What i need to do is to UPDATE that single field in Oracle from the values stored in IQ .
Any idea on how to organize efficiently the dataflow and the target writing mode ? bulk load ? api ?
thank you
Maurizio

Hi,
You cannot do bulk load when you need to UPDATE a field. Because all a bulk load does is add records to your table.
Since you have to UPDATE a field, i would suggest to go for SCD with
source > TC > MO > KG >target
Arun

Slow bulk load

Similar Messages

Maybe you are looking for