Proper array size to be used in bulk insert

What is the proper array size to be used in bulk insert?
I have around 1 million records. Should I insert them all at a time or distribute them over many iterations.

I'd generally expect external tables to be more efficient than SQL*Loader if only because you don't have to spend cycles loading data into a staging table. Depending on the file, Oracle may be able to access the data in parallel via the external table interface.
From a pure efficiency standpoint, it would be best if the processing could be encapsulated into a single multi-table insert statement. Whether that is realistic, of course, depends on your logic, how complicated that SQL statement would be, etc. If you have to resort to PL/SQL, bulk processing is surely the way to go. it doesn't matter too much what LIMIT size you choose-- something like 100 or 1000 is generally appropriate-- but the marginal difference is pretty small. On a load of only 1 million rows, it may not be particularly easy to measure.
Justin

Similar Messages

Using of bulk insert create problem

How to use the bulk insert statement in my jsp and which should be updated into the database.
I've tried this one:
bulk insert test from test1.txt with (fieldterminator=':')but it gives an error as:
u don't have permission to use bulk statement.
Could any1 get me rid off from this problem.
in advance thanks.....

Most likely the person responsible for permissions in your database could help you out there. If you're here looking for a way to execute commands that you aren't authorized to execute, you are wasting your time.

How to use BULK INSERT for a data from a cursor?

Oracle 10G enterprise edition.
I tried to Bulk insert datas returning from a cursor, its returning error.
PLS-00302: component 'LAST' must be declared
I need some help to use the Bulk INSERT here.Can any one help me to specify what error i have made?
CREATE OR REPLACE PROCEDURE HOT_ADMIN.get_search_keyword_stats_prc
IS
CURSOR c_get_scenarios
IS
SELECT a.*,ROWNUM rnum
FROM (
SELECT TRUNC(r.search_date) sdate,
r.search_hits hits,
r.search_type stype,
r.search_qualification qual,
r.search_location loc,
r.search_town stown,
r.search_postcode pcode,
r.search_college college,
r.search_colname colname,
r.search_text text,
r.affiliate_id affiliate,
r.search_study_mode smode,
r.location_hint hint,
r.search_posttown ptown,
COUNT(1) cnt
FROM w_search_headers r
WHERE search_text IS NOT NULL
AND NVL(search_type,' ') <> 'C'
AND TRUNC(search_date)= TO_DATE(TO_CHAR(SYSDATE-1,'DD-MON-RRRR'))
GROUP BY TRUNC(r.search_date),
r.search_hits,
r.search_type,
r.search_qualification,
r.search_location,
r.search_town,
r.search_postcode,
r.search_college,
r.search_colname,
r.search_text,
r.affiliate_id,
r.search_study_mode,
r.location_hint,
r.search_posttown
ORDER BY cnt desc
) a
WHERE ROWNUM <=1000;
lc_get_data c_get_scenarios%ROWTYPE;
BEGIN
OPEN c_get_scenarios;
FETCH c_get_scenarios into lc_get_data;
CLOSE c_get_scenarios;
FORALL i IN 1..lc_get_data.last
INSERT INTO W_SEARCH_SCENARIO_STATS VALUES ( i.sdate,
i.hits,
i.stype,
i.qual,
i.loc,
i.stown,
i.pcode,
i.college,
i.colname,
i.text,
i.affiliate,
i.smode,
i.hint,
i.ptown,
i.cnt
COMMIT;
END;

This isn't what you asked, but I've generally found it helpful to list the columns in an INSERT statement before the values. It is of course optional, but useful for reference when looking at the statement later

How to pass a parameter into a select statement using a bulk collect

Using a cursors I can use execute immediate to pass a parameter to specify a table name.
I have modified the procedure not to use the cursor. I am using a collection using a bulk insert into a nested table. How can I specify the table the table name as a parameter without using the execute immediate statement.
Thank you

Hi Donald,
The functionality of using a parameter in a function is only available in the formula editor and not in SQL Expression. Parameters cannot be passed to the SQL Expression, the main purpose of the SQL Expression is to make use of Database functions to make calculations at the database level but the current functionality of the product only allows you to use Database fields.
There is an article that talks on SQL Expressions and can be useful http://technicalsupport.businessobjects.com/KanisaSupportSite/search.do?cmd=displayKC&docType=kc&externalId=c2016184&sliceId=&dialogID=6018612&stateId=1%200%206020316
Regards,
Vinay

Sequence in Bulk Insert

Hi All
Just one doubt,
whether sequence can be used in Bulk insert query ?

Have you tried creating a test case? See below:
BANNER
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production
PL/SQL Release 11.2.0.1.0 - Production
CORE    11.2.0.1.0      Production
TNS for 32-bit Windows: Version 11.2.0.1.0 - Production
NLSRTL Version 11.2.0.1.0 - Production
SQL > CREATE table test_tab(id NUMBER, ename VARCHAR2(10));
Table created.
SQL > CREATE SEQUENCE test_seq;
Sequence created.
SQL > DECLARE
2          TYPE e_list is TABLE OF test_tab.ename%TYPE INDEX BY PLS_INTEGER;
3
4          emps e_list;
5 BEGIN
6          emps(1) := 'Jeff';
7          emps(2) := 'Jane';
8          emps(3) := 'Joe';
9
10          FORALL i IN emps.FIRST .. emps.LAST
11                  INSERT INTO test_tab(id, ename) VALUES(test_seq.NEXTVAL,emps(i));
12
13          COMMIT;
14 END;
15 /
PL/SQL procedure successfully completed.
SQL > SELECT * FROM test_tab;
                  ID ENAME
                   2 Jeff
                   3 Jane
                   4 Joe
SQL > DROP SEQUENCE test_seq;
Sequence dropped.
SQL > DROP TABLE test_tab PURGE;
Table dropped.

Help with BULK Insert

Hello SQL Buddies,
I am trying to run a BULK insert.
Here is my code:
Code start:
use Lagerliste
drop table Import
use Lagerliste
create table Import
TOTALQTY float null,
PARTNO nvarchar (255) null,
[DESC] nvarchar (255) null,
CO nvarchar (255) null,
BIN nvarchar (255) null,
PRICE float null,
DISCOUNT nvarchar (255) null,
LASTPURC nvarchar (255) null,
AGECODE nvarchar (255) null,
MANFPART nvarchar (255) null
use Lagerliste
bulk
insert Import
from 'D:\FTP\RG\Stockfile.csv'
with
formatfile = 'D:\FormatFile\test.xml',
Errorfile = 'D:\FormatFile\error.txt'
Code stop..
My format file code is here:
Code start:
<?xml version="1.0"?>
<BCPFORMAT xmlns = "http://schemas.microsoft.com/sqlserver/2004/bulkload/format"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <RECORD>
      <FIELD ID="1"   xsi:type="CharTerm"       TERMINATOR= ',"'   />
      <FIELD ID="2"   xsi:type="CharTerm"   MAX_LENGTH="255"   TERMINATOR= '","' />
      <FIELD ID="3"   xsi:type="CharTerm"   MAX_LENGTH="255"   TERMINATOR= '","'   />
      <FIELD ID="4"   xsi:type="CharTerm"   MAX_LENGTH="255"   TERMINATOR= '","'   />
      <FIELD ID="5"   xsi:type="CharTerm"   MAX_LENGTH="255"   TERMINATOR= '",'   />
      <FIELD ID="6"   xsi:type="CharTerm"      TERMINATOR= ',"'   />
      <FIELD ID="7"   xsi:type="CharTerm"   MAX_LENGTH="255"   TERMINATOR= '","'   />
      <FIELD ID="8"   xsi:type="CharTerm"   MAX_LENGTH="255"   TERMINATOR= '","'   />
      <FIELD ID="9"   xsi:type="CharTerm"   MAX_LENGTH="255"   TERMINATOR= '","'   />
      <FIELD ID="10"   xsi:type="CharTerm"   MAX_LENGTH="255"   TERMINATOR= '"'       />
    </RECORD>
    <ROW>
      <COLUMN SOURCE="1" NAME="TOTALQTY"   xsi:type="SQLFLT8"/>
      <COLUMN SOURCE="2" NAME="PARTNO"   xsi:type="SQLNVARCHAR"/>
      <COLUMN SOURCE="3" NAME="DESC"       xsi:type="SQLNVARCHAR"/>
      <COLUMN SOURCE="4" NAME="CO"       xsi:type="SQLNVARCHAR"/>
      <COLUMN SOURCE="5" NAME="BIN"       xsi:type="SQLNVARCHAR"/>
      <COLUMN SOURCE="6" NAME="PRICE"       xsi:type="SQLFLT8"/>
      <COLUMN SOURCE="7" NAME="DISCOUNT"   xsi:type="SQLNVARCHAR"/>
      <COLUMN SOURCE="8" NAME="LASTPURC"   xsi:type="SQLNVARCHAR"/>
      <COLUMN SOURCE="9" NAME="AGECODE"   xsi:type="SQLNVARCHAR"/>
      <COLUMN SOURCE="10" NAME="MANFPART"   xsi:type="SQLNVARCHAR"/>
    </ROW>
</BCPFORMAT>
Code stop..
If i run the code it says:
Msg 4832, Level 16, State 1, Line 20
Bulk load: An unexpected end of file was encountered in the data file.
Msg 7399, Level 16, State 1, Line 20
The OLE DB provider "BULK" for linked server "(null)" reported an error. The provider did not give any information about the error.
Msg 7330, Level 16, State 2, Line 20
Cannot fetch a row from OLE DB provider "BULK" for linked server "(null)".
And the error file says:
Row 473629 File Offset 42503226 ErrorFile Offset 0 - HRESULT 0x80004005
If i then in my bulk insert adds: "Lastrow = '473629' it works all fine. So it will mean that it gets an error at the last line where there is no more data? But this sql query should run every day, so i have to do so it dont gets the error.
Can someone help me?
Looking forward to an answer, i hope someone have a solution.
Regards Christian.

Try the below format file,
<?xml version="1.0"?>
<BCPFORMAT xmlns = "http://schemas.microsoft.com/sqlserver/2004/bulkload/format"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<RECORD>
<FIELD ID="1" xsi:type="CharTerm" TERMINATOR= ',"' />
<FIELD ID="2" xsi:type="CharTerm" MAX_LENGTH="255" TERMINATOR= '","' />
<FIELD ID="3" xsi:type="CharTerm" MAX_LENGTH="255" TERMINATOR= '","' />
<FIELD ID="4" xsi:type="CharTerm" MAX_LENGTH="255" TERMINATOR= '","' />
<FIELD ID="5" xsi:type="CharTerm" MAX_LENGTH="255" TERMINATOR= '",' />
<FIELD ID="6" xsi:type="CharTerm" TERMINATOR= ',"' />
<FIELD ID="7" xsi:type="CharTerm" MAX_LENGTH="255" TERMINATOR= '","' />
<FIELD ID="8" xsi:type="CharTerm" MAX_LENGTH="255" TERMINATOR= '","' />
<FIELD ID="9" xsi:type="CharTerm" MAX_LENGTH="255" TERMINATOR= '","' />
<FIELD ID="10" xsi:type="CharTerm" MAX_LENGTH="255" TERMINATOR= '"\r\n' />
</RECORD>
<ROW>
<COLUMN SOURCE="1" NAME="TOTALQTY" xsi:type="SQLFLT8"/>
<COLUMN SOURCE="2" NAME="PARTNO" xsi:type="SQLNVARCHAR"/>
<COLUMN SOURCE="3" NAME="DESC" xsi:type="SQLNVARCHAR"/>
<COLUMN SOURCE="4" NAME="CO" xsi:type="SQLNVARCHAR"/>
<COLUMN SOURCE="5" NAME="BIN" xsi:type="SQLNVARCHAR"/>
<COLUMN SOURCE="6" NAME="PRICE" xsi:type="SQLFLT8"/>
<COLUMN SOURCE="7" NAME="DISCOUNT" xsi:type="SQLNVARCHAR"/>
<COLUMN SOURCE="8" NAME="LASTPURC" xsi:type="SQLNVARCHAR"/>
<COLUMN SOURCE="9" NAME="AGECODE" xsi:type="SQLNVARCHAR"/>
<COLUMN SOURCE="10" NAME="MANFPART" xsi:type="SQLNVARCHAR"/>
</ROW>
</BCPFORMAT>
Refer
http://stackoverflow.com/questions/8530353/sql-bulk-insert-xml-format-file-with-double-quotes-in-terminator
http://technet.microsoft.com/en-us/library/ms191234(v=sql.105).aspx
Regards, RSingh

Set Fixed Array Size

Hello. I am trying to develop a VI for an FPGA target and I'm trying to make it so whoever is using the VI can specify the size of an input array when they instantiate it (such as by having an input to the VI that says "max array size" or something). So far it seems like the only way to do something like this is to go into the VI itself and manually set the array size for each input array for the VI (i.e. open the VI, right click on each input array, select "Set Dimension SIze", choose "fixed", set number of elements). I'd like to have one point where I can set the size of all the arrays at once (since in my case they are all the same size), and let the person using the VI set it to the proper size for whatever application they are using it for (to be clear, let's say I have two uses for this VI, and in one case I have an input array of size 100 and in anohter use of size 1000, I'd like to be able to set these somehow instead of doing it through the dialog). Thanks.
Solved!
Go to Solution.

Unfortunately, per-call configuration of array size is not something that is supported today. This is a great idea, I would suggest that you post it on the FPGA Idea Exchange so we can get a feel for how many other people need this same feature.

Data type of preprend array size in Flatten To String block

Hi
The data type of the preprend array size in Flatten To String block is given as I32. Is it somehow possible to reduce the data type to I8, since the width and height of my array won't exceed 255 ?. I also need to do the same in Unflatten To String as well.
Best regards
Oesen

Oesen wrote:
Hi
The data type of the preprend array size in Flatten To String block is given as I32. Is it somehow possible to reduce the data type to I8, since the width and height of my array won't exceed 255 ?. I also need to do the same in Unflatten To String as well.
The short answer is no. This is because the index is an I32. NI likes to keep integers as an I32 whenever possible for reasons like this.
Since you are dealing with a 2D array (width and height), it will actually use 2 I32s before the actual data.
As Ravens stated, you could put in your own array sizes before the array in the string. But it is worth saving 6 bytes? Not likely.
There are only two ways to tell somebody thanks: Kudos and Marked Solutions
Unofficial Forum Rules and Guidelines

Word document won't 'print' to PDF on proper paper size

I have a document in MS Word that's on a 5.5x8.5 page size. If I do a Save As, the resulting PDF uses the proper paper size, but it also shows color--which I don't want for this particular document.
When I do Print and choose Adobe PDF, I can choose Black/white which is great, but I'm struggling to get it to use the right paper size. I've tried ever combination I can think of in the Default Setting and Adobe PDF Page Size options, but can't get it to show anything but 8.5x11.
Any thoughts on how to get the right paper size using Print to PDF?

Is this a Word document generated by Adobe's RoboHelp product or just a document generated directly in Word?
If the latter, you are asking in the wrong place. Even if you are using RoboHelp, this looks more like a question to ask in the Acrobat forums.
http://forums.adobe.com/community/acrobat
See www.grainge.org for RoboHelp and Authoring tips
@petergrainge

How do I determine array size in a formula node

I am feeding an array into a formula node to perform an operation on. However the size of the array can be an arbitrary size. How can I determine its size to use for iterating through a for loop?
I suppose that I could add another input that is fed with the array size from the array palate but I would like to know if it can be done inside the formula node itself.

Your own advice is the only and best one.
greetings from the Netherlands

Problem Array size in Windows 7

Hello All,
I have a VI that by opening in a new Laptop, it show some Arrays controls misfits (with a diferent array size), however i can´t put it right.
But when i make a EXE of this VI and if i run the EXE in another Computer, the front panel appears aligned and Ok.
I´m using the labview 2009 Sp1 with Windows 7, this vi have been created in another laptop in a older version and always with the Windows Xp, and whenever i was updated the version of labview it always appeared correct on those Laptop, until now when i changed the laptop and the Operating System to Win7.
Any help/advice is much apreciated
Thanks,
Solved!
Go to Solution.

Make sure you use the same font settings for all the controls that you want to be aligned. You might have some controls set to a specific font name and others left to the default Application font. LabVIEW substitutes the Application font for whatever the system default font is on a particular computer.
Different fonts are default and available on different Windows versions so you can certainly get into trouble here if you mix font families and names in the same UI.
If you request a specific font name and it is not available on a certain machine Windows will substitute it with a different font that it thinks looks similar but the heuristics of that substitution can sometimes go really way south of the equator and simply give crazy results.
Rolf Kalbermatter
CIT Engineering Netherlands
a division of Test & Measurement Solutions

Setting array size, effect on memory

Does setiing the array size to a huge no has any affect on the memory, performance and is it considered good programming. If not what are the other options?
String [] arrayString = new String [2354];

Your question is rather ambiguously worded.
In Java arrays cannot be resized, so you should set the array to the exact size you need. If you need a huge array, create one. If you need a small array, create one. If you don't know how large it should be, or need it to be dynamically sized, use any of these great classes - ArrayList, HashMap, HashSet, etc... Try reading the Java API documentation on these classes...
And yes, when you create a huge array, it actually allocates the space for each element in the array right then, and in fact initializes each with the default values of null, zero, or false depending on the type of the array.

How to adjust a "control array" size

I have a program that creates a digital PWM signal with variable duty cycle. The duty cycle changes every 50 ms, but the overall wave frequency stays at a frequency determined by the user. I have the VI attatched.
My problem is this...I need to be able to control the overall cycle time. This means the size of my array of duty cycles needs to adjust based on the cycle time (i.e. a cycle time of 1 second would require 20 of the 50 ms slots, while a 1.3 second cycle would need 26).
I currently have only 4 slots in my array, meaning the total cycle time is .2 s. I understand how to manually add and remove elements to the array, but I can't figure out how to add a control to it so that the array size changes automatically, allowing much quicker entry of data. As of now I can make it work, it just takes super long to add or delete array elements. The ideal situaton would have a constant control for cycle time divided by 50 that would change the array size.
I can't find any info on this, and I think I even saw a post asking that this kind of feature be added. I'm relatively new to this program.
Thanks
Solved!
Go to Solution.
Attachments:
var_duty_cycle.vi ‏25 KB

Currently, your VI is a one-shot deal. What you need is a state machine that updates the pct duty cycle array control as a function of the frequency while the current I/O code is idle.
I would use an array of clusters, where each cluster contains e.g. a sequence number, a duty cycle, and a string as element label. Show the scrollbar and set the size whenever the relevant inputs change. Do you want to reset the current setting when the number changes or do you want to retain the current settings as much as possible? You could just use the existing values, reshape to the larger or smaller size, and write it back to the control via a local variable. Set all elements except the percent to disabled so they act as indicators and cannot be changed by the user.
(You also need to program around it if the operator tries to manually add more elements. A better solution would be this idea, so vote for it .)
Some more general comments bout your code:
Why do you use extended precision floating point. All your waits are internally just integers. EXT makes no sense
There is a primitive for 1/x. However, you could just do a 1000/x and eliminate the multiplication afterwards. SInce you are dealing with integers, you can even do the division using quotient&remainder. Depending on the allowed frequency range there are possibly no orange data needed at all.
Make the diagram constant representation match the rest of the code.
The pulses/50ms indicator belongs before the loop. No need to recalculate and refresh it over and over from the same input values.
LabVIEW Champion . Do more with less code and in less time .

Bulk Insert to oracle DB from flat file using C#

I am using odp.net to connect to an oracle database for a C# console application. I have a fixed width file from which the values have to be inserted to multiple tables in oracle database. The data should be bulk inserted as there are lots of records and
no tools should be used. Please let me know the efficient way to do this task with C#.

Your question is how to load data from a text file into arrays?
Start with a
StreamReader to read the file one line at a time. Then break each line into individual values, and add each value to a
List<T>. Then call .ToArray() on each list to set the value of each array-bound parameter.
David
David http://blogs.msdn.com/b/dbrowne/

How to increase Array size while copying from remote table

Hi everyone,
I am using remote copy to copy 900,000(9 millions) records to local data base ,but it is taking long time obviuosly. The message i see while copying is
Array fetch/bind size is 15 (array size is 15)
can Array size be changed to make copy faster or
any otherway to copy 9 million reocrds fastly?
regards,
Karna

I guess that you are using the 'copy' command in sql*plus.
1. You could export the remote table and import the data into your local table. That perhaps would be efficient.
2. You could increase the array size for example,
SQL> show arraysize
arraysize 15
SQL> set arraysize 5000
SQL> show arraysize
arraysize 5000
SQL> What is the value for sql*plus option 'copycommit'?
SQL> show copycommit
copycommit 0
SQL>

Proper array size to be used in bulk insert

Similar Messages

Maybe you are looking for