How to speed insert my 1000000 records into the database?

my code like:
<cfloop from="1" to="#inserteddb.getrecordcount()#"
index="x">

<cfquery datasource="#cfdsn#" name="insertdata">
insert into inputtest (#InsertFieldList#)
values (
<cfqueryparam value="#InsertValueList#"
cfsqltype="cf_sql_varchar" list="yes">
</cfquery>
</cfloop>
The test inserts 100,000 records, has spend I 30 minutes
time,but I have 1,000,000 record to insert , is there any way to
enhance the insertion speed?
Thanks a lot.

By removing ColdFusion from the process as much as possible.
Where is the 'insertedDB' data coming from? It looks to be a
record set?
Are you moving data from one data source to another? If so,
some DBMS
have the ability to insert an entire record set in one step.
I do not
have the exact syntax at my finger tips, but I have done
something like
this in the past with Oracle. INSERT INTO aTable SELECT FROM
bTable.
Are you building a record set from a text file such as CSV?
If so, many
DBMS have the ability to do 'bulk' inserts from such text
files and CF
does not even need to be involved.
As you can see, knowing exactly what are you working with
will help us
provide suggestions an how to improve your process.

Similar Messages

Insert multiple rows of records into the database

The codes below allow me to insert a row of record into the database. How would I changed these to insert multiple rows at once? Please help!
String sql = "INSERT INTO EMPLOYEES" +
"(First_Name, Last_Name, Title, Phone) " +
" VALUES " +
PreparedStatement statement = conn.prepareStatement(sql);
statement.setObject (1, First_Name);
statement.setObject (2, Last_Name);
statement.setObject (3, Title);
statement.setObject (4, Phone);
boolean returnValue = statement.execute();

Hi mystiqueX,
As wmolosho has suggested in his answer to this very same question that you also posted to the JavaServer Pages forum, you can create a batch of inserts and perform them using the "executeBatch()" method. I will use Craig's sample code to demonstrate:
(Note that this code is untested!)
conn.setAutoCommit(false);
PreparedStatement statement = conn.prepareStatement(sql);
// assume you have an array of objects here
for (int i = 0; i < data.length; i++) {
statement.setString(1, data<i>.getFirstName());
statement.setString(2, data<i>.getLastName());
statement.setString(3, data<i>.getTitle());
statement.setString(4, data<i>.getPhone());
statement.addBatch();
statement.executeBatch();
conn.commit();If you are not familiar with it, allow me to suggest looking at the Making Batch Updates lesson on the Java Tutorial.
Hope it helps.
Good Luck,
Avi.

Best way to insert millions of records into the table

Hi,
Performance point of view, I am looking for the suggestion to choose best way to insert millions of records into the table.
Also guide me How to implement in easier way to make better performance.
Thanks,
Orahar.

Orahar wrote:
Its Distributed data. No. of clients and N no. of Transaction data fetching from the database based on the different conditions and insert into another transaction table which is like batch process.Sounds contradictory.
If the source data is already in the database, it is centralised.
In that case you ideally do not want the overhead of shipping that data to a client, the client processing it, and the client shipping the results back to the database to be stored (inserted).
It is must faster and more scalable for the client to instruct the database (via a stored proc or package) what to do, and that code (running on the database) to process the data.
For a stored proc, the same principle applies. It is faster for it to instruct the SQL engine what to do (via an INSERT..SELECT statement), then pulling the data from the SQL engine using a cursor fetch loop, and then pushing that data again to the SQL engine using an insert statement.
An INSERT..SELECT can also be done as a direct path insert. This introduces some limitations, but is faster than a normal insert.
If the data processing is too complex for an INSERT..SELECT, then pulling the data into PL/SQL, processing it there, and pushing it back into the database is the next best option. This should be done using bulk processing though in order to optimise the data transfer process between the PL/SQL and SQL engines.
Other performance considerations are the constraints on the insert table, the triggers, the indexes and so on. Make sure that data integrity is guaranteed (e.g. via PKs and FKs), and optimal (e.g. FKs should be indexes on the referenced table). Using triggers - well, that may not be the best approach (like for exampling using a trigger to assign a sequence value when it can be faster done in the insert SQL itself). Personally, I avoid using triggers - I rather have that code residing in a PL/SQL API for manipulating data in that table.
The type of table also plays a role. Make sure that the decision about the table structure, hashed, indexed, partitioned, etc, is the optimal one for the data structure that is to reside in that table.

How do I insert a video clip into the timeline?

Do you happen to know how I could insert a video clip into the middle of a movie like you do in FCP? Can you add tracks and insert the clip above the spot i need to change for each movie.
I don't want to render the whole movie (convert it into a MPeg2) each time.

No - DVD timelines are a little different to FCP. If you need to add a clip to the stream, you need to split the exisitng clip (you could trim it back to where you want to insert the new piece) and drag the new piece onto the same timeline. If you trimmed the existing piece then add another instance of it to the timeline and trim the new part from the left until you get to what you want as the out point.
If you have a collection of video clips on your DVDSP timeline, simply dragging a new one on will shuffle the others along appropriately. However, the audio will not shuffle with them, and you'll end up moving it all manually.
The very best thing is to create the timeline precisely as you want it in FCP and encode it to MPEG2 before bringing it in to DVDSP. Remember that DVDSP is not an editor, it is a place to assemble your work - it won't do anything remotely like FCP in that respect.

Inserting new input record into the table through screen

Hi,
Example UserTable and PaymentTable
UserTable - UserID - Primary Key
PaymentTable - containg couple of fields and UserId is foreign Key.
I designed the page for to insert the new input record using the steps given in the doc "http://download-uk.oracle.com/docs/cd/B32110_01/web.1013/b28967/web_adv007.htm".
When i click the persistEntity method its throws TopLink error cannot insert null value into UserId column in PaymentTable.
Currently i am posting this reply from home... that's why i mentioning the scenario rather than posting the error.
Please tell how to populate the values from the screen to PaymentTable.
Thanks & Regards
Vimalan Balan

Hi Vimalan,
Did you find an answer for you question?
Regards,
Phil

How To Insert List of Record To the Database Using Linq

Good day every one, Please i need little help, I have a list of record like this
lblID.Text = string.Join("<br/>", numbers);
gives the Numbers below...
6131
5241
3978
6824
3842//To Save to the database
int pin = int.parse(lblID.Text);
Now i want to insert those Numbers to the database
Persons ps = new Persons();
ps.Number = pin;
cdc.Persons.AddObject(ps);
cdc.SaveChanges();
it throws up error...
what i want to achieve on button insert
all Numbers will be inserted but before inserting the number it will check if those numbers exist first on the database
it should be like this on the database
ID Number
1 6131
2 5241
3 3978
4 6824
5 3842
thanks.. I do appreciate your time and effort

Hello,
>>Now i want to insert those Numbers to the database
For inserting a list of items to database, a loop statement is needed. I do not how you get these numbers, I assume they are stored in a collection and you could check below demo sample to insert these numbers and I assume your person table is identified:
using (DFDBEntities db = new DFDBEntities())
List<int> numbers = new List<int>() { 6131, 5241, 3978, 6824, 3842 };
for (int i = 0; i < numbers.Count; i++)
int number = numbers[i];
//Check if the local and database already contains this number
bool isRecordExist = db.Tables.Any(t => t.Number == number) || db.Tables.Local.Any(t => t.Number == number);
if (!isRecordExist)
Table t = new Table();
t.Number = number;
db.Tables.Add(t);
db.SaveChanges();
Regards.
We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
Click
HERE to participate the survey.

Insert a new record in the database table in between the records.

i va a database table which ve 100 records. but i want to insert my new record as 50th record. how i want to proceed?
thanks ,
velu.

V,
This is an odd request. Why?
Ignoring that, you can ATTEMPT to insert into the 50th position IF:
1) The DB table has just had the primary key index re-built/re-shuffled to GUARANTEE that it IS in primary key order
2) And the primary key of the new record is built so that it follows the 49th record
Regardless, once this table has activity against it, its sort order can/will get out of promary key order (with adds/changes/deletes).
But when you select data from it, you can use ORDER BY or an ABAP SORT to organzie the date as needed.
Again... not sure WHY you need a record in a particular physical position... who cares... it is a relational DB.

How to read/insert MS .xls record into Oracle without .csv conversion etc.

i really have this challenge to download .xls file at oracle level. In other word no apache utility like Sheet etc. need to use. Any thin at oracle level is permitted though. E.g. stored procedure/ function etc.
File is as such in .xls format. Not text or .csv or any other delimitered.
Thanks

Can you explain exactly why these limitations are in place? That would really help us figure out whether any of the options we might be able to suggest would work for you. A few options off the top of my head:
- There are various Java libraries that can parse at least some reasonable subset of Excel formatting. I'd expect that you could probably load one of those into Oracle and write a Java stored procedure that would parse the file. Of course, I'm not sure whether that falls into your "at Oracle level" bucket or into your "no Apache utility" bucket.
- Oracle APEX has various Excel upload utilities, but that requires the use of APEX which might not be trivial to integrate with an existing application.
- Heterogenous Services with Generic Connectivity would allow you to create a database link over ODBC to an Excel file and to query the Excel data via SQL. That requires, though, that there is an ODBC DSN for the Excel file and probably requires that your Oracle database is on Windows.
Justin

Somehow ejbCreate method of an Entity EJB is not inserting a record into the database. I'm using BMP and calling the Entity bean method from a servlet. I'm using NAS 4.0 sp5 on Win 2k.

Also if I call the insert query from within the servlet it works fine. I have disabled global transactions. Am I missing something out ? Please
any help would be greatly appreciated.

May be your servlet is not able to lookup the EJB. You can use some print statements and see whether lookup ok or not.
Just a guess.
Plese get back, if any queries.
Thanks,
Rakesh.

Can you use ESB to insert multiple records into a Database

We have an XML file that has a Parent/Child relationship.
In situation 1 we have a single parent record in the xml file. We can insert the record into the database successfully.
In situation 2 we have a parent and a an associated child record. By using a filter expression to identify the existence of the child record we can route to the first DB Adapter to insert the parent then to the second DB Adapter to insert the child.
Is this final scenario logical and also possible.
Situation 3 we have multiple child records associated to a single parent. So we would first insert the parent record then insert all the child records.
Thanks

I use PL/SQL for this if you are using Oracle.
Have a look at this post, it is for AQ but the concept can be used for PL/SQL
Re: Can I pass a pl/sql table as a parameter from a database adaptor
I will send you an example
cheers
James

How to insert a specific record into alv

Hi everyone,
here is my problem:
    I put an ALV GRID control in my screen , it display some records.
    and I create a new button on the toolbar, and in the "handle user command class" , I need to
select a record from DB table into a work area.
    after I select the record I need, here is the key question:
      *I want to insert the record into the ALV at a specific row position which the user decided,
what should I do?*
METHOD HANDLE_ON_USER_COMMAND.
    CASE E_UCOMM.
      WHEN CL_GUI_ALV_GRID=>MC_FC_LOC_INSERT_ROW.
      WHEN 'FC_ASSIGN'.
***I do some selection here
        select %%%$$%%%^&&** into GS_HOLIDAYS.
*** after the selection, what should I do then????
*** I want to insert the work area GS_HOLIDAYS into ALV at a specific position
*** e.g. into the 3rd row.
*** how can I achieve that????
***call a method or something??????I don't know
      WHEN 'FC_DELETE'.
      WHEN OTHERS.
    ENDCASE.
ENDMETHOD.
pls don't let me go through the programs in package "SLIS",because I have already done that and
I haven't solved my problems yet.
Thanks for your help.

All u need to do is on user command for inserting new records u just insert a blank record in your internal table that u are displaying with required style informaion for making it editable and then refresh alv display by method REFRESH_TABLE_DISPLAY.
Thanks & Regards,
Vivek Gaur

How do I insert a session variable into a record?

I can’t figure out how to insert the value passed by a
session variable into a record.
I have a form I want to use to insert a record. I pass
session variables to this page and want to insert one of them into
the record. The session information is:
session_start();
$_SESSION['usename'] = $_POST['username'];
$idvar = $_SESSION['id'];
$_SESSION[‘id’] is the user_id that I want to
insert into the record.
I know $idvar is getting to the page with the form because I
can echo it.
The insert code looks like this:
if ((isset($_POST["MM_insert"])) &&
($_POST["MM_insert"] == "register")) {
$insertSQL = sprintf("INSERT INTO h_genres (user_id,
username, contrib_to, Advertising, Annual_Reports, Leader_Content,
Member_Content, Brochures) VALUES (%s, %s, %s, %s, %s, %s, %s,
%s)",
GetSQLValueString($_POST['user_id'], "int"),
GetSQLValueString($_POST['username'], "text"),
GetSQLValueString($_POST['contrib_to'], "text"),
GetSQLValueString(isset($_POST['Advertising']) ? "true" :
"", "defined","'Y'","'N'"),
GetSQLValueString(isset($_POST['Annual_Reports']) ? "true" :
"", "defined","'Y'","'N'"),
GetSQLValueString(isset($_POST['Leader_Content']) ? "true" :
"", "defined","'Y'","'N'"),
GetSQLValueString(isset($_POST['Member_Content']) ? "true" :
"", "defined","'Y'","'N'"),
GetSQLValueString(isset($_POST['Brochures']) ? "true" : "",
"defined","'Y'","'N'"));
mysql_select_db($database_hauw, $hauw);
$Result1 = mysql_query($insertSQL, $hauw) or
die(mysql_error());
I have a hidden field for the user_id:
<input name="user_id" type="hidden" id="user_id" />
I think I need to get the value in $idvar into user_id but I
don’t know how.
Any help is appreciated.

awsweb wrote:
> I have a hidden field for the user_id:
> <input name="user_id" type="hidden" id="user_id"
/>
>
> I think I need to get the value in $idvar into user_id
but I don?t know how.
<input name="user_id" type="hidden" id="user_id"
value="<?php echo
$idvar; ?>" />
David Powers
Adobe Community Expert
Author, "Foundation PHP for Dreamweaver 8" (friends of ED)
http://foundationphp.com/

Different methods involved in insertion of records into the table

hi all,
i am using Forms [32 Bit] Version 6.0.8.24.1 (Production)
Oracle Database 10g Release 10.2.0.1.0 - Production
i have parameters like emp_code(optional) and year(mandatory field-four digit year).
Based on the parameter i have to get the details from one table and those records should be inserted into other table along with additions columns(by hardcoding the additional columns).
To acheive this i am planning to get a cursor by selecting the necessay fields from the table passing the parameters in the where clause of the cursor.
by looping through the cursor i will insert the records into the other table.
here the thing is the data is going to be huge (since year is the parameter).
so is my approach leads to the performance issue.
i heard that there are different method of insetions,
can anybody suggest me the better way(peformance wise) other than what i have mentioned (if you are aware of)..
Thanks..

This is not really a Forms issue, so you'd better go to the sql and pl/sql forum. Here, this question has been asked many times: how do I insert large volumes of data in a table?
insert into ...
select ....
or CTAS (create table as select ....)
by looping through the cursor i will insert the records into the other table.That is really the worst idea. If you can do it in sql (insert into ... select ...) DON'T do it in pl/sql.

How do I insert/update certain records that I've identified in cursors..?

Morning gurus..
As the title asks, I have this script that loads records into a temp table then compares these with whats in the ORACLE table. When each temp table record has been identified as being either needing to be inserted or not, how do I then insert these identified records? As in, how can I do..
INSERT INTO xxxxxxxxxxx_SUPPLIERS (vendor_id, vendor_name,segment1, end_date_active) VALUES (2, 'SUPPLIER NAME 2', 100002, sysdate); ..but I'll need to put in some variable names where the values have been written in above - is this allowed..?
thanks,
E

hi,
here is a small sample for your query.... given Procedure InsertExcep is reading data from table code_master and inserting all fetched records into another table test_code_master..... before insert you can do any comparison within loop..... or even can give any Update statement....
CREATE TABLE code_master
CODE VARCHAR2(10),
CODE_DESC VARCHAR2(70)
INSERT INTO CODE_MASTER ( CODE, CODE_DESC ) VALUES (
'ORG0007', 'PARTNERSHIP FIRM');
INSERT INTO CODE_MASTER ( CODE, CODE_DESC ) VALUES (
'ORG0008', 'CONSULTANCY FIRM OR PROFESSIONAL BODY');
INSERT INTO CODE_MASTER ( CODE, CODE_DESC ) VALUES (
'ORG0009', 'SOCIAL ORGANIZATION');
INSERT INTO CODE_MASTER ( CODE, CODE_DESC ) VALUES (
'ORG0010', 'EMBASSY');
CREATE TABLE test_code_master
CODE VARCHAR2(10),
CODE_DESC VARCHAR2(70)
CREATE OR REPLACE PROCEDURE InsertExcep
AS
CURSOR insertdata
IS
SELECT code,code_desc
FROM code_master ;
BEGIN
     FOR i IN insertdata
     LOOP
               BEGIN
               INSERT INTO test_code_master (code,code_desc)
               VALUES(i.code,i.code_desc);
               EXCEPTION WHEN OTHERS THEN
               DBMS_OUTPUT.PUT_LINE ( 'Error during Insert code - '||i.code );
               END;
     END LOOP;
EXCEPTION WHEN OTHERS THEN
DBMS_OUTPUT.PUT_LINE ( 'Error in InsertExcep' );
END;
hope this will work...
Thanx
MB

How do I insert my Edge Animation into Wordpress as an animated/interactive graphic

How do I insert my Edge Animation into Wordpress as an animated/interactive graphic?

Hi ColleenSyron,
Can you check out this video: http://tv.adobe.com/watch/create-like-crazy-with-adobe-edge/episode-5-spice-up-your-wordpr ess-site-with-edge-animate/
Thanks,
Preran

How to speed insert my 1000000 records into the database?

Similar Messages

Maybe you are looking for