Avoid Commit after every Insert that requires a SELECT

Hi everybody,
Here is the problem:
I have a table of generator alarms which is populated daily. On daily basis there are approximately 50,000 rows to be inserted in it.
Currently i have one month's data in it ... Approximately 900,000 rows.
here goes the main problem.
before each insert command, whole table is checked if the record does not exist already. Two columns "SiteName" and "OccuranceDate" are checked... this means, these two columns are making a unique record when checked together with an AND operation in WHERE clause.
we have also implemented partition on this table. and it is basically partitioned on the basis of OccuranceDate and each partition has 5 days' data.
say
01-Jun to 06 Jun
07-Jun to 11 Jun
12-Jun to 16 Jun
and so on
26-Jun to 30 Jun
NOW:
we have a commit command within the insertion loop, and the each row is committed once inserted, making approximately 50,000 commits daily.
Question:
Can we commit data after say each 500 inserted rows, but my real question is can we Query the records using SELECT which are Just Inserted but not yet committed ?
a friend told me that, you can query the records which are inserted in the same connection session but not yet committed.
Can any one help ?
Sorry for the long question but it was to make u understand the real issue. :(
Khalid Mehmood Awan
khalidmehmoodawan @ gmail.com
Edited by: user5394434 on Jun 30, 2009 11:28 PM

Don't worry about it - I just said that because the experts over there will help you much better. If you post your code details there they will give suggestions on optimizing it.
Doing a SELECT between every INSERT doesn't seem very natural to me, but it all depends on the details of your code.
Also, not committing on time may cause loss of the uncommitted changes. Depending on how critical the data is and the dependency of the changes, you have to commit after every INSERT, in between, or at the end.
Regards,
K.

Similar Messages

Need to commit after every 10 000 records inserted ?

What would be the best way to Commit after every 10 000 records inserted from one table to the other using the following script :
DECLARE
l_max_repa_id x_received_p.repa_id%TYPE;
l_max_rept_id x_received_p_trans.rept_id%TYPE;
BEGIN
SELECT MAX (repa_id)
INTO l_max_repa_id
FROM x_received_p
WHERE repa_modifieddate <= ADD_MONTHS (SYSDATE, -6);
SELECT MAX (rept_id)
INTO l_max_rept_id
FROM x_received_p_trans
WHERE rept_repa_id = l_max_repa_id;
INSERT INTO x_p_requests_arch
SELECT *
FROM x_p_requests
WHERE pare_repa_id <= l_max_rept_id;
DELETE FROM x__requests
WHERE pare_repa_id <= l_max_rept_id;

1006377 wrote:
we are moving between 5 and 10 million records from the one table to the other table and it takes forever.
Please could you provide me with a script just to commit after every x amount of records ? :)I concur with the other responses.
Committing every N records will slow down the process, not speed it up.
The fastest way to move your data (and 10 million rows is nothing, we do those sorts of volumes frequently ourselves) is to use a single SQL statement to do an INSERT ... SELECT ... statement (or a CREATE TABLE ... AS SELECT ... statement as appropriate).
If those SQL statements are running slowly then you need to look at what's causing the performance issue of the SELECT statement, and tackle that issue, which may be a case of simply getting the database statistics up to date, or applying a new index to a table etc. or re-writing the select statement to tackle the query in a different way.
So, deal with the cause of the performance issue, don't try and fudge your way around it, which will only create further problems.

Is it possible to automatically add a comma after every 6 characters has been typed in a text field?

Hope someone can help me.
What I basically need help with is how to make Acrobat add a comma after every 6 characters in a text field:
XXXXXX,YYYYYY, ZZZZZZ etc

I'm sorry, but I did not understand that (i'm using Acrobat Pro X)
Am I supposed to go to:
Text Field Properties > Format > Custom
and then use Custom Format Script or Custom Keystroke Script?
I tried both and it did not work.
And do the Text Field have to be named "chunkSize"?
Seems like it works. I had to move to the next formfield in order to see the effect.
Is it possible to make it happen in real time (as you type the comma is inserted?)

COMMIT after every 10000 rows

I'm getting probelms with the following procedure. Is there any that I can do to commit after every 10,000 rows of deletion? Or is there any other alternative! The DBAs are not willing to increase the undo tablespace value!
create or replace procedure delete_rows(v_days number)
is
l_sql_stmt varchar2(32767) := 'DELETE TABLE_NAME WHERE ROWID IN (SELECT ROWID FROM TABLE_NAME W
where_cond VARCHAR2(32767);
begin
   where_cond := 'DATE_THRESHOLD < (sysdate - '|| v_days ||' )) ';
   l_sql_stmt := l_sql_stmt ||where_cond;
   IF v_days IS NOT NULL THEN
       EXECUTE IMMEDIATE l_sql_stmt;
   END IF;
end;I think I can use cursors and for every 10,000 %ROWCOUNT, I can commit, but even before posting the thread, I feel i will get bounces! ;-)
Please help me out in this!
Cheers
Sarma!

Hello
In the event that you can't persuede the DBA to configure the database properly, Why not just use rownum?
SQL> CREATE TABLE dt_test_delete AS SELECT object_id, object_name, last_ddl_time FROM dba_objects;
Table created.
SQL>
SQL> select count(*) from dt_test_delete WHERE last_ddl_time < SYSDATE - 100;
COUNT(*)
     35726
SQL>
SQL> DECLARE
2
3     ln_DelSize                      NUMBER := 10000;
4     ln_DelCount                     NUMBER;
5
6 BEGIN
7
8     LOOP
9
10             DELETE
11             FROM
12                     dt_test_delete
13             WHERE
14                     last_ddl_time < SYSDATE - 100
15             AND
16                     rownum <= ln_DelSize;
17
18             ln_DelCount := SQL%ROWCOUNT;
19
20             dbms_output.put_line(ln_DelCount);
21
22             EXIT WHEN ln_DelCount = 0;
23
24             COMMIT;
25
26     END LOOP;
27
28 END;
29 /
10000
10000
10000
5726
0
PL/SQL procedure successfully completed.
SQL>HTH
David
Message was edited by:
david_tyler

Using DVD Player, the show goes back to menu after every chapter, even though I selected "Play Entire Episode".

It only does this on Reading Rainbow DVDs so far: Using DVD Player, the show goes back to menu after every chapter, even though I selected "Play Entire Episode". It does this whether the laptop is plugged in or is using battery power. How do I get it to play all?? Thanks.

Turns out it was NOT the software or the Mac. I bought a portable DVD player and this problem happens with it, too.

Commit after every three UPDATEs in CURSOR FOR loop

DB Version: 11g
I know that experts in here despise the concept of COMMITing inside loop.
But most of the UPDATEs being fired by the code below are updating around 1 million records and it is breaking our UNDO tablespace.
begin
for rec in
      (select owner,table_name,column_name
      from dba_tab_cols where column_name like 'ABCD%' and owner = p_schema_name)
      loop
        begin
        execute immediate 'update '||rec.owner||'.'||rec.table_name||' set '||rec.column_name|| ' = '''||rec.owner||'';
        end;
      end loop;
end;We are not expecting ORA-01555 error as these are just batch updates.
I was thinking of implementing something like
FOR i IN 1..myarray.count
LOOP
                         DBMS_OUTPUT.PUT_LINE('event_key at' || i || ' is: ' || myarray(i));
                         INSERT INTO emp
                         empid,
                         event_id,
                         dept,
                         event_key
                         VALUES
                         v_empid,
                         3423,
                         p_dept,
                         myarray(i)
                         if(MOD(i, p_CommitFreq) = 0) --- When the loop counter becomes fully divisible by p_commit_frequency, it COMMITs
                         then
                                   commit;
                         end if;
END LOOP;(Found from an OTN thread)
But i don't know how to access the loop counter value in a CURSOR FOR loop.

To be fair, what is really despised is code that takes an operation that could have been performed in a single SQL statement and steps through it in the slowest possible way, committing pointlessly as it goes along (exactly like the example you found). Your original version doesn't do that - it looks more like some sort of one-off migration where you have to set every value of every column that matches some naming standard pattern to a constant. If that's the case, and if there are huge volumes involved and you can't simply add a bit more undo, then I don't see much wrong with committing after each update, especially if you track how far you've got so you can restart cleanly if it fails.
If you really want an incrementing counter in an unnamed cursor, apart from the explicit variable others have suggested, you could add rownum to the cursor (alias it to something that isn't an Oracle keyword), although it could complicate the ORDER BY that you might be considering for the restart logic. You could have that instead of the redundant 'owner' column in your example which is always the same as the constant p_schema_name.
Another approach would be to keep track of SQL%ROWCOUNT after each update by adding it to a variable, and commit when the total number of rows updated so far reaches, say, a million.
Could the generated statement use a WHERE clause, or does it really have to update every row in every table it finds?
Could there be more than one column to update per table? If so it might be worth generating one multi-column update statement per table, although it'll complicate things a bit.
btw you don't need the inner 'begin' and 'end' keywords, and whoever supplied the MOD example you found should know that three or four spaces usually make a good indent and you don't need brackets around IF conditions.

Commit after every 1000 records

Hi dears ,
i have to update or insert arround 1 lakhs records every day incremental basis,
while doing it , after completing all the records commit happens, In case some problem in between all my processed records are getting rollbacked,
I need to commit it after every frequency of records say 1000 records.
Any one know how to do it??
Thanks in advance
Regards
Raja

Raja,
There is an option in the configuration of a mapping in which you can set the Commit Frequency. The Commit Frequency only applies to non-bulk mode mappings. Bulk mode mappings commit according the bulk size (which is also an configuration setting of the mapping).
When you set the Default Operating Mode to row based and Bulk Processing Code to false, Warehouse Builder uses the Commit Frequency parameter when executing the package. Warehouse Builder commits data to the database after processing the number of rows specified in this parameter.
If you set Bulk Processing Code to true, set the Commit Frequency equal to the Bulk Size. If the two values are different, Bulk Size overrides the commit frequency and Warehouse Builder implicitly performs a commit for every bulk size.
Regards,
Ilona

I keep getting reminder that I need to download the latest Flash Player after every email that I open or close. I have downloaded several times the update and it tells me that I already have this program do you want to replace it. So what is going on?

reminder for downloading the new version of flash player continues to happen. I have downloaded and installed it and I get an alert that it has already been installed and do you want to replace it. What is going on with this.

Hi Pilaar39, Please let me know if you are still having problems or have found a solution.
Thanks,
eidnolb

CA website support page provides a main heading that requires a selection from the options provided. These options do not show. Other browsers operate correctly.

This webpage does not present correctly. It is a support page which provides for a user to make a selection. These selection options do not appear. Other browsers such as Iexplorer and Google present correctly.

This webpage does not present correctly. It is a support page which provides for a user to make a selection. These selection options do not appear. Other browsers such as Iexplorer and Google present correctly.

Commit in procedures after every 100000 records possible?

Hi All,
I am using an ODI procedure to insert data into a table.
I checked that in the ODI procedure there is an option of selecting transaction and setting the commit option as 'Commit after every 1000 records'.
Since the record count to be inserted is 38489152, I would like to know is this option configurable.
Can i ensure that commits are made at a logical step of 100000 instead of 1000 records?
Thank You.
Prerna

recently added on this
http://dwteam.in/commit-interval-in-odi/
Thanks
Bhabani
http://dwteam.in

Commit after insertion

I use Oracle 10G Rel2. I'm trying to improve the performance of my database insertions. Every two days I perform a process of inserting 10000 rows in a table. I call a PL/SQL procedure
for inserting every row, which checks data and perform the insert command on the table.
Should I commit after every call to the procedure ??
Is it better to perform a commit at the end of calling 10000 times to the insertion procedure?? So the question is : is "commit" a cheap operation ??
Any idea to improve performance with this operation ??

> So the question is : is "commit" a cheap operation ??
Yes. A commit for a billion rows is as fast as a commit for a single row.
So there are no commit overheads for doing a commit on a large transaction versus a commit on a small transaction. So is this the right question to ask? The commit itself does not impact performance.
But HOW you use the commit in your code, does. Which is why the points raised by Daniel is important.. how the commit is used. In Oracle, the "best place" is at the end of the business transaction. When the business transaction is done and dusted, commit. That is after all the very purpose of the commit command - protecting the integrity of the data and the business transaction.

Back flush to ECC after every operation required..... Is it possible?

Hi,
I am having 10 operations in my Routing , need to consume inventory at operation 2, 4 and 9 (say), My Problem is that can I see the ECC inventory back flush after every operation?
Component A Inventory is 10 at ECC and SAP ME too it is 10
Example - In operation 2 I consumed one then SAP ME Inventory should be 9 and same time ECC inventory also should be 9 at the same time.
Back flush after work order/SFC is complete is ok but I need Back flush after every operation that is what is required? Is it possible?
Regards
Suhas

Hi Jay
I would think you could accomplish this by configuring some variables. You would need one variable to store a total of the others. You would configure the other variables to store a zero initially, then a one when clicked. The buttons would trigger an advanced action that would set the variable to one, then check the tally. Perhaps reveal a hidden button after all buttons had been clicked. The buttons would simply jump to different slides. You wouldn't insert buttons on the slides so the user would be forced to view them in full.
I have somewhat of an example up at the link below.
Click here to view
My example forces the user to view four slidelets in any order before allowing the next slide to be reached.
Hopefully this will help you in some way... Rick
Helpful and Handy Links
Captivate Wish Form/Bug Reporting Form
Adobe Certified Captivate Training
SorcerStone Blog
Captivate eBooks

Commit after 10000 rows

Hi
Iam inserting around 2.5 mmillion records in aconversion project
let me know how i can commit after every 10000 rows please can u tell me whether i can use bulk insert or bulk bind because i have never used please resolve my problem.
Thanks
Madhu

as sundar said, per link of TOM you are better of not commiting in th loop other wise it will give you snapshot too old error,
still if you want
1. set the counter to 0. ct number:=0;
increment counter in the loop ct:=ct+1;
IF ct=10000 THEN
COMMIT;
END IF;
2. you can use bulk collect and FORALL also. and commit.
but still follow the tread as per TOM
typo
Message was edited by:
devmiral

Commit after n rows while update

i have 1 update query
UPDATE security
SET display_security_alias = CONCAT ('BAD_ALIAS_', display_security_alias)
WHERE security_id IN (
SELECT DISTINCT security_id
FROM security_xref
WHERE security_alias_type IN ('BADSYMBOL', 'ERRORSYMBOL', 'BADCUSIP', 'ERRORCUSIP' )
i want to perform commit after every 500rows ( due to Business requirement) how can we achieve this. please help

J99 wrote:
As mentioned by karthic_arp --Not a good idea still if you want it you can do this way
DECLARE
CURSOR C
IS
SELECT DISTINCT security_id
FROM security_xref
WHERE security_alias_type IN ('BADSYMBOL', 'ERRORSYMBOL', 'BADCUSIP', 'ERRORCUSIP' );
-Counter to count records
counter number(4) default 0;
BEGIN
FOR rec in C
Loop
UPDATE security SET display_security_alias = CONCAT ('BAD_ALIAS_', display_security_alias)
WHERE security_id = rec.security_id
counter := counter +SQL%rowcount ;
If counter >= 500 then
counter := 0;
commit;
end if;
END Loop;
end ;
/NOT tested.Apart from committing every N rows, being a completely technically stupid idea, committing inside a cursor loop just take it one step further to stupiddom.

Commit after 2000 records in update statement but am not using loop

Hi
My oracle version is oracle 9i
I need to commit after every 2000 records.Currently am using the below statement without using the loop.how to do this?
do i need to use rownum?
BEGIN
UPDATE
(SELECT A.SKU,M.TO_SKU,A.TO_STORE FROM
RT_TEMP_IN_CARTON A,
CD_SKU_CONV M
WHERE
A.SKU=M.FROM_SKU AND
A.SKU<>M.TO_SKU AND
M.APPROVED_FLAG='Y')
SET SKU = TO_SKU,
     TO_STORE=(SELECT(
          DECODE(TO_STORE,
          5931,'931',
          5935,'935',
          5928,'928',
          5936,'936'))
          FROM
          RT_TEMP_IN_CARTON WHERE TO_STORE IN ('5931','5935','5928','5936'));
COMMIT;
end;
Thanks for your help

I need to commit after every 2000 recordsWhy?
Committing every n rows is not recommended....
Currently am using the below statement without using the loop.how to do this?Use a loop? (not recommended)

Avoid Commit after every Insert that requires a SELECT

Similar Messages

Maybe you are looking for