Inserting millions of records into new table based on condition
Hi All,
We have a range partitioned table that contains 950,000,000 records (since from 2004) which is again list sub-partitioned on status. Possible values of stauts are 0,1,2,3 and 4.
The requirement is to get all the rows with status 1 and date less than 24-Aug 2011. (Oracle 11g R2).
I trying below code
CREATE TABLE RECONCILIATION_TAB PARALLEL 3 NOLOGGING
AS SELECT /*+ INDEX(CARDS_TAB STATUS_IDX) */ ID,STATUS,DATE_D
FROM CARDS_TAB
WHERE DATE_D < TO_DATE('24-AUG-2011','DD-MON-YYYY')
AND STATUS=1; CARDS_TAB has tow global indexes one on status and another on date_d.
Above query is running for last 28Hrs! Is this the right approach?
With Regards,
Farooq Abdulla
You said the table was range partitioned but you didn't say by what. I'm guessing the table is range partitioned by DATE_D. Is that a valid assumption?
You said that the table was subpartitioned by status. If the table is subpartitioned by status, what do you mean that the data is randomly distributed? Surely it's confined to particular subpartitions, right?
What is the query plan without the hint?
What is the query plan with the hint?
Why do you believe that adding the hint will be beneficial?
Justin
Similar Messages
-
Best way to insert millions of records into the table
Hi,
Performance point of view, I am looking for the suggestion to choose best way to insert millions of records into the table.
Also guide me How to implement in easier way to make better performance.
Thanks,
Orahar.Orahar wrote:
Its Distributed data. No. of clients and N no. of Transaction data fetching from the database based on the different conditions and insert into another transaction table which is like batch process.Sounds contradictory.
If the source data is already in the database, it is centralised.
In that case you ideally do not want the overhead of shipping that data to a client, the client processing it, and the client shipping the results back to the database to be stored (inserted).
It is must faster and more scalable for the client to instruct the database (via a stored proc or package) what to do, and that code (running on the database) to process the data.
For a stored proc, the same principle applies. It is faster for it to instruct the SQL engine what to do (via an INSERT..SELECT statement), then pulling the data from the SQL engine using a cursor fetch loop, and then pushing that data again to the SQL engine using an insert statement.
An INSERT..SELECT can also be done as a direct path insert. This introduces some limitations, but is faster than a normal insert.
If the data processing is too complex for an INSERT..SELECT, then pulling the data into PL/SQL, processing it there, and pushing it back into the database is the next best option. This should be done using bulk processing though in order to optimise the data transfer process between the PL/SQL and SQL engines.
Other performance considerations are the constraints on the insert table, the triggers, the indexes and so on. Make sure that data integrity is guaranteed (e.g. via PKs and FKs), and optimal (e.g. FKs should be indexes on the referenced table). Using triggers - well, that may not be the best approach (like for exampling using a trigger to assign a sequence value when it can be faster done in the insert SQL itself). Personally, I avoid using triggers - I rather have that code residing in a PL/SQL API for manipulating data in that table.
The type of table also plays a role. Make sure that the decision about the table structure, hashed, indexed, partitioned, etc, is the optimal one for the data structure that is to reside in that table. -
How can i use BAPI to insert a few records into standard table
Can anyone help me with how can i use BAPI to insert some records into a standard table from an internal table?
Hi,
First of All try to Explain your Question first.
This is a general question without mentioning the Table you want to Update.
Please give the details before posting a question so it will help people to understand your Problem.
Regards
Sandipan -
INSERT of two records into different tables (pk value from first to second)
Hi there!
Have probably stupid question
Need to insert one record into table with primary key and then insert into other table record with value of primary key field from first record
How can I do it?
Thanks a lot!!!You have several possibilities. Most easiest one is listed first :)
SQL> create table a (a number);
Table created.
SQL> alter table a add constraint a_pk primary key (a);
Table altered.
SQL> create table b (a number);
Table created.
SQL> alter table b add constraint b_a_fk foreign key (a) references a(a);
Table altered.
SQL> insert into a values (0);
1 row created.
SQL> insert into b values (0);
1 row created.Though that may not help always, so the next possibility maybe just using sequence with nextval and currval (currval can be used only in the same session and only after you have issued at least one nextval)
SQL> create sequence a_seq;
Sequence created.
SQL> insert into a values (a_seq.nextval);
1 row created.
SQL> insert into b values (a_seq.currval);
1 row created.And you can use also famous returning clause. It is a bit easier to show that in the pl/sql block than pure SQL.
SQL> declare
2 v number;
3 begin
4 insert into a values (a_seq.nextval) returning a into v;
5 insert into b values (v);
6 end;
7 /
PL/SQL procedure successfully completed.And at last contents of the tables :)
SQL> select * from b;
A
0
1
2
SQL> select * from a;
A
0
1
2Gints Plivna
http://www.gplivna.eu -
How can i use BAPI to insert a few records into standard table usobt_c
I needed to compare the records of this table on two different systems and update the records that were in system A but not in system B into system B. Please give your inputs.
Hi,
First of All try to Explain your Question first.
This is a general question without mentioning the Table you want to Update.
Please give the details before posting a question so it will help people to understand your Problem.
Regards
Sandipan -
Extracting from table based on conditions from two internal tables
Hi,
i to have select few records from a table based on conditions from two different internal tables. How can I achieve this.?
ex:
select objid from HRVPAD25 into table t_pad25
where PLVAR = 01
OTYPE = E
OBJID = itab1-sobid
sobid = itab2-pernr.
How can this be written? can i use "for all entries..." addition with 2 tables?Hi Maansi_SAP,
you can use exactly one internal table in the FOR ALL ENTRIES clause. Consider this alternative:
data:
itab_sobid_hash like itab1 with unique key sobid,
ls_pad25 like line of t_pad25.
sort itab1.
delete adjacend duplicates from itab1 comparing sobid.
itab_sobid_hash = itab1.
select objid
into ls_pad25
from HRVPAD25
for all entries in itab2
where PLVAR = '01'
and OTYPE = E
and sobid = itab2-pernr..
read table itab_sobid_hash with table key sobid = ls_pad25-objid.
check sy-subrc = 0.
append ls_pad25 to t_pad25.
endselect.
You may decide if itab1 or itab2 is better used as hashed table. Make a performance test.
The critics will tell you that SELECT ... ENDSELECT is not performant. That was very true, back in last milleniums's 90ies
Regards,
Clemens -
Inserting new records into database table at runtime
Hi all ,
How to insert new records into database table at runtime on click update?
Thanks.Hi Sasikala,
Just for your understanding am giving a sample code snippet which you can use to read the contents of your Table UI element & save the data on to your database. Suppose you have a button up on pressing which you want to read the data from your screens table & save on to the database then you can proceed as shown below:
1) Obtain the reference of your context node.
2) Fetch all the data present in your table into an internal table using methods of if_wd_context_node
3) Use your normal ABAP logic to update the database table with the data from your internal table
In my example I have a node by name SFLIGHT_NODE and under this I have the desired attributes from SFLIGHT. Am displaying these in an editable table & the user would press up on a push button after making the necessary changes to the tables data. I would then need to obtain the tables information & save on to the database.
data: node_sflight type ref to if_wd_context_node,
elem_sflight type ref to if_wd_context_element,
lt_elements type WDR_CONTEXT_ELEMENT_SET,
stru_sflight type if_main=>element_sflight_node,
it_flights type if_main=>elements_sflight_node.
" navigate from <CONTEXT> to <SFLIGHT_NODE> via lead selection
node_sflight_node = wd_context->get_child_node( name = 'SFLIGHT_NODE' ).
lt_elements = node_sflight->get_elements( ).
" Get all the rows from the table for saving on to the database
loop at lt_elements into elem_sflight.
elem_sflight->get_static_attributes( importing static_attributes = stru_sflight ).
append stru_sflight to it_flights.
endloop.
" Finally save the entries on to the database
modify ZSFLIGHT99 from table it_flights.
if sy-subrc eq 0.
endif.
However a word of caution here.... SAP doesn't ever recommend directly modifying the database through an SQL query. You would preferably make use of a BAPI for the same. Try go through Thomas Jung's comments in [here|modify the data base table which is comming dynamiclly;.
Regards,
Uday -
How to insert new record into oracle table from GridView in VS2005.
I need to insert records into Oracle10g table from VS2005 GridView using the Insert Method in Business Logic Class. The Update and Delete Methods are working fine. What's the best way out?
How is it "not possible"?
Either modify the Class for the new fields, or give the new fields default values (if applicable). If the type of an existing column has changed, then only the first option is available.
Where's the problem? -
Insert Records into another tables
Hi guyz,
I have four tables as below
1) employee
empid (primary key)
deptid
emp_name
work_location
date_of_join
emp_status
date
2) Termination
empid reference employee
deptid
emp_name
work_location
terminate_date
3) Resignation
empid reference employee
deptid
emp_name
work_location
resign_date
4) Vacation
empid reference employee
deptid
emp_name
work_location
vac_date
i have list item on employee form --> emp_status is list item whic contain Termination, Resignation, Vacation
i want to update employee as well other tables if user select the respected field from the list item.
if the user select termination/resignation from the list item the date item will be enable and user should enter the date,otherwise give the error on commit please enter the date and when user commit the record on employee it should update the table employee and delete the record belongs to Termination/Reisgnation and the record will be insert into Termination/Resignation table as select by user from list item.i tried this before but im fail i write the below code on pre-update trigger as below
begin
insert into termination
(empid,deptid,emp_name,work_location,termination_date)select (empid,deptid,emp_name,work_location,date from employee where emp_status='Termination';
delete from employee where emp_status='Termination' ;
end;
same as on resignation
anybuddy help me how to do this task? to update the tables. im new in oracle.
Regards
Edited by: user10648713 on Dec 21, 2008 4:29 AMYou run into the classic Mutating table problem. You cannot issue DML-statement against the same table the trigger is based on in a FOR EACH ROW-trigger.
But even worse, you will cause any kind of client program to get in trouble if you implement such a logic. The client issues an update and as a result the record gets deleted. And, for you have a forms-module based i tell you that you will get problems with your forms-module.
I still can't see whats the sense behind it, putting records into different tables dependant on a status. Oracle is built to manage large data volumes, so even for a large number of records that doesn't make sense to me.
But anyway if you still want to deal with that multi-table-approach, go for the following:
You will need three triggers on your table: a before-statement, an after-statement and a for-each-row-trigger:
The before-statement will initialize a list of PK-Items
The for-each-row will add the id of the processed row to the list
The after-statement will loop over the list and process the rows.
Create a Package for handling the logic including three procedures
CREATE OR REPLACE PACKAGE PK_TRIGGER IS
PROCEDURE PR_BS;
PROCEDURE PR_ARIU(i_nEmpId IN NUMBER);
PROCEDURE PR_AS;
END;
CREATE OR REPLACE PACKAGE BODY IS
TYPE tData IS TABLE OF NUMBER INDEX BY BINARY_INTEGER;
lData tData;
PROCEDURE PR_BS IS
BEGIN
lData.DELETE;
END;
PROCEDURE PR_ARIU(i_nEmpId IN NUMBER) IS
iPos PLS_INTEGER:=lData.COUNT+1;
BEGIN
lData(iPos):=i_nEmpId;
END;
PROCEDURE PR_AS IS
CURSOR crEmp(i_nEmpId IN NUMBER) IS
SELECT *
FROM EMP
WHERE EMP_ID=i_nEmpId;
recEmp crEmp%ROWTYPE;
iPos PLS_INTEGER;
BEGIN
iPos:=lData.FIRST;
LOOP
EXIT WHEN iPos IS NULL;
-- Do whatever you want with the data in the record
OPEN crEmp(lData(iPos));
FETCH crEmp INTO recEmp;
IF crEmp%FOUND THEN
IF :NEW.EMP_STATUS='Termination' THEN
DELETE FROM emp
WHERE EMPID=lData(iPos);
INSERT INTO TERMINATION (
empid,
deptid,
emp_name,
work_location,
t_date
) VALUES (
recEmp.empid,
recEmp.deptid,
recEmp.emp_name,
recEmp.work_location,
sysdate
END IF;
END IF;
CLOSE crEmp;
-- Process next record
iPos:=lIds.NEXT(iPos);
END LOOP;
lIds.DELETE;
END;
END;Now create the three triggers calling the appropiate procedures. -
Inserting records into a table with all caps
Hello
I have a procedure that inserts records into a table. How do I ensure that the text values inserted are recorded all capital letters into the table?
Thanks.You can use UPPER(..) function in your insert statement, so that values are converted to UPPER, before insert.
If you want to check at table level, you can achieve that by writting a before insert trigger and in that trigger check
IF UPPER(:new.<col>) != :new.<col> THEN
RAISE_APPLICATION_ERROR(-20101,'Error: Not all values are in upper case')
END IF; -
To add new record into the table Data Service client
When I am trying to add new record into the table Employeedetails using Data Service client it is giving a sql exception: "java.sql.SQLException: Violation of PRIMARY KEY constraint 'PK__EmployeeDetails__6383C8BA'. Cannot insert duplicate key in object 'EmployeeDetails'. Severity 14, State 1, Procedure 'PC-P41403 null', Line 1."
Code:
DataService ds=DataServiceFactory.newDataService(getInitialContext(),"EmplDetApp","ld:EmplDetAppDataServices/EmployeeDetails");
EmployeeDetailsDocument edoc=EmployeeDetailsDocument.Factory.newInstance();
edoc.addNewEmployeeDetails();
//here I set the primary key value empid
edet.setEmpid("1212");
edet.setEmpname("manu");
ds.submit(edoc);
Error
EmpName 5 java.sql.SQLException: Violation of PRIMARY KEY constraint 'PK__EmployeeDetails__6383C8BA'. Cannot insert duplicate key in object 'EmployeeDetails'. Severity 14, State 1, Procedure 'PC-P41403 null', Line 1
com.bea.ld.dsmediator.DataServiceException: java.sql.SQLException: Violation of PRIMARY KEY constraint 'PK__EmployeeDetails__6383C8BA'. Cannot insert duplicate key in object 'EmployeeDetails'. Severity 14, State 1, Procedure 'PC-P41403 null', Line 1
at com.bea.ld.dsmediator.update.JDBCAdaptor.save(JDBCAdaptor.java:247)
at com.bea.ld.dsmediator.update.DataServiceMediator.submit(DataServiceMediator.java:528)
at com.bea.ld.dsmediator.update.DataServiceMediator.submit(DataServiceMediator.java:245)
at com.bea.ld.ServerBean.submit(ServerBean.java:529)
at com.bea.ld.Server_ydm4ie_EOImpl.submit(Server_ydm4ie_EOImpl.java:910)
at com.bea.ld.Server_ydm4ie_EOImpl_WLSkel.invoke(Unknown Source)
at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:492)
at weblogic.rmi.cluster.ReplicaAwareServerRef.invoke(ReplicaAwareServerRef.java:108)
at weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:435)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:147)
at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:430)
at weblogic.rmi.internal.BasicExecuteRequest.execute(BasicExecuteRequest.java:35)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)
at java.lang.Thread.startThreadFromVM(Unknown Source)
Caused by: java.sql.SQLException: Violation of PRIMARY KEY constraint 'PK__EmployeeDetails__6383C8BA'. Cannot insert duplicate key in object 'EmployeeDetails'. Severity 14, State 1, Procedure 'PC-P41403 null', Line 1
at weblogic.jdbc.mssqlserver4.TdsStatement.processWarning(TdsStatement.java:1178)
at weblogic.jdbc.mssqlserver4.TdsStatement.parseMsWarning(TdsStatement.java:1089)
at weblogic.jdbc.mssqlserver4.TdsStatement.getMoreResults(TdsStatement.java:756)
at weblogic.jdbc.mssqlserver4.TdsStatement.execute(TdsStatement.java:210)
at weblogic.jdbc.mssqlserver4.TdsStatement.executeUpdate(TdsStatement.java:97)
at weblogic.jdbc.mssqlserver4.TdsStatement.executeUpdate(TdsStatement.java:1455)
at weblogic.jdbc.wrapper.PreparedStatement.executeUpdate(PreparedStatement.java:147)
at com.bea.ld.dsmediator.update.JDBCAdaptor.save(JDBCAdaptor.java:151)
... 15 more
But I am not sure why it is giving an exception as Violation of PRIMARY KEY.
The update method works fine for the same client.
The exception only happens when I am trying to insert a new record.
Please help me to figure out this problem.If you are absolutely sure that you do not have such a row already in your table, open a case with customer support and reference CR321312. I believe the work-around is to put ld-server-core.jar in your client classpath.
Correction: put ld-server-app.jar in the client classpath
Edited by mreiche at 09/17/2007 3:28 PM -
Split records into two files based on lookup table
Hi,
I'm new to ODI and want to know on how I could split records into two files based on a value in one of the columns in the table.
Example:
Table:
my columns are
account name country
100 USA
200 USA
300 UK
200 AUS
So from the 4 records I maintain list of countries in a lookup file and split the records into 2 different files based on values in the file...
Say I have records AUS and UK in my lookup file...
So my ODI routine should send all records with country into file1 and rest to file2.
So from above records
File1:
300 UK
200 AUS
File2:
100 USA
200 USA
Can you help me how to achieve this?
Thanks,
Sam1. where and how do i create filter to restrict countries? In source or target? Should I include some kind of filter operator in interface.
You need to have the Filter on the Source side so that we can filter records accordingly the capture the same in the File. To have a Filter . In the source data store click and drag the column outside the data store and you will have Cone shaped icon and now you can click and type the Filter.
Please look into this link for ODI Documentation -http://www.oracle.com/technetwork/middleware/data-integrator/documentation/index.html
Also look into this Getting started guide - http://download.oracle.com/docs/cd/E15985_01/doc.10136/getstart/GSETL.pdf . You can find information as how to create Filter in this guide.
2. If I have include multipe countries like (USA,CANADA,UK) to go to one file and rest to another file; Can I use some kind of lookup file...? Instead of modifying filter inside interface...Can i Update entries in the file?
there are two ways of handling your situation.
Solution 1.
1. Create Variable Country_Variable
2. Create a Filter in the Source datastore in the First Interface ( SOURCE.COLUMN = #Country_Variable)
3. Create a new Package Country File Unload
4. Call the Variable in Country_Variable in Set Mode and provide the Country (USA )
5. Next call the First Interface
6. Next call the Second Interface where the Filter condition will be ( SOURCE.COLUMN ! = #Country_Variable )
7. Now run the package .
Solution 2.
If you need a solution to handle through Filer.
1. Use this Method (http://odiexperts.com/how-to-refresh-odi-variables-from-file-%E2%80%93-part-1-%E2%80%93-just-one-value ) to call the File where you wish to create store the country name into the variable Country_Variable
2. Pretty much the same Create a Filter in the Source datastore in the First Interface ( SOURCE.COLUMN = #Country_Variable)
3.Create a new Package Country File Unload
4.Next call the Second Interface where the Filter condition will be ( SOURCE.COLUMN ! = #Country_Variable )
5. Now run the package .
Now through this way using File you can control the File.
Please try and let us know , if you need any other help. -
Can BO Enterprize SDK inserts records into user table
From Infostore can we create a jsp script using Java SDK to inserts records into user table??
Thanks
AmarHi Amar,
I want to retrieve data/records from Infostore and insert into a user table using JSP script. Is it possible to do this?
Infostore is a database used by BO Server. so any changes made in infostore through BO enterprise session is valid.
Say u want to retrive on of report present in folder <my folder>.
The you have to query for that. for eg.
boinfostore.query("select * from ci_infoobjects where si_kind ='report' and si_foldername='my folder'");
Create/add/insert any new information in infostore is done by functionalities provide by SDK.
like adding the user or scheduling a report will add new object to infostore.
If you directly access cms database and make any changes then , I am afraid you will end up with nightmare.
So it is always recommneded to access infostore/ cms database only from bo session.
For more information refer below link
[http://devlibrary.businessobjects.com/BusinessObjectsXIR2SP2/en/devsuite.htm]
then under Contents
BusinessObjects Enterprise SDK >> COM developer guide and API reference >> Query Language Reference
do revert if any queries
Thanks,
Praveen. -
Insert multiple records into a table(Oracle 9i) from a single PHP statement
How can I insert multiple records into a table(Oracle 9i) from a single PHP statement?
From what all I've found, the statement below would work if I were using MySQL:
insert into scen
(indx,share,expire,pitch,curve,surface,call)
values
(81202, 28, 171, .27, 0, 0, 'C' ),
(81204, 28, 501, .25, 0, 0, 'C' ),
(81203, 17, 35, .222, 0, 0, 'C' ),
(81202, 28, 171, .27, 2, 0, 'C' ),
(81204, 28, 501, .20, 0, 1, 'C' ),
(81203, 28, 135, .22, 1, 0, 'C' )
The amount of records varies into the multiple-dozens. My aim is to utilize the power of Oracle while avoiding the i/o of dozens of single-record inserts.
Thank you,
WillYou could look at the INSERT ALL statement found in the documentation here:
http://download.oracle.com/docs/cd/B10501_01/server.920/a96540/statements_913a.htm#2133161
My personal opinion is that you probably won't see any benefit because under the hood I think Oracle will still be doing single row inserts. I could be wrong though.
The only way to confirm was if you did a test of multiple inserts vs an INSERT ALL, that is if the INSERT ALL met your requirements.
HTH. -
Using Crystal 2008 to insert records into a table
Hi,
We have a unique need to use Crystal to insert records into a table. We have managed to test a report that can write into a temporary table. This is done by using sql command object and uses the following code :
INSERT INTO TEMP_TABLE (ORDERID)
VALUES ({?orderid}) (-- where orderid a parameter).
This test report asks for an order id and then inserts the record perfectly fine.
Now moving on to the real report - This report basically prints orders in batches and we want to insert order id into a temporary table to ensure we don't print orders that were already printed. To do this we created a sub report "insert orders" that has the above insert command. The main report passes the orderid to subreport and the idea is that the subreport would insert each time an order is passed. So if main report printed 50 orders ids, the then it would do 50 inserts individually into the temp table.
This however is NOT working. The report runs fine but there is no insert. Our hunch is that Crystal is not committing after every order id is passed from the main report. Not sure if we can set the AUTO COMMIT ON as a default somewhere?
Wondering if any one has attempted this or has any insights?
Regards,
Mohit.
Environment is - Crystal 2008 and Oracle 11GR2, we are using Oracle drivers (and not odbc)Hmmm... I don't use Oracle but the syntax looks good...
You've already tested it and I assume that you are using the same driver in the production report as you used in the test, so that shouldn't be an issue...
how are you pulling the data? Is the final SELECT statement that pulls the report data in the same command as the INSERT script, or is the INSERT script in it's own command?
The reason I ask... If you are trying to pass a multi-valued parameter to a command, it won't work. If you have the insert command as it's own command while the data is being pulled with linked tables or a separate command, it is possible that the report itself will execute as expected w/o passing a value to the insert script.
If it's all in 1 command (as it should be), a bad parameter would fail in the final SELECT causing an error.
Also... are rows null or empty string values being added to table when the report executes? This would be an indication that the command is being executed but isn't getting the parameter value.
Jason
Maybe you are looking for
-
My itunes movie I just rented is not showing up in my purchased list. Where do i find it?
I have rented a movie in itunes and it is not showing in my purchased list.
-
Shell we update ios 6.1.4 to ios 7
Hello, This is vijay and am using Iphone 5 with IOS 6.1.4. Now i want to update my IOS from 6.1.4 to IOS 7. Please help me how to update my phone by step by step.
-
Why can I only text 10 people with the IPhone 4s
Why can I only text 10 people at a time with the IPhone 4s?
-
Toggle layout in the virtual console
How can I set a keyboard shortcut to change a layout from one to another in the virtual console?
-
when the belle will receive next update after belle fp 2