Selecting Records from 125 million record table to insert into smaller table
Oracle 11g
I have a large table of 125 million records - t3_universe. This table never gets updated or altered once loaded, but holds data that we receive from a lead company.
I need to select records from this large table that fit certain demographic criteria and insert those into a smaller table - T3_Leads - that will be updated with regard to when the lead is mailed and for other relevant information.
My question is what is the best (fastest) approach to select records from this 125 million record table to insert into the smaller table. I have tried a variety of things - views, materialized views, direct insert into smaller table...I think I am probably missing other approaches.
My current attempt has been to create a View using the query that selects the records as shown below. Then use a second query that inserts into T3_Leads from this View V_Market. This is very slow. Can I just use an Insert Into T3_Leads with this query - it did not seem to work with the WITH clause? My Index on the large table is t3_universe_composite and includes zip_code, address_key, household_key.
CREATE VIEW V_Market as
WITH got_pairs AS
SELECT /*+ INDEX_FFS(t3_universe t3_universe_composite) */ l.zip_code, l.zip_plus_4, l.p1_givenname, l.surname, l.address, l.city, l.state, l.household_key, l.hh_type as l_hh_type, l.address_key, l.narrowband_income, l.p1_ms, l.p1_gender, l.p1_exact_age, l.p1_personkey, e.hh_type as filler_data, 1.p1_seq_no, l.p2_seq_no
, ROW_NUMBER () OVER ( PARTITION BY l.address_key
ORDER BY l.hh_verification_date DESC
) AS r_num
FROM t3_universe e
JOIN t3_universe l ON
l.address_key = e.address_key
AND l.zip_code = e.zip_code
AND l.p1_gender != e.p1_gender
AND l.household_key != e.household_key
AND l.hh_verification_date >= e.hh_verification_date
SELECT *
FROM got_pairs
where l_hh_type !=1 and l_hh_type !=2 and filler_data != 1 and filler_data != 2 and zip_code in (select * from M_mansfield_02048) and p1_exact_age BETWEEN 25 and 70 and narrowband_income >= '8' and r_num = 1
Then
INSERT INTO T3_leads(zip, zip4, firstname, lastname, address, city, state, household_key, hh_type, address_key, income, relationship_status, gender, age, person_key, filler_data, p1_seq_no, p2_seq_no)
select zip_code, zip_plus_4, p1_givenname, surname, address, city, state, household_key, l_hh_type, address_key, narrowband_income, p1_ms, p1_gender, p1_exact_age, p1_personkey, filler_data, p1_seq_no, p2_seq_no
from V_Market;
I had no trouble creating the view exactly as you posted it. However, be careful here:
and zip_code in (select * from M_mansfield_02048)
You should name the column explicitly rather than select *. (do you really have separate tables for different zip codes?)
About the performance, it's hard to tell because you haven't posted anything we can use, like explain plans or traces but simply encapsulating your query into a view is not likely to make it any faster.
Depending on the size of the subset of rows you're selecting, the /*+ INDEX hint may be doing your more harm than good.
Similar Messages
-
How to select data from 3rd row of Excel to insert into Sql server table using ssis
Hi,
Iam having Excel files with headers in first two rows , i want two skip that two rows and select data from 3rd row to insert into Sql Server table using ssis.3rd row is having column names.CUSTOMER DETAILS
REGION
COL1 COL2 COL3 COL4 COL5 COL6 COL7
COL8 COL9 COL10 COL11
1 XXX yyyy zzzz
2 XXX yyyy zzzzz
3 XXX yyyy zzzzz
4 XXX yyyy zzzzz
First two rows having cells merged and with headings in excel , i want two skip the first two rows and select the data from 3rd row and insert into sql server using ssis
Set range within Excel command as per below
See
http://www.joellipman.com/articles/microsoft/sql-server/ssis/646-ssis-skip-rows-in-excel-source-file.html
Please Mark This As Answer if it solved your issue
Please Mark This As Helpful if it helps to solve your issue
Visakh
My MSDN Page
My Personal Blog
My Facebook Page -
Hi all,
I am doing pick release thru API.
For that , I have run following API successfully but my problem is that records are not inserted into MTL_TXN_REQUEST_HEADERS , MTL_TXN_REQUEST_LINES
tables so that I cannot generate the move order. Can anyone suggent the valuable tips for that ?
1. WSH_DELIVERIES_PUB.create_update_delivery
2. WSH_DELIVERY_DETAILS_PUB.Detail_to_Delivery
3. WSH_PICKING_BATCHES_PUB.CREATE_BATCH
4. WSH_PICKING_BATCHES_PUB.RELEASE_BATCH
5. WSH_DELIVERIES_PUB.Delivery_Action ( for p_action_code = 'PICK-RELEASE' )
regards with thanks in advance
sanjayHi All,
To create a Move Order Header & Line Record in MTL_TXN_REQUEST_HEADERS and MTL_TXN_REQUEST_LINES tables , I have run an seeded
procedure : CreateMoveOrderHeader & CreateMoveOrderLines .
The procedure : CreateMoveOrderHeader was run successfully but when I am going to run procedure : CreateMoveOrderLines , it is giving
an error .
Following Error text I am receiving at the time of running of procedure : CreateMoveOrderLines .
Initialized applications context: 5707 50138 660
*==========================================================*
Calling INV_MOVE_ORDER_PUB.Create_Move_Order_Lines API
*==========================================================*
Return Status: E
Error Message :INV
*==========================================================*
From the above error message, no specific error is received . Only " Error Message : INV " is received hence it is very difficult to judge & resolve an error .
Can anyone guide me that what is an error ? For what reason I am getting an error ?
regrads
Sanjay -
Record is not inserting into the Table thourgh forms 10g
Hi All,
I have the form built in 10g(10.1.2.0.2).
Basically the form has 2 blocks.
Block 1 with only one item, where we enter some value and hit enter(this will navigate to block2 and execute the query).
Block 2(tabular) will fetch the records. Now this block2 have 3 columns(caseid, userid, date).
Now when I insert a new record, I just need to enter the caseid only. And userid and date have to be automatically populated.
I can populate the USERID and DATE fields.
And when I enter some value in caseid item of block 2, and then do Control+S(to insert the record and Save the transaction),
i get the message saying FRM-40400: Transaction complete: 1 records applied and saved.
But when I query again for the same, I dont se the record inserted into the table.
Why is this happening?
Help please...@ Inol
There is no promary or foriegn key relation ship. The form is fetching records from just one table. As I said block1 has col1. and Block2 has col2,3,4.
@Andreas
Yes I did select * from table, select count(*) from table in SQL*Plus. Nothing is inserted.
And one thing I have to tall here is, I have a ON-INSERT trigger in block2.
The code in ON-INSERT trigger is
-- :BLOCK2.DATE := TO_DATE(:DATE,'DD-MON-YYYY') ;
/* commented since I was populating date by assigning $$DATETIME$$ to the Initial value property of the data item. */
:BLOCK2.SOURCE_CASE := :BLOCK1.SOURCE_CASEID;Forgot to tell you,
Since I was inserting the record from block2(3 columns), to the table which has 4 columns, there is another item in block 2 which has the visible property No.
So the block 2 has 4 columns. And the hidden column will hold the value that is in the item of block1. This is what i'm pushing in the ON-INSERT trigger.
Edited by: Charan on Sep 20, 2011 1:51 PM -
How to select data from a PL/SQL table
Hi,
I am selecting data from database after doing some screening i want to store it in a PL/SQL table (temporary area) and pass it to oracle reports.
Is there any way to select the data from a PL/SQL table as a cursor. Or is there any other way of holding the temporary data and then pass it back as a cursor.
Regards
KamalA PL/SQL "table" is anything but a table. Whoever came up with this term in PL/SQL to describe what is known as dynamic arrays (the correct programming terminology that existed since the 70's if not earlier and what is used in all other programming languages I'm familiar with)... well, several descriptions come to mind and none of them are complimentary.
You cannot "select" from a PL/SQL dynamic array as it is not a table within the Oracle context of tables.
Thus you need to convert (cast) a PL/SQL dynamic array into a temporary Oracle data set/table in order to select from it. This is in general a Bad Idea (tm). Oracle tables and SQL and concurrency controls and all that are especially designed for processing data. PL/SQL arrays is a very simplistic data structure with very limited usage. Why would you want to use that in SQL via a SELECT statement when you can use Oracle tables (or proper temp tables) instead? Besides that, it is also slow to cast a dynamic PL/SQL array into an Oracle SQL data set structure (context switching, copying of memory, etc).
The proper way to use PL/SQL to generate data sets for use via the SQL engine is pipelined table functions.
This is not to say that you should never use PL/SQL arrays and casting in SQL.. simply that you need to make sure that this is the correct and scalable way to do it. And that will also always be an exception to the rule when you do. -
Select data from 4 or more tables
Hi All,
I got stUck up in a problem in Oracle.
I have 4 tables say t1,t2,t3,t4
t1 has c1 is a primary key,c2,c3
t2 has c1 is a foreign key ,c4,c5
t3 has c1 is a foreign key,c6,c7
t4 has c1 is a foreign key,c8,c9
I want to retreive the data from all these tables at a time.i.e. c1,c2,c3,c4,c5,c6,c7,c8,c9 such that
it doesnot result in duplicacy od records.
that is the record should be grouped if it has same value
FOR EXAMPLE
a query
"select company.company_name,import_from,export_to from company,importfrom,exportto
where company.company_name=exportto.company_name
and
company.company_name=importfrom.company_name"
will retreive
COMPANY_NAME iMPORT_FROM EXPORT_TO
IMG CANADA INDIA
IMG CANADA ITALY
IMG CANADA BRAZIL
IMG CANADA CHINA
IMG CANADA SOUTH AFRICA
IMG MEXICO INDIA
IMG MEXICO ITALY
IMG MEXICO BRAZIL
IMG MEXICO CHINA
IMG MEXICO SOUTH AFRICA
AND I WANT
COMPANY_NAME iMPORT_FROM EXPORT_TO
IMG null INDIA
IMG null ITALY
IMG NULL BRAZIL
IMG NULL CHINA
IMG NULL SOUTH AFRICA
IMG CANADA NULL
IMG MEXICO NULL
PLEASE REPLY SOON
ManojTry this query.
select c.company_name, e.exp, null
from company c, export_to e
where c.company_name = e.company_name
union
select c.company_name, null, i.imp
from company c, import_from i
where c.company_name = i.company_name
If you have 4 tables, you'll need 3 queries with union statement. -
Delete from 95 million rows table ...
Hi folks, need to delete from a 95 millions rows regular table, what should be my best options, have tried CTAS using parallel, but it failed after 1+ hrs ... it was due to bad query, but checking is there any other way to achieve this.
Thanks in advance.user8604530 wrote:
Hi folks, need to delete from a 95 millions rows regular table, what should be my best options, have tried CTAS using parallel, but it failed after 1+ hrs ... it was due to bad query, but checking is there any other way to achieve this.
Thanks in advance.how many rows in the table BEFORE the DELETE?
how many rows in the table AFTER the DELETE?
How do I ask a question on the forums?
SQL and PL/SQL FAQ
Handle: user8604530
Status Level: Newbie
Registered: Mar 10, 2010
Total Posts: 64
Total Questions: 26 (22 unresolved)
I extend to you my condolences since you rarely get your questions answered. -
How to select data from all HR Master Tables(PA0000 to PA****)
Hi Gurus!
My requirement is to pick the changedon(AEDTM) , changedby(UNAME) and reason for change(PREAS) from all tables (pa0000 to pa****)of HR in which he made any change during a specific date.
Now if it is specific for two or three tables then inner join can solve it. but now it is the matter of picking data from maximum 9999 tables for which inner join will not be accurate.
plz suggest me any appropriate way to solve it out. or there is any table which store the changes in All HR Master Tables.
Thanking in Advance.
Tahir NaqqashHi,
it is very tidious to get it in one go, as u know hr tables contains huge data.
You can chack my wiki in https://wiki.sdn.sap.com/wiki/x/XYC0Aw .
Here you need to make the following modification.
1. in the selection screen set on date range with select option.
2. in the select query, change select * by ur specified fields.
3. change dynamic internal table and ALV as per ur required fields
In this way you have to put the table name and the date range and will get the data.
for repeation u can call the same job in background again and again store data in a seperate internal table and display finaly ..... but huge chance of getting 'Time Out ' error in that case for comparetively large day gap.
Regards,
Anirban -
Record not getting inserted into R/3 table.
Hi Experts,
I am trying to save some information in R/3 using BAPI. I have written codes like as shown below:
Code in View
public void onActionInsertRecord(com.sap.tc.webdynpro.progmodel.api.IWDCustomEvent wdEvent )
//@@begin onActionInsertRecord(ServerEvent)
try { wdThis.wdGetTimesheetCompController().executeBapi_Catimesheetmgr_Insert_Input();
wdThis.wdGetTimesheetCompController().executeBapi_Transaction_Commit_Input();
} catch (Exception ex)
{wdComponentAPI.getMessageManager().reportException(ex.getMessage(), false);
//@@end
Code in Component Controller
public void executeBapi_Catimesheetmgr_Insert_Input( )
//@@begin executeBapi_Catimesheetmgr_Insert_Input()
IWDMessageManager manager = wdComponentAPI.getMessageManager();
try
{ Bapicats1 xx = new Bapicats1();
xx.setWbs_Element(wdContext.currentWorklistElement().getRec_Wbs());
xx.setEmployeenumber(wdContext.currentZbapi_Cat_Stech00_Getdetail_InputElement().getEmployeenumber());
xx.setCatshours(new BigDecimal ("1"));
xx.setWorkdate(wdContext.currentOutputElement().getDate_From());
wdContext.currentBapi_Catimesheetmgr_Insert_InputElement().setProfile(wdContext.currentZbapi_Cat_Stech00_Getdetail_InputElement().getProfile());
wdContext.currentBapi_Catimesheetmgr_Insert_InputElement().modelObject().execute();
wdContext.nodeOutputnew().invalidate();
catch(WDDynamicRFCExecuteException e)
manager.reportException(e.getMessage(), false);
//@@end
My records are not getting saved in R/3 tables. Can you please let me know what is going wrong here. The BAPI input parameters values are correctly getting populated in Component Controller.
I have also created error message table in View. This error message table is mapped to output of Bapi_Catimesheetmgr_Insert. This table is not getting populated.
Due to this reason, I believe that the values of input parameters of BAPI are not getting passed to R/3. However, I am not sure where the problem lies. Can you please help me in resolving the issue?
Regards,
BrianHi Sumit,
I am not getting any exception. Using SE37, I can save the transaction. Can you please let me know what is going wrong here.
Regards,
Sumit -
Hi,
I have a requirement as follows
I need to have a report (say in page1 on table1) with the checkboxes in order to select the required records.
When I click on next button, I should be able to display these selected records only to the next page(say page2) .
Page2 has submit button and when I click on submit these selected records should be inserted into a table(say table2).
I am able to create a report with checkboxes but not understanding how to collect these selected records to the next page and insert them into database table.
Kindly advice me how this can be achieved.
Thanks in Advance.
Sarvani.Sarvani,
here is an example on how to select checked rows:
http://htmldb.oracle.com/pls/otn/f?p=31517:95
You should use an array to collect the picked records - item with concatenated primary
keys. After that, you may use the example here:
http://htmldb.oracle.com/pls/otn/f?p=31517:84
to insert those records into a table.
Denes Kubicek
http://deneskubicek.blogspot.com/
http://htmldb.oracle.com/pls/otn/f?p=31517:1
------------------------------------------------------------------- -
How can I find out the place from where a record is being inserted in table
Hi,
I want to find out the exact code statement from the SAP code from where record is being inserted into a SAP table. I tried ST05 SQL trace but it seems to only give the SQL statement and not the program from where it is being fired (unless you guys can show me a way to find it from the trace)
I m trying to see that when sales order is changed in VA02, from which piece of code in SAP the tables CDHDR and CDPOS get update. Also I tried using SE30 but it doesnt seem to give the results properly. Please help.
Thanks for readingHello Srikrishna,
Did you try the following?
In your trace with ST05:
i) Find the line that you are interested, right-click and select 'Display ABAP Source'.
OR
ii) Go to Trace List menu and click 'Summarize Trace by SQL Statement'. Then here, select the line you want and again right click to see 'Display ABAP Source'.
Regards,
Guven. -
Deleting selected record in Dynamic Internal table
Hi Friends,
I want to delete selected row from a dynamic internal table. The selected entry should be deleted from Adobe and as well as Webdynpro abap.
Kindly help me how to solve this problem.
Thanks in advance.
Regards,
Phani.Hi Matthias,
Thanks a lot for responding.
In my adobe i kept a check box to delete the record. It should delete the selected ones. But right now, its deleting one by one.
I have coded following script for delete button:
IT_TIME_SHEET --> is my internal table
G_ROW_STATUS is the field to track the action performed.
var tlength = xfa.resolveNodes("IT_TIME_SHEET.DATA[*]").length;
for ( var i=0; i<tlength; i++
if(xfa.resolveNode("IT_TIME_SHEET.DATA["i"].FLAG").rawValue ==
1 )
IT_TIME_SHEET.DATA.instanceManager.removeInstance(i);
G_ROW_STATUS.rawValue = "DELETE";
In the Webdynpro abap following code I wrote:
row_count = 1.
FIELD-SYMBOLS <wa> TYPE zshr_time_sheet_time.
LOOP AT it_time_sheet ASSIGNING <wa>.
<wa>-srl_no = row_count.
row_count = row_count + 1.
ENDLOOP.
last_row = <wa>-srl_no.
DELETE it_time_sheet WHERE srl_no EQ last_row.
Suppose if two records checkboxes are selected I can loop at the internal table IT_TIME_SHEET in webdynpro and delete where checkbox is selected.
But problem is once user deletes records, adobe is deleting only one record and if i delete two records from webdynpro it looks like something wrong for the user.
Please help me how to solve this problem.
Regards,
Phani. -
Record not inserting into the table through Forms 10g
Hi all,
I have created a form in 10g(10.1.2.0.2) based on just one table that has 4 columns(col1, col2, col3, col4).
Here col1, col2 and col3 are VARCHAR2 and col4 is date and all the columns are not null columns(There are no primary and foriegn key constrains, which means duplicates are allowed).
My form contains 2 blocks where block 1 has one text item (col1) and 3 buttons (Delete, Save, Exit).
And block2 is a database block and has col2,col3,col4 which are in tabluar layout frame displaying 10 records.
When the form is opened the cursor has to be in block1.col1 for querrying. Here i enter a value in col1, and then when I click on col2 in the block2, then I put execute_query in new_block_instance of block2, which displays the records.
The block2 properties are not updatable, insertable and query is allowed.
Everything is working good until here. But here in the block2 when I want to insert another record into the table, by navigating all the way down to the last empty record and entering the new values for col2, col3 and col4 And then Ctrl+S will display the message "*FRM-40400: Transaction complete: 1 record applied and saved.*" But actually the record is not inserted into the table.
I also disabled the col4 by setting the Enabled property to No, since while inserting new record the date have to be populated into it and it shouldnt be changed by the user. And im populating the sysdate into the new record by setting Intial Value property to *$$DATE$$*.
And another requirement which I could not work arround here is that, the col3 also should be populated with the username of the user while inserting.
please help me...Hi Sarah,
I do not want to update the existing record. So I kept Udate Allowed to No in property palette for the items in block2.
Do I have to do this property at block level also?
I'm inserting a new record here.
Edited by: Charan on Sep 19, 2011 8:48 AM -
Record not inserting into sap table with connector framework ?
here is the code, but record not being inserting into the table ... but same piece of code working fine while updating ... the record ...
try {
interaction = connection.createInteractionEx();
IInteractionSpec interactionSpec = interaction.getInteractionSpec();
String functionName = "Z_XYZ";
interactionSpec.setPropertyValue("Name", functionName);
String writingTable = "MYTABLE";
RecordFactory rf = interaction.getRecordFactory();
MappedRecord importParams = rf.createMappedRecord("input");
importParams.put("ATTR1", "VALUE1");
importParams.put("ATTR2", "VALUE2");
IFunction function = connection.getFunctionsMetaData().getFunction(functionName);
IStructureFactory sf = interaction.retrieveStructureFactory();
IRecordSet table = (IRecordSet) sf.getStructure(function.getParameter(writingTable).getStructure());
table.insertRow();
table.setString("ATNAME", "VALUE");
table.setString("ATWRT", "VALUE");
importParams.put(writingTable, table);
MappedRecord output = (MappedRecord) interaction.execute(interactionSpec, importParams);
} catch (Exception e) {
any idea?
than ks
MMKHi Mohan,
Does a creation through SE37 with the same input work?
Yoav. -
How Select data from four tables
Hi Sdners,
How select data from four or more tables please give me any suggestion. iam learner please help me
Advance wishes
sreenu
Moderator Message: I've already locked 2 threads of yours. Please read the Rules of Engagement of this forum and abide by it.
Edited by: kishan P on Sep 5, 2010 9:12 PMI think there was a sample of using CURSOR in the XSQL docs.Check section:
Using the CURSOR Operator for Nested Rowsets
null
Maybe you are looking for
-
Simple button to play symbol....?
I have a symbol "NA". It is an animated map that scales up and then back in place when closed. I have added a button to the main stage to play this symbol with the following code on the "click" event: sym.getSymbol("NA").play(); What am I missing? So
-
Help: consuming an external web service with user name token
Hello Together! I need to consume an external web service secured with WS-UserNameToken. The way, how did I do it: 1. I generated a web service consumer (proxy) in SE80 from the wsdl file 2. I created logical port for the consumer in SOAMANAGER 3. I
-
How to upload a PDF file in iCloud
Hi everybody, I would like to know how to upload a PDF file in iCloud. I want to share some PDF files with my iPhone, MacBook and iMac. Thank you a lot!
-
Reviewing Slides On Wrong Answer?
When in questions is there a way of reviewing previous slides when a wrong answer is given; for example: A standard presentation of 50 slides is viewed and then candidates are taken to the question slides If question 1 is answered correctly it moves
-
Connecting iPhone 3G to Macbook with dead battery
Battery died. Using Magcord for now. iPhone won't connect to download music. Is a new battery necessary to do that??