Bulk collect and varrays
Oracle 9.2.0.6
anyway to bulk collect into a varray where the index is part what is returned by the bulk collect?
declare
type tv is varray(5) of int;
tvList tv := tv(0,0,0,0,0);
begin
select l,v bulk collect into tvList
from
(select 1 as l,2 as v from dual
union
select 2,3 from dual);
end;
also tried:
declare
type tv is varray(5) of int;
tvList tv := tv(0,0,0,0,0);
begin
select l,v bulk collect into tvList(l)
from
(select 1 as l,2 as v from dual
union
select 2,3 from dual);
end;
neither worked.
Thanks
You can't do that; bulk collect only copies into consecutive entries, indexed by an integer.
If you use varrays, all the return values must fit in the varray's declared size. Elements are inserted starting at index 1, overwriting any existing elements.
See http://download-uk.oracle.com/docs/cd/B19306_01/appdev.102/b14261/tuning.htm#sthref2213
The workaround is to
- bilk collect into an associative array indexed by binary/pls integer
- after the bulk collect, copy the data into your varray
- do this in batches using BULK COLLECT ... LIMIT ,,,
Sounds painful, but the bulk collect can really save time on context switching between PL/SQL and SQL.
HTH
Regards Nigel
Similar Messages
-
Using bulk collect and for all to solve a problem
Hi All
I have a following problem.
Please forgive me if its a stupid question :-) im learning.
1: Data in a staging table xx_staging_table
2: two Target table t1, t2 where some columns from xx_staging_table are inserted into
Some of the columns from the staging table data are checked for valid entries and then some columns from that row will be loaded into the two target tables.
The two target tables use different set of columns from the staging table
When I had a thousand records there was no problem with a direct insert but it seems we will now have half a million records.
This has slowed down the process considerably.
My question is
Can I use the bulk collect and for all functionality to get specific columns from a staging table, then validate the row using those columns
and then use a bulk insert to load the data into a specific table.?
So code would be like
get_staging_data cursor will have all the columns i need from the staging table
cursor get_staging_data
is select * from xx_staging_table (about 500000) records
Use bulk collect to load about 10000 or so records into a plsql table
and then do a bulk insert like this
CREATE TABLE t1 AS SELECT * FROM all_objects WHERE 1 = 2;
CREATE OR REPLACE PROCEDURE test_proc (p_array_size IN PLS_INTEGER DEFAULT 100)
IS
TYPE ARRAY IS TABLE OF all_objects%ROWTYPE;
l_data ARRAY;
CURSOR c IS SELECT * FROM all_objects;
BEGIN
OPEN c;
LOOP
FETCH c BULK COLLECT INTO l_data LIMIT p_array_size;
FORALL i IN 1..l_data.COUNT
INSERT INTO t1 VALUES l_data(i);
EXIT WHEN c%NOTFOUND;
END LOOP;
CLOSE c;
END test_proc;
In the above example t1 and the cursor have the same number of columns
In my case the columns in the cursor loop are a small subset of the columns of table t1
so can i use a forall to load that subset into the table t1? How does that work?
Thanks
Juser7348303 wrote:
checking if the value is valid and theres also some conditional processing rules ( such as if the value is a certain value no inserts are needed)
which are a little more complex than I can put in a simpleWell, if the processing is too complex (and conditional) to be done in SQL, then doing that in PL/SQL is justified... but will be slower as you are now introducing an additional layer. Data now needs to travel between the SQL layer and PL/SQL layer. This is slower.
PL/SQL is inherently serialised - and this also effects performance and scalability. PL/SQL cannot be parallelised by Oracle in an automated fashion. SQL processes can.
To put in in simple terms. You create PL/SQL procedure Foo that processes SQL cursor and you execute that proc. Oracle cannot run multiple parallel copies of Foo. It perhaps can parallelise that SQL cursor that Foo uses - but not Foo itself.
However, if Foo is called by the SQL engine it can run in parallel - as the SQL process calling Foo is running in parallel. So if you make Foo a pipeline table function (written in PL/SQL), and you design and code it as a thread-safe/parallel enabled function, it can be callled and used and executed in parallel, by the SQL engine.
So moving your PL/SQL code into a parallel enabled pipeline function written in PL/SQL, and using that function via parallel SQL, can increase performance over running that same basic PL/SQL processing as a serialised process.
This is of course assuming that the processing that needs to be done using PL/SQL code, can be designed and coded for parallel processing in this fashion. -
How to use Bulk Collect and Forall
Hi all,
We are on Oracle 10g. I have a requirement to read from table A and then for each record in table A, find matching rows in table B and then write the identified information in table B to the target table (table C). In the past, I had used two ‘cursor for loops’ to achieve that. To make the new procedure, more efficient, I would like to learn to use ‘bulk collect’ and ‘forall’.
Here is what I have so far:
DECLARE
TYPE employee_array IS TABLE OF EMPLOYEES%ROWTYPE;
employee_data employee_array;
TYPE job_history_array IS TABLE OF JOB_HISTORY%ROWTYPE;
Job_history_data job_history_array;
BatchSize CONSTANT POSITIVE := 5;
-- Read from File A
CURSOR c_get_employees IS
SELECT Employee_id,
first_name,
last_name,
hire_date,
job_id
FROM EMPLOYEES;
-- Read from File B based on employee ID in File A
CURSOR c_get_job_history (p_employee_id number) IS
select start_date,
end_date,
job_id,
department_id
FROM JOB_HISTORY
WHERE employee_id = p_employee_id;
BEGIN
OPEN c_get_employees;
LOOP
FETCH c_get_employees BULK COLLECT INTO employee_data.employee_id.LAST,
employee_data.first_name.LAST,
employee_data.last_name.LAST,
employee_data.hire_date.LAST,
employee_data.job_id.LAST
LIMIT BatchSize;
FORALL i in 1.. employee_data.COUNT
Open c_get_job_history (employee_data(i).employee_id);
FETCH c_get_job_history BULKCOLLECT INTO job_history_array LIMIT BatchSize;
FORALL k in 1.. Job_history_data.COUNT LOOP
-- insert into FILE C
INSERT INTO MY_TEST(employee_id, first_name, last_name, hire_date, job_id)
values (job_history_array(k).employee_id, job_history_array(k).first_name,
job_history_array(k).last_name, job_history_array(k).hire_date,
job_history_array(k).job_id);
EXIT WHEN job_ history_data.count < BatchSize
END LOOP;
CLOSE c_get_job_history;
EXIT WHEN employee_data.COUNT < BatchSize;
END LOOP;
COMMIT;
CLOSE c_get_employees;
END;
When I run this script, I get
[Error] Execution (47: 17): ORA-06550: line 47, column 17:
PLS-00103: Encountered the symbol "OPEN" when expecting one of the following:
. ( * @ % & - + / at mod remainder rem select update with
<an exponent (**)> delete insert || execute multiset save
merge
ORA-06550: line 48, column 17:
PLS-00103: Encountered the symbol "FETCH" when expecting one of the following:
begin function package pragma procedure subtype type use
<an identifier> <a double-quoted delimited-identifier> form
current cursorWhat is the best way to code this? Once, I learn how to do this, I apply the knowledge to the real application in which file A would have around 200 rows and file B would have hundreds of thousands of rows.
Thank you for your guidance,
SeyedHello BlueShadow,
Following your advice, I modified a stored procedure that initially was using two cursor for loops to read from tables A and B to write to table C to use instead something like your suggestion listed below:
INSERT INTO tableC
SELECT …
FROM tableA JOIN tableB on (join condition).I tried this change on a procedure writing to tableC with keys disabled. I will try this against the real table that has primary key and indexes and report the result later.
Thank you very much,
Seyed -
BULK COLLECT and FORALL with dynamic INSERT.
Hello,
I want to apply BULK COLLECT and FORALL feature for a insert statement in my procedure for performance improvements as it has to insert a huge amount of data.
But the problem is that the insert statement gets generated dynamically and even the table name is found at the run-time ... so i am not able to apply the performance tuning concepts.
See below the code
PROCEDURE STP_MES_INSERT_GLOBAL_TO_MAIN
(P_IN_SRC_TABLE_NAME VARCHAR2 ,
P_IN_TRG_TABLE_NAME VARCHAR2 ,
P_IN_ED_TRIG_ALARM_ID NUMBER ,
P_IN_ED_CATG_ID NUMBER ,
P_IN_IS_PIECEID_ALARM IN CHAR,
P_IN_IS_LAST_RECORD IN CHAR
IS
V_START_DATA_ID NUMBER;
V_STOP_DATA_ID NUMBER;
V_FROM_DATA_ID NUMBER;
V_TO_DATA_ID NUMBER;
V_MAX_REC_IN_LOOP NUMBER := 30000;
V_QRY1 VARCHAR2(32767);
BEGIN
EXECUTE IMMEDIATE 'SELECT MIN(ED_DATA_ID), MAX(ED_DATA_ID) FROM '|| P_IN_SRC_TABLE_NAME INTO V_START_DATA_ID , V_STOP_DATA_ID;
--DBMS_OUTPUT.PUT_LINE('ORIGINAL START ID := '||V_START_DATA_ID ||' ORIGINAL STOP ID := ' || V_STOP_DATA_ID);
V_FROM_DATA_ID := V_START_DATA_ID ;
IF (V_STOP_DATA_ID - V_START_DATA_ID ) > V_MAX_REC_IN_LOOP THEN
V_TO_DATA_ID := V_START_DATA_ID + V_MAX_REC_IN_LOOP;
ELSE
V_TO_DATA_ID := V_STOP_DATA_ID;
END IF;
LOOP
BEGIN
LOOP
V_QRY1 := ' INSERT INTO '||P_IN_TRG_TABLE_NAME||
' SELECT * FROM '||P_IN_SRC_TABLE_NAME ||
' WHERE ED_DATA_ID BETWEEN ' || V_FROM_DATA_ID ||' AND ' || V_TO_DATA_ID;
EXECUTE IMMEDIATE V_QRY1;
commit;
V_FROM_DATA_ID := V_TO_DATA_ID + 1;
IF ( V_STOP_DATA_ID - V_TO_DATA_ID > V_MAX_REC_IN_LOOP ) THEN
V_TO_DATA_ID := V_TO_DATA_ID + V_MAX_REC_IN_LOOP;
ELSE
V_TO_DATA_ID := V_TO_DATA_ID + (V_STOP_DATA_ID - V_TO_DATA_ID);
END IF;
EXCEPTION
WHEN OTHERS THEN.............
....................so on Now you can observer here that P_IN_SRC_TABLE_NAME is the source table name which we get as a parameter at run-time. I have used 2 table in the insert statement P_IN_TRG_TABLE_NAME (in which i have to insert data) and P_IN_SRC_TABLE_NAME(from where i have to insert data)
V_QRY1 := ' INSERT INTO '||P_IN_TRG_TABLE_NAME||
' SELECT * FROM '||P_IN_SRC_TABLE_NAME ||
' WHERE ED_DATA_ID BETWEEN ' || V_FROM_DATA_ID ||' AND ' || V_TO_DATA_ID;
EXECUTE IMMEDIATE V_QRY1;now when i appy the bulk collect and forall feature i am facing the out of scope problem....see the code below ...
BEGIN
EXECUTE IMMEDIATE 'SELECT MIN(ED_DATA_ID), MAX(ED_DATA_ID) FROM '|| P_IN_SRC_TABLE_NAME INTO V_START_DATA_ID , V_STOP_DATA_ID;
--DBMS_OUTPUT.PUT_LINE('ORIGINAL START ID := '||V_START_DATA_ID ||' ORIGINAL STOP ID := ' || V_STOP_DATA_ID);
V_FROM_DATA_ID := V_START_DATA_ID ;
IF (V_STOP_DATA_ID - V_START_DATA_ID ) > V_MAX_REC_IN_LOOP THEN
V_TO_DATA_ID := V_START_DATA_ID + V_MAX_REC_IN_LOOP;
ELSE
V_TO_DATA_ID := V_STOP_DATA_ID;
END IF;
LOOP
DECLARE
TYPE TRG_TABLE_TYPE IS TABLE OF P_IN_SRC_TABLE_NAME%ROWTYPE;
V_TRG_TABLE_TYPE TRG_TABLE_TYPE;
CURSOR TRG_TAB_CUR IS
SELECT * FROM P_IN_SRC_TABLE_NAME
WHERE ED_DATA_ID BETWEEN V_FROM_DATA_ID AND V_TO_DATA_ID;
V_QRY1 varchar2(32767);
BEGIN
OPEN TRG_TAB_CUR;
LOOP
FETCH TRG_TAB_CUR BULK COLLECT INTO V_TRG_TABLE_TYPE LIMIT 30000;
FORALL I IN 1..V_TRG_TABLE_TYPE.COUNT
V_QRY1 := ' INSERT INTO '||P_IN_TRG_TABLE_NAME||' VALUES V_TRG_TABLE_TYPE(I);'
EXECUTE IMMEDIATE V_QRY1;
EXIT WHEN TRG_TAB_CUR%NOTFOUND;
END LOOP;
CLOSE TRG_TAB_CUR;
V_FROM_DATA_ID := V_TO_DATA_ID + 1;
IF ( V_STOP_DATA_ID - V_TO_DATA_ID > V_MAX_REC_IN_LOOP ) THEN
V_TO_DATA_ID := V_TO_DATA_ID + V_MAX_REC_IN_LOOP;
ELSE
V_TO_DATA_ID := V_TO_DATA_ID + (V_STOP_DATA_ID - V_TO_DATA_ID);
END IF;
EXCEPTION
WHEN OTHERS THEN.........so on
But the above code is not helping me , what i am doing wrong ??? how can i tune this dynamically generated statement to use bulk collect for better performace ......
Thanks in Advance !!!!Hello,
a table name cannot be bind as a parameter in SQL, this wont't compile:
EXECUTE IMMEDIATE ' INSERT INTO :1 VALUES ......
USING P_IN_TRG_TABLE_NAME ...but this should work:
EXECUTE IMMEDIATE ' INSERT INTO ' || P_IN_TRG_TABLE_NAME || ' VALUES ......You cannot declare a type that is based on a table which name is in a variable.
PL/SQL is stronly typed language, a type must be known at compile time, a code like this is not allowed:
PROCEDURE xx( src_table_name varchar2 )
DECLARE
TYPE tab IS TABLE OF src_table_name%ROWTYPE;
...This can be done by creating one big dynamic SQL - see example below (tested on Oracle 10 XE - this is a slightly simplified version of your procedure):
CREATE OR REPLACE
PROCEDURE stp1(
p_in_src_table_name VARCHAR2 ,
p_in_trg_table_name VARCHAR2 ,
v_from_data_id NUMBER := 100,
v_to_data_id NUMBER := 100000
IS
BEGIN
EXECUTE IMMEDIATE q'{
DECLARE
TYPE trg_table_type IS TABLE OF }' || p_in_src_table_name || q'{%ROWTYPE;
V_TRG_TABLE_TYPE TRG_TABLE_TYPE;
CURSOR TRG_TAB_CUR IS
SELECT * FROM }' || p_in_src_table_name ||
q'{ WHERE ED_DATA_ID BETWEEN :V_FROM_DATA_ID AND :V_TO_DATA_ID;
BEGIN
OPEN TRG_TAB_CUR;
LOOP
FETCH TRG_TAB_CUR BULK COLLECT INTO V_TRG_TABLE_TYPE LIMIT 30000;
FORALL I IN 1 .. V_TRG_TABLE_TYPE.COUNT
INSERT INTO }' || p_in_trg_table_name || q'{ VALUES V_TRG_TABLE_TYPE( I );
EXIT WHEN TRG_TAB_CUR%NOTFOUND;
END LOOP;
CLOSE TRG_TAB_CUR;
END; }'
USING v_from_data_id, v_to_data_id;
COMMIT;
END;But this probably won't give any performace improvements. Bulk collect and forall can give performance improvements when there is a DML operation inside a loop,
and this one single DML operates on only one record or relatively small number of records, and this DML is repeated many many times in the loop.
I guess that your code is opposite to this - it contains insert statement that operates on many records (one single insert ~ 30000 records),
and you are trying to replace it with bulk collect/forall - INSERT INTO ... SELECT FROM will almost alwayst be faster than bulk collect/forall.
Look at simple test - below is a procedure that uses INSERT ... SELECT :
CREATE OR REPLACE
PROCEDURE stp(
p_in_src_table_name VARCHAR2 ,
p_in_trg_table_name VARCHAR2 ,
v_from_data_id NUMBER := 100,
v_to_data_id NUMBER := 100000
IS
V_QRY1 VARCHAR2(32767);
BEGIN
V_QRY1 := ' INSERT INTO '|| P_IN_TRG_TABLE_NAME ||
' SELECT * FROM '|| P_IN_SRC_TABLE_NAME ||
' WHERE ed_data_id BETWEEN :f AND :t ';
EXECUTE IMMEDIATE V_QRY1
USING V_FROM_DATA_ID, V_TO_DATA_ID;
COMMIT;
END;
/and we can compare both procedures:
SQL> CREATE TABLE test333
2 AS SELECT level ed_data_id ,
3 'XXX ' || LEVEL x,
4 'YYY ' || 2 * LEVEL y
5 FROM dual
6 CONNECT BY LEVEL <= 1000000;
Table created.
SQL> CREATE TABLE test333_dst AS
2 SELECT * FROM test333 WHERE 1 = 0;
Table created.
SQL> set timing on
SQL> ed
Wrote file afiedt.buf
1 BEGIN
2 FOR i IN 1 .. 100 LOOP
3 stp1( 'test333', 'test333_dst', 1000, 31000 );
4 END LOOP;
5* END;
SQL> /
PL/SQL procedure successfully completed.
Elapsed: 00:00:22.12
SQL> ed
Wrote file afiedt.buf
1 BEGIN
2 FOR i IN 1 .. 100 LOOP
3 stp( 'test333', 'test333_dst', 1000, 31000 );
4 END LOOP;
5* END;
SQL> /
PL/SQL procedure successfully completed.
Elapsed: 00:00:14.86without bulk collect ~ 15 sec.
bulk collect version ~ 22 sec. .... 7 sec longer / 15 sec. = about 45% performance decrease. -
PL/SQL using BULK COLLECT and MERGE
what i am trying to do is to use bulk collect to create an array of row data, then loop through the array and either insert or update a table, hence, merge:
FORALL i in ID.first .. ID.last SAVE EXCEPTIONS
MERGE INTO table1 t USING (
select ID(i) ID, {other array fields...} from dual) s
ON t.ID = s.ID
WHEN MATCHED THEN UPDATE...
WHEN NOT MATCHED THEN INSERT ...
The problem is that Oracle always do a INSERT. Has anyone had the same problem? Any workaround?
Thanks.in package header:
TYPE ID_TYPE IS TABLE OF table1.ID%TYPE;
in the proc, I declared
ID ID_TYPE;
then a bulk collect:
select * from .. BULK COLLECT INTO ID...
In addition, i truncate the destination table and run the Proc. all records are insert's. then I ran the same proc again, expecting all records to be updated, but insert occured again causing exceptions due to violation of unique keys.
Message was edited by:
zliao01 -
Hi all,
I have a performance issue in the below code,where i am trying to insert the data from table_stg into target_tab and in parent_tab tables and then to child tables via cursor with bulk collect .the target_tab and parent_tab are huge tables and have a row wise trigger enabled on it .the trigger is mandatory . This timetaken for this block to execute is 5000 seconds.Now my requirement is to reduce it to 5 to 10 mins.
can someone please guide me here.Its bit urgent .Awaiting for your response.
declare
vmax_Value NUMBER(5);
vcnt number(10);
id_val number(20);
pc_id number(15);
vtable_nm VARCHAR2(100);
vstep_no VARCHAR2(10);
vsql_code VARCHAR2(10);
vsql_errm varchar2(200);
vtarget_starttime timestamp;
limit_in number :=10000;
idx number(10);
cursor stg_cursor is
select
DESCRIPTION,
SORT_CODE,
ACCOUNT_NUMBER,
to_number(to_char(CORRESPONDENCE_DATE,'DD')) crr_day,
to_char(CORRESPONDENCE_DATE,'MONTH') crr_month,
to_number(substr(to_char(CORRESPONDENCE_DATE,'DD-MON-YYYY'),8,4)) crr_year,
PARTY_ID,
GUID,
PAPERLESS_REF_IND,
PRODUCT_TYPE,
PRODUCT_BRAND,
PRODUCT_HELD_ID,
NOTIFICATION_PREF,
UNREAD_CORRES_PERIOD,
EMAIL_ID,
MOBILE_NUMBER,
TITLE,
SURNAME,
POSTCODE,
EVENT_TYPE,
PRIORITY_IND,
SUBJECT,
EXT_PRD_ID_TX,
EXT_PRD_HLD_ID_TX,
EXT_SYS_ID,
EXT_PTY_ID_TX,
ACCOUNT_TYPE_CD,
COM_PFR_TYP_TX,
COM_PFR_OPT_TX,
COM_PFR_RSN_CD
from table_stg;
type rec_type is table of stg_rec_type index by pls_integer;
v_rt_all_cols rec_type;
BEGIN
vstep_no := '0';
vmax_value := 0;
vtarget_starttime := systimestamp;
id_val := 0;
pc_id := 0;
success_flag := 0;
vstep_no := '1';
vtable_nm := 'before cursor';
OPEN stg_cursor;
vstep_no := '2';
vtable_nm := 'After cursor';
LOOP
vstep_no := '3';
vtable_nm := 'before fetch';
--loop
FETCH stg_cursor BULK COLLECT INTO v_rt_all_cols LIMIT limit_in;
vstep_no := '4';
vtable_nm := 'after fetch';
--EXIT WHEN v_rt_all_cols.COUNT = 0;
EXIT WHEN stg_cursor%NOTFOUND;
FOR i IN 1 .. v_rt_all_cols.COUNT
LOOP
dbms_output.put_line(upper(v_rt_all_cols(i).event_type));
if (upper(v_rt_all_cols(i).event_type) = upper('System_enforced')) then
vstep_no := '4.1';
vtable_nm := 'before seq sel';
select PC_SEQ.nextval into pc_id from dual;
vstep_no := '4.2';
vtable_nm := 'before insert corres';
INSERT INTO target1_tab
(ID,
PARTY_ID,
PRODUCT_BRAND,
SORT_CODE,
ACCOUNT_NUMBER,
EXT_PRD_ID_TX,
EXT_PRD_HLD_ID_TX,
EXT_SYS_ID,
EXT_PTY_ID_TX,
ACCOUNT_TYPE_CD,
COM_PFR_TYP_TX,
COM_PFR_OPT_TX,
COM_PFR_RSN_CD,
status)
VALUES
(pc_id,
v_rt_all_cols(i).party_id,
decode(v_rt_all_cols(i).product_brand,'LTB',2,'HLX',1,'HAL',1,'BOS',3,'VER',4,0),
v_rt_all_cols(i).sort_code,
'XXXX'||substr(trim(v_rt_all_cols(i).ACCOUNT_NUMBER),length(trim(v_rt_all_cols(i).ACCOUNT_NUMBER))-3,4),
v_rt_all_cols(i).EXT_PRD_ID_TX,
v_rt_all_cols(i).EXT_PRD_HLD_ID_TX,
v_rt_all_cols(i).EXT_SYS_ID,
v_rt_all_cols(i).EXT_PTY_ID_TX,
v_rt_all_cols(i).ACCOUNT_TYPE_CD,
v_rt_all_cols(i).COM_PFR_TYP_TX,
v_rt_all_cols(i).COM_PFR_OPT_TX,
v_rt_all_cols(i).COM_PFR_RSN_CD,
NULL);
vstep_no := '4.3';
vtable_nm := 'after insert corres';
else
select COM_SEQ.nextval into id_val from dual;
vstep_no := '6';
vtable_nm := 'before insertcomm';
if (upper(v_rt_all_cols(i).event_type) = upper('REMINDER')) then
vstep_no := '6.01';
vtable_nm := 'after if insertcomm';
insert into parent_tab
(ID ,
CTEM_CODE,
CHA_CODE,
CT_CODE,
CONTACT_POINT_ID,
SOURCE,
RECEIVED_DATE,
SEND_DATE,
RETRY_COUNT)
values
(id_val,
lower(v_rt_all_cols(i).event_type),
decode(v_rt_all_cols(i).product_brand,'LTB',2,'HLX',1,'HAL',1,'BOS',3,'VER',4,0),
'Email',
v_rt_all_cols(i).email_id,
'IADAREMINDER',
systimestamp,
systimestamp,
0);
else
vstep_no := '6.02';
vtable_nm := 'after else insertcomm';
insert into parent_tab
(ID ,
CTEM_CODE,
CHA_CODE,
CT_CODE,
CONTACT_POINT_ID,
SOURCE,
RECEIVED_DATE,
SEND_DATE,
RETRY_COUNT)
values
(id_val,
lower(v_rt_all_cols(i).event_type),
decode(v_rt_all_cols(i).product_brand,'LTB',2,'HLX',1,'HAL',1,'BOS',3,'VER',4,0),
'Email',
v_rt_all_cols(i).email_id,
'CORRESPONDENCE',
systimestamp,
systimestamp,
0);
END if;
vstep_no := '6.11';
vtable_nm := 'before chop';
if (v_rt_all_cols(i).ACCOUNT_NUMBER is not null) then
v_rt_all_cols(i).ACCOUNT_NUMBER := 'XXXX'||substr(trim(v_rt_all_cols(i).ACCOUNT_NUMBER),length(trim(v_rt_all_cols(i).ACCOUNT_NUMBER))-3,4);
insert into child_tab
(COM_ID,
KEY,
VALUE)
values
(id_val,
'IB.Correspondence.AccountNumberMasked',
v_rt_all_cols(i).ACCOUNT_NUMBER);
end if;
vstep_no := '6.1';
vtable_nm := 'before stateday';
if (v_rt_all_cols(i).crr_day is not null) then
insert into child_tab
(COM_ID,
KEY,
VALUE)
values
(id_val,
--'IB.Correspondence.Date.Day',
'IB.Crsp.Date.Day',
v_rt_all_cols(i).crr_day);
end if;
vstep_no := '6.2';
vtable_nm := 'before statemth';
if (v_rt_all_cols(i).crr_month is not null) then
insert into child_tab
(COM_ID,
KEY,
VALUE)
values
(id_val,
--'IB.Correspondence.Date.Month',
'IB.Crsp.Date.Month',
v_rt_all_cols(i).crr_month);
end if;
vstep_no := '6.3';
vtable_nm := 'before stateyear';
if (v_rt_all_cols(i).crr_year is not null) then
insert into child_tab
(COM_ID,
KEY,
VALUE)
values
(id_val,
--'IB.Correspondence.Date.Year',
'IB.Crsp.Date.Year',
v_rt_all_cols(i).crr_year);
end if;
vstep_no := '7';
vtable_nm := 'before type';
if (v_rt_all_cols(i).product_type is not null) then
insert into child_tab
(COM_ID,
KEY,
VALUE)
values
(id_val,
'IB.Product.ProductName',
v_rt_all_cols(i).product_type);
end if;
vstep_no := '9';
vtable_nm := 'before title';
if (trim(v_rt_all_cols(i).title) is not null) then
insert into child_tab
(COM_ID,
KEY,
VALUE )
values
(id_val,
'IB.Customer.Title',
trim(v_rt_all_cols(i).title));
end if;
vstep_no := '10';
vtable_nm := 'before surname';
if (v_rt_all_cols(i).surname is not null) then
insert into child_tab
(COM_ID,
KEY,
VALUE)
values
(id_val,
'IB.Customer.LastName',
v_rt_all_cols(i).surname);
end if;
vstep_no := '12';
vtable_nm := 'before postcd';
if (trim(v_rt_all_cols(i).POSTCODE) is not null) then
insert into child_tab
(COM_ID,
KEY,
VALUE)
values
(id_val,
'IB.Customer.Addr.PostCodeMasked',
substr(replace(v_rt_all_cols(i).POSTCODE,' ',''),length(replace(v_rt_all_cols(i).POSTCODE,' ',''))-2,3));
end if;
vstep_no := '13';
vtable_nm := 'before subject';
if (trim(v_rt_all_cols(i).SUBJECT) is not null) then
insert into child_tab
(COM_ID,
KEY,
VALUE)
values
(id_val,
'IB.Correspondence.Subject',
v_rt_all_cols(i).subject);
end if;
vstep_no := '14';
vtable_nm := 'before inactivity';
if (trim(v_rt_all_cols(i).UNREAD_CORRES_PERIOD) is null or
trim(v_rt_all_cols(i).UNREAD_CORRES_PERIOD) = '3' or
trim(v_rt_all_cols(i).UNREAD_CORRES_PERIOD) = '6' or
trim(v_rt_all_cols(i).UNREAD_CORRES_PERIOD) = '9') then
insert into child_tab
(COM_ID,
KEY,
VALUE)
values
(id_val,
'IB.Correspondence.Inactivity',
v_rt_all_cols(i).UNREAD_CORRES_PERIOD);
end if;
vstep_no := '14.1';
vtable_nm := 'after notfound';
end if;
vstep_no := '15';
vtable_nm := 'after notfound';
END LOOP;
end loop;
vstep_no := '16';
vtable_nm := 'before closecur';
CLOSE stg_cursor;
vstep_no := '17';
vtable_nm := 'before commit';
DELETE FROM table_stg;
COMMIT;
vstep_no := '18';
vtable_nm := 'after commit';
EXCEPTION
WHEN OTHERS THEN
ROLLBACK;
success_flag := 1;
vsql_code := SQLCODE;
vsql_errm := SUBSTR(sqlerrm,1,200);
error_logging_pkg.inserterrorlog('samp',vsql_code,vsql_errm, vtable_nm,vstep_no);
RAISE_APPLICATION_ERROR (-20011, 'samp '||vstep_no||' SQLERRM:'||SQLERRM);
end;
ThanksIts bit urgent
NO - it is NOT urgent. Not to us.
If you have an urgent problem you need to hire a consultant.
I have a performance issue in the below code,
Maybe you do and maybe you don't. How are we to really know? You haven't posted ANYTHING indicating that a performance issue exists. Please read the FAQ for how to post a tuning request and the info you need to provide. First and foremost you have to post SOMETHING that actually shows that a performance issue exists. Troubleshooting requires FACTS not just a subjective opinion.
where i am trying to insert the data from table_stg into target_tab and in parent_tab tables and then to child tables via cursor with bulk collect .the target_tab and parent_tab are huge tables and have a row wise trigger enabled on it .the trigger is mandatory . This timetaken for this block to execute is 5000 seconds.Now my requirement is to reduce it to 5 to 10 mins.
Personally I think 5000 seconds (about 1 hr 20 minutes) is very fast for processing 800 trillion rows of data into parent and child tables. Why do you think that is slow?
Your code has several major flaws that need to be corrected before you can even determine what, if anything, needs to be tuned.
This code has the EXIT statement at the beginning of the loop instead of at the end
FETCH stg_cursor BULK COLLECT INTO v_rt_all_cols LIMIT limit_in;
vstep_no := '4';
vtable_nm := 'after fetch';
--EXIT WHEN v_rt_all_cols.COUNT = 0;
EXIT WHEN stg_cursor%NOTFOUND;
The correct place for the %NOTFOUND test when using BULK COLLECT is at the END of the loop; that is, the last statement in the loop.
You can use a COUNT test at the start of the loop but ironically you have commented it out and have now done it wrong. Either move the NOTFOUND test to the end of the loop or remove it and uncomment the COUNT test.
WHEN OTHERS THEN
ROLLBACK;
That basically says you don't even care what problem occurs or whether the problem is for a single record of your 10,000 in the collection. You pretty much just throw away any stack trace and substitute your own message.
Your code also has NO exception handling for any of the individual steps or blocks of code.
The code you posted also begs the question of why you are using NAME=VALUE pairs for child data rows? Why aren't you using a standard relational table for this data?
As others have noted you are using slow-by-slow (row by row processing). Let's assume that PL/SQL, the bulk collect and row-by-row is actually necessary.
Then you should be constructing the parent and child records into collections and then inserting them in BULK using FORALL.
1. Create a collection for the new parent rows
2. Create a collection for the new child rows
3. For each set of LIMIT source row data
a. empty the parent and child collections
b. populate those collections with new parent/child data
c. bulk insert the parent collection into the parent table
d. bulk insert the child collection into the child table
And unless you really want to either load EVERYTHING or abandon everything you should use bulk exception handling so that the clean data gets processed and only the dirty data gets rejected. -
How to use BULK COLLECT, FORALL and TREAT
There is a need to read match and update data from and into a custom table. The table would have about 3 millions rows and holds key numbers. BAsed on a field value of this custom table, relevant data needs to be fetched from joins of other tables and updated in the custom table. I plan to use BULK COLLECT and FORALL.
All examples I have seen, do an insert into a table. How do I go about reading all values of a given field and fetching other relevant data and then updating the custom table with data fetched.
Defined an object with specifics like this
CREATE OR REPLACE TYPE imei_ot AS OBJECT (
recid NUMBER,
imei VARCHAR2(30),
STORE VARCHAR2(100),
status VARCHAR2(1),
TIMESTAMP DATE,
order_number VARCHAR2(30),
order_type VARCHAR2(30),
sku VARCHAR2(30),
order_date DATE,
attribute1 VARCHAR2(240),
market VARCHAR2(240),
processed_flag VARCHAR2(1),
last_update_date DATE
Now within a package procedure I have defined like this.
type imei_ott is table of imei_ot;
imei_ntt imei_ott;
begin
SELECT imei_ot (recid,
imei,
STORE,
status,
TIMESTAMP,
order_number,
order_type,
sku,
order_date,
attribute1,
market,
processed_flag,
last_update_date
BULK COLLECT INTO imei_ntt
FROM (SELECT stg.recid, stg.imei, cip.store_location, 'S',
co.rtl_txn_timestamp, co.rtl_order_number, 'CUST',
msi.segment1 || '.' || msi.segment3,
TRUNC (co.txn_timestamp), col.part_number, 'ZZ',
stg.processed_flag, SYSDATE
FROM custom_orders co,
custom_order_lines col,
custom_stg stg,
mtl_system_items_b msi
WHERE co.header_id = col.header_id
AND msi.inventory_item_id = col.inventory_item_id
AND msi.organization_id =
(SELECT organization_id
FROM hr_all_organization_units_tl
WHERE NAME = 'Item Master'
AND source_lang = USERENV ('LANG'))
AND stg.imei = col.serial_number
AND stg.processed_flag = 'U');
/* Update staging table in one go for COR order data */
FORALL indx IN 1 .. imei_ntt.COUNT
UPDATE custom_stg
SET STORE = TREAT (imei_ntt (indx) AS imei_ot).STORE,
status = TREAT (imei_ntt (indx) AS imei_ot).status,
TIMESTAMP = TREAT (imei_ntt (indx) AS imei_ot).TIMESTAMP,
order_number = TREAT (imei_ntt (indx) AS imei_ot).order_number,
order_type = TREAT (imei_ntt (indx) AS imei_ot).order_type,
sku = TREAT (imei_ntt (indx) AS imei_ot).sku,
order_date = TREAT (imei_ntt (indx) AS imei_ot).order_date,
attribute1 = TREAT (imei_ntt (indx) AS imei_ot).attribute1,
market = TREAT (imei_ntt (indx) AS imei_ot).market,
processed_flag =
TREAT (imei_ntt (indx) AS imei_ot).processed_flag,
last_update_date =
TREAT (imei_ntt (indx) AS imei_ot).last_update_date
WHERE recid = TREAT (imei_ntt (indx) AS imei_ot).recid
AND imei = TREAT (imei_ntt (indx) AS imei_ot).imei;
DBMS_OUTPUT.put_line ( TO_CHAR (SQL%ROWCOUNT)
|| ' rows updated using Bulk Collect / For All.'
EXCEPTION
WHEN NO_DATA_FOUND
THEN
DBMS_OUTPUT.put_line ('No Data: ' || SQLERRM);
WHEN OTHERS
THEN
DBMS_OUTPUT.put_line ('Other Error: ' || SQLERRM);
END;
Now for the unfortunate part. When I compile the pkg, I face an error
PL/SQL: ORA-00904: "LAST_UPDATE_DATE": invalid identifier
I am not sure where I am wrong. Object type has the last update date field and the custom table also has the same field.
Could someone please throw some light and suggestion?
Thanks
udsI suspect your error comes from the »bulk collect into« and not from the »forall loop«.
From a first glance you need to alias sysdate with last_update_date and some of the other select items need to be aliased as well :
But a simplified version would be
select imei_ot (stg.recid,
stg.imei,
cip.store_location,
'S',
co.rtl_txn_timestamp,
co.rtl_order_number,
'CUST',
msi.segment1 || '.' || msi.segment3,
trunc (co.txn_timestamp),
col.part_number,
'ZZ',
stg.processed_flag,
sysdate
bulk collect into imei_ntt
from custom_orders co,
custom_order_lines col,
custom_stg stg,
mtl_system_items_b msi
where co.header_id = col.header_id
and msi.inventory_item_id = col.inventory_item_id
and msi.organization_id =
(select organization_id
from hr_all_organization_units_tl
where name = 'Item Master' and source_lang = userenv ('LANG'))
and stg.imei = col.serial_number
and stg.processed_flag = 'U';
... -
ORA-06502 during a procedure which uses Bulk collect feature and nested tab
Hello Friends,
have created one procedure which uses Bulk collect and nested table to hold the bulk data. This procedure was using one cursor and a nested table with the same type as the cursor to hold data fetched from cursor. Bulk collection technique was used to collect data from cursor to nested table. But it is giving ORA-06502 error.
I reduced code of procedure to following to trace the error point. But still error is comming. Please help us to find the cause and solve it.
Script which is giving error:
declare
v_Errorflag BINARY_INTEGER;
v_flag number := 1;
CURSOR cur_terminal_info Is
SELECT distinct
'a' SettlementType
FROM
dual;
TYPE typ_cur_terminal IS TABLE OF cur_terminal_info%ROWTYPE;
Tab_Terminal_info typ_cur_Terminal;
BEGIN
v_Errorflag := 2;
OPEN cur_terminal_info;
LOOP
v_Errorflag := 4;
FETCH cur_terminal_info BULK COLLECT INTO tab_terminal_info LIMIT 300;
EXIT WHEN cur_terminal_info%rowcount <= 0;
v_Errorflag := 5;
FOR Y IN Tab_Terminal_Info.FIRST..tab_terminal_info.LAST
LOOP
dbms_output.put_line(v_flag);
v_flag := v_flag + 1;
end loop;
END LOOP;
v_Errorflag := 13;
COMMIT;
END;
I have updated script as following to change datatype as varchar2 for nested table, but still same error is
comming..
declare
v_Errorflag BINARY_INTEGER;
v_flag number := 1;
CURSOR cur_terminal_info Is
SELECT distinct
'a' SettlementType
FROM
dual;
TYPE typ_cur_terminal IS TABLE OF varchar2(50);
Tab_Terminal_info typ_cur_Terminal;
BEGIN
v_Errorflag := 2;
OPEN cur_terminal_info;
LOOP
v_Errorflag := 4;
FETCH cur_terminal_info BULK COLLECT INTO tab_terminal_info LIMIT 300;
EXIT WHEN cur_terminal_info%rowcount <= 0;
v_Errorflag := 5;
FOR Y IN Tab_Terminal_Info.FIRST..tab_terminal_info.LAST
LOOP
dbms_output.put_line(v_flag);
v_flag := v_flag + 1;
end loop;
END LOOP;
v_Errorflag := 13;
COMMIT;
I could not find the exact reason of error.
Please help us to solve this error.
Thanks and Regards..
Dipali..Hello Friends,
I got the solution.. :)
I did one mistake in procedure where the loop should end.
I used the statemetn: EXIT WHEN cur_terminal_info%rowcount <= 0;
But it should be: EXIT WHEN Tab_Terminal_Info.COUNT <= 0;
Now my script is working fine.. :)
Thanks and Regards,
Dipali.. -
Can I use Bulk Collect results as input parameter for another cursor
MUSIC ==> remote MUSIC_DB database, MUSIC table has 60 million rows
PRICE_DATA ==> remote PRICING_DB database, PRICE_DATE table has 1 billion rows
These two table once existed in same database, but size of database exceeded available hardware size and hardware budget, so the PRICE_DATA table was moved to another Oracle database. I need to create a single report that combines data from both of these tables, and a distributed join with DRIVING_SITE hint will not work because the size of both table is too large to push to one DRIVING_SITE location, so I wrote this PLSQL block to process in small blocks.
QUESTION: how can use bulk collect from one cursor and pass that bulk collected information as input to second cursor without specifically listing each cell of the PLSQL bulk collection? See sample pseudo-code below, I am trying to determine more efficient way to code than hard-coding 100 parameter names into 2nd cursor.
NOTE: below is truly pseudo-code, I had to change the names of everything to adhere to NDA, but below works and is fast enough for my purposes, but if I want to change from 100 input parameters to 200, I have to add more hard-coded values. There has got to be a better way.
DECLARE
-- define cursor that retrieves distinct SONG_IDs from MUSIC table in remote music database
CURSOR C_CURRENT_MUSIC
IS
select distinct SONG_ID
from MUSIC@MUSIC_DB
where PRODUCTION_RELEASE=1
/* define a parameterized cursor that accepts 100 SONG_IDs and retrieves
required pricing information
CURSOR C_get_music_price_data
P_SONG_ID_001 NUMBER, P_SONG_ID_002 NUMBER, P_SONG_ID_003 NUMBER, P_SONG_ID_004 NUMBER, P_SONG_ID_005 NUMBER, P_SONG_ID_006 NUMBER, P_SONG_ID_007 NUMBER, P_SONG_ID_008 NUMBER, P_SONG_ID_009 NUMBER, P_SONG_ID_010 NUMBER,
P_SONG_ID_011 NUMBER, P_SONG_ID_012 NUMBER, P_SONG_ID_013 NUMBER, P_SONG_ID_014 NUMBER, P_SONG_ID_015 NUMBER, P_SONG_ID_016 NUMBER, P_SONG_ID_017 NUMBER, P_SONG_ID_018 NUMBER, P_SONG_ID_019 NUMBER, P_SONG_ID_020 NUMBER,
P_SONG_ID_021 NUMBER, P_SONG_ID_022 NUMBER, P_SONG_ID_023 NUMBER, P_SONG_ID_024 NUMBER, P_SONG_ID_025 NUMBER, P_SONG_ID_026 NUMBER, P_SONG_ID_027 NUMBER, P_SONG_ID_028 NUMBER, P_SONG_ID_029 NUMBER, P_SONG_ID_030 NUMBER,
P_SONG_ID_031 NUMBER, P_SONG_ID_032 NUMBER, P_SONG_ID_033 NUMBER, P_SONG_ID_034 NUMBER, P_SONG_ID_035 NUMBER, P_SONG_ID_036 NUMBER, P_SONG_ID_037 NUMBER, P_SONG_ID_038 NUMBER, P_SONG_ID_039 NUMBER, P_SONG_ID_040 NUMBER,
P_SONG_ID_041 NUMBER, P_SONG_ID_042 NUMBER, P_SONG_ID_043 NUMBER, P_SONG_ID_044 NUMBER, P_SONG_ID_045 NUMBER, P_SONG_ID_046 NUMBER, P_SONG_ID_047 NUMBER, P_SONG_ID_048 NUMBER, P_SONG_ID_049 NUMBER, P_SONG_ID_050 NUMBER,
P_SONG_ID_051 NUMBER, P_SONG_ID_052 NUMBER, P_SONG_ID_053 NUMBER, P_SONG_ID_054 NUMBER, P_SONG_ID_055 NUMBER, P_SONG_ID_056 NUMBER, P_SONG_ID_057 NUMBER, P_SONG_ID_058 NUMBER, P_SONG_ID_059 NUMBER, P_SONG_ID_060 NUMBER,
P_SONG_ID_061 NUMBER, P_SONG_ID_062 NUMBER, P_SONG_ID_063 NUMBER, P_SONG_ID_064 NUMBER, P_SONG_ID_065 NUMBER, P_SONG_ID_066 NUMBER, P_SONG_ID_067 NUMBER, P_SONG_ID_068 NUMBER, P_SONG_ID_069 NUMBER, P_SONG_ID_070 NUMBER,
P_SONG_ID_071 NUMBER, P_SONG_ID_072 NUMBER, P_SONG_ID_073 NUMBER, P_SONG_ID_074 NUMBER, P_SONG_ID_075 NUMBER, P_SONG_ID_076 NUMBER, P_SONG_ID_077 NUMBER, P_SONG_ID_078 NUMBER, P_SONG_ID_079 NUMBER, P_SONG_ID_080 NUMBER,
P_SONG_ID_081 NUMBER, P_SONG_ID_082 NUMBER, P_SONG_ID_083 NUMBER, P_SONG_ID_084 NUMBER, P_SONG_ID_085 NUMBER, P_SONG_ID_086 NUMBER, P_SONG_ID_087 NUMBER, P_SONG_ID_088 NUMBER, P_SONG_ID_089 NUMBER, P_SONG_ID_090 NUMBER,
P_SONG_ID_091 NUMBER, P_SONG_ID_092 NUMBER, P_SONG_ID_093 NUMBER, P_SONG_ID_094 NUMBER, P_SONG_ID_095 NUMBER, P_SONG_ID_096 NUMBER, P_SONG_ID_097 NUMBER, P_SONG_ID_098 NUMBER, P_SONG_ID_099 NUMBER, P_SONG_ID_100 NUMBER
IS
select
from PRICE_DATA@PRICING_DB
where COUNTRY = 'USA'
and START_DATE <= sysdate
and END_DATE > sysdate
and vpc.SONG_ID IN
P_SONG_ID_001 ,P_SONG_ID_002 ,P_SONG_ID_003 ,P_SONG_ID_004 ,P_SONG_ID_005 ,P_SONG_ID_006 ,P_SONG_ID_007 ,P_SONG_ID_008 ,P_SONG_ID_009 ,P_SONG_ID_010,
P_SONG_ID_011 ,P_SONG_ID_012 ,P_SONG_ID_013 ,P_SONG_ID_014 ,P_SONG_ID_015 ,P_SONG_ID_016 ,P_SONG_ID_017 ,P_SONG_ID_018 ,P_SONG_ID_019 ,P_SONG_ID_020,
P_SONG_ID_021 ,P_SONG_ID_022 ,P_SONG_ID_023 ,P_SONG_ID_024 ,P_SONG_ID_025 ,P_SONG_ID_026 ,P_SONG_ID_027 ,P_SONG_ID_028 ,P_SONG_ID_029 ,P_SONG_ID_030,
P_SONG_ID_031 ,P_SONG_ID_032 ,P_SONG_ID_033 ,P_SONG_ID_034 ,P_SONG_ID_035 ,P_SONG_ID_036 ,P_SONG_ID_037 ,P_SONG_ID_038 ,P_SONG_ID_039 ,P_SONG_ID_040,
P_SONG_ID_041 ,P_SONG_ID_042 ,P_SONG_ID_043 ,P_SONG_ID_044 ,P_SONG_ID_045 ,P_SONG_ID_046 ,P_SONG_ID_047 ,P_SONG_ID_048 ,P_SONG_ID_049 ,P_SONG_ID_050,
P_SONG_ID_051 ,P_SONG_ID_052 ,P_SONG_ID_053 ,P_SONG_ID_054 ,P_SONG_ID_055 ,P_SONG_ID_056 ,P_SONG_ID_057 ,P_SONG_ID_058 ,P_SONG_ID_059 ,P_SONG_ID_060,
P_SONG_ID_061 ,P_SONG_ID_062 ,P_SONG_ID_063 ,P_SONG_ID_064 ,P_SONG_ID_065 ,P_SONG_ID_066 ,P_SONG_ID_067 ,P_SONG_ID_068 ,P_SONG_ID_069 ,P_SONG_ID_070,
P_SONG_ID_071 ,P_SONG_ID_072 ,P_SONG_ID_073 ,P_SONG_ID_074 ,P_SONG_ID_075 ,P_SONG_ID_076 ,P_SONG_ID_077 ,P_SONG_ID_078 ,P_SONG_ID_079 ,P_SONG_ID_080,
P_SONG_ID_081 ,P_SONG_ID_082 ,P_SONG_ID_083 ,P_SONG_ID_084 ,P_SONG_ID_085 ,P_SONG_ID_086 ,P_SONG_ID_087 ,P_SONG_ID_088 ,P_SONG_ID_089 ,P_SONG_ID_090,
P_SONG_ID_091 ,P_SONG_ID_092 ,P_SONG_ID_093 ,P_SONG_ID_094 ,P_SONG_ID_095 ,P_SONG_ID_096 ,P_SONG_ID_097 ,P_SONG_ID_098 ,P_SONG_ID_099 ,P_SONG_ID_100
group by
vpc.SONG_ID
,vpc.STOREFRONT_ID
TYPE SONG_ID_TYPE IS TABLE OF MUSIC@MUSIC_DB%TYPE INDEX BY BINARY_INTEGER;
V_SONG_ID_ARRAY SONG_ID_TYPE ;
v_commit_counter NUMBER := 0;
BEGIN
/* open cursor you intent to bulk collect from */
OPEN C_CURRENT_MUSIC;
LOOP
/* in batches of 100, bulk collect ADAM_ID mapped TMS_IDENTIFIER into PLSQL table or records */
FETCH C_CURRENT_MUSIC BULK COLLECT INTO V_SONG_ID_ARRAY LIMIT 100;
EXIT WHEN V_SONG_ID_ARRAY.COUNT = 0;
/* to avoid NO DATA FOUND error when pass 100 parameters to OPEN cursor, if the arrary
is not fully populated to 100, pad the array with nulls to fill up to 100 cells. */
IF (V_SONG_ID_ARRAY.COUNT >=1 and V_SONG_ID_ARRAY.COUNT <> 100) THEN
FOR j IN V_SONG_ID_ARRAY.COUNT+1..100 LOOP
V_SONG_ID_ARRAY(j) := null;
END LOOP;
END IF;
/* pass a batch of 100 to cursor that get price information per SONG_ID and STOREFRONT_ID */
FOR j IN C_get_music_price_data
V_SONG_ID_ARRAY(1) ,V_SONG_ID_ARRAY(2) ,V_SONG_ID_ARRAY(3) ,V_SONG_ID_ARRAY(4) ,V_SONG_ID_ARRAY(5) ,V_SONG_ID_ARRAY(6) ,V_SONG_ID_ARRAY(7) ,V_SONG_ID_ARRAY(8) ,V_SONG_ID_ARRAY(9) ,V_SONG_ID_ARRAY(10) ,
V_SONG_ID_ARRAY(11) ,V_SONG_ID_ARRAY(12) ,V_SONG_ID_ARRAY(13) ,V_SONG_ID_ARRAY(14) ,V_SONG_ID_ARRAY(15) ,V_SONG_ID_ARRAY(16) ,V_SONG_ID_ARRAY(17) ,V_SONG_ID_ARRAY(18) ,V_SONG_ID_ARRAY(19) ,V_SONG_ID_ARRAY(20) ,
V_SONG_ID_ARRAY(21) ,V_SONG_ID_ARRAY(22) ,V_SONG_ID_ARRAY(23) ,V_SONG_ID_ARRAY(24) ,V_SONG_ID_ARRAY(25) ,V_SONG_ID_ARRAY(26) ,V_SONG_ID_ARRAY(27) ,V_SONG_ID_ARRAY(28) ,V_SONG_ID_ARRAY(29) ,V_SONG_ID_ARRAY(30) ,
V_SONG_ID_ARRAY(31) ,V_SONG_ID_ARRAY(32) ,V_SONG_ID_ARRAY(33) ,V_SONG_ID_ARRAY(34) ,V_SONG_ID_ARRAY(35) ,V_SONG_ID_ARRAY(36) ,V_SONG_ID_ARRAY(37) ,V_SONG_ID_ARRAY(38) ,V_SONG_ID_ARRAY(39) ,V_SONG_ID_ARRAY(40) ,
V_SONG_ID_ARRAY(41) ,V_SONG_ID_ARRAY(42) ,V_SONG_ID_ARRAY(43) ,V_SONG_ID_ARRAY(44) ,V_SONG_ID_ARRAY(45) ,V_SONG_ID_ARRAY(46) ,V_SONG_ID_ARRAY(47) ,V_SONG_ID_ARRAY(48) ,V_SONG_ID_ARRAY(49) ,V_SONG_ID_ARRAY(50) ,
V_SONG_ID_ARRAY(51) ,V_SONG_ID_ARRAY(52) ,V_SONG_ID_ARRAY(53) ,V_SONG_ID_ARRAY(54) ,V_SONG_ID_ARRAY(55) ,V_SONG_ID_ARRAY(56) ,V_SONG_ID_ARRAY(57) ,V_SONG_ID_ARRAY(58) ,V_SONG_ID_ARRAY(59) ,V_SONG_ID_ARRAY(60) ,
V_SONG_ID_ARRAY(61) ,V_SONG_ID_ARRAY(62) ,V_SONG_ID_ARRAY(63) ,V_SONG_ID_ARRAY(64) ,V_SONG_ID_ARRAY(65) ,V_SONG_ID_ARRAY(66) ,V_SONG_ID_ARRAY(67) ,V_SONG_ID_ARRAY(68) ,V_SONG_ID_ARRAY(69) ,V_SONG_ID_ARRAY(70) ,
V_SONG_ID_ARRAY(71) ,V_SONG_ID_ARRAY(72) ,V_SONG_ID_ARRAY(73) ,V_SONG_ID_ARRAY(74) ,V_SONG_ID_ARRAY(75) ,V_SONG_ID_ARRAY(76) ,V_SONG_ID_ARRAY(77) ,V_SONG_ID_ARRAY(78) ,V_SONG_ID_ARRAY(79) ,V_SONG_ID_ARRAY(80) ,
V_SONG_ID_ARRAY(81) ,V_SONG_ID_ARRAY(82) ,V_SONG_ID_ARRAY(83) ,V_SONG_ID_ARRAY(84) ,V_SONG_ID_ARRAY(85) ,V_SONG_ID_ARRAY(86) ,V_SONG_ID_ARRAY(87) ,V_SONG_ID_ARRAY(88) ,V_SONG_ID_ARRAY(89) ,V_SONG_ID_ARRAY(90) ,
V_SONG_ID_ARRAY(91) ,V_SONG_ID_ARRAY(92) ,V_SONG_ID_ARRAY(93) ,V_SONG_ID_ARRAY(94) ,V_SONG_ID_ARRAY(95) ,V_SONG_ID_ARRAY(96) ,V_SONG_ID_ARRAY(97) ,V_SONG_ID_ARRAY(98) ,V_SONG_ID_ARRAY(99) ,V_SONG_ID_ARRAY(100)
LOOP
/* do stuff with data from Song and Pricing Database coming from the two
separate cursors, then continue processing more rows...
END LOOP;
/* commit after each batch of 100 SONG_IDs is processed */
COMMIT;
EXIT WHEN C_CURRENT_MUSIC%NOTFOUND; -- exit when there are no more rows to fetch from cursor
END LOOP; -- bulk fetching loop
CLOSE C_CURRENT_MUSIC; -- close cursor that was used in bulk collection
/* commit rows */
COMMIT; -- commit any remaining uncommitted data.
END;I've got a problem when using passing VARRAY of numbers as parameter to remote cursor: it takes a super long time to run, sometimes doesn't finish even after an hour as passed.
Continuing with my example in original entry, I replaced the bulk collect into PLSQL table collection with a VARRAY and i bulk collect into the VARRAY, this is fast and I know it works because I can DBMS_OUTPUT.PUT_LINE cells of VARRAY so I know it is getting populated correctly. However, when I pass the VARRAY containing 100 cells populated with SONG_IDs as parameter to cursor, execution time is over an hour and when I am expecting a few seconds.
Below code example strips the problem down to it's raw details, I skip the bulk collect and just manually populate a VARRAY with 100 SONG_ID values, then try to pass to as parameter to a cursor, but the execution time of cursor is unexpectedly long, over 30 minutes, sometime longer, when I am expecting seconds.
IMPORTANT: If I take the same 100 SONG_IDs and place them directly in the cursor query's where IN clause, the SQL runs in under 5 seconds and returns result. Also, if I pass the 100 SONG_IDs as individual cells of a PLSQL table collection, then it also runs fast.
I thought that since the VARRAY is used via select subquery that is it queried locally, but the cursor is remote, and that I had a distribute problem on my hands, so I put in the DRIVING_SITE hint to attempt to force the result of query against VARRAY to go to remote server and rest of query will run there before returning result, but that didn't work either, still got slow response.
Is something wrong with my code, or I am running into a Oracle problem that may require support to resolve?
DECLARE
/* define a parameterized cursor that accepts XXX number of in SONG_IDs and
retrieves required pricing information
CURSOR C_get_music_price_data
p_array_song_ids SYS.ODCInumberList
IS
select /*+DRIVING_SITE(pd) */
count(distinct s.EVE_ID)
from PRICE_DATA@PRICING_DB pd
where pd.COUNTRY = 'USA'
and pd.START_DATE <= sysdate
and pd.END_DATE > sysdate
and pd.SONG_ID IN
select column_value from table(p_array_song_ids)
group by
pd.SONG_ID
,pd.STOREFRONT_ID
V_ARRAY_SONG_IDS SYS.ODCInumberList := SYS.ODCInumberList();
BEGIN
V_ARRAY_SONG_IDS.EXTEND(100);
V_ARRAY_SONG_IDS( 1 ) := 31135 ;
V_ARRAY_SONG_IDS( 2 ) := 31140 ;
V_ARRAY_SONG_IDS( 3 ) := 31142 ;
V_ARRAY_SONG_IDS( 4 ) := 31144 ;
V_ARRAY_SONG_IDS( 5 ) := 31146 ;
V_ARRAY_SONG_IDS( 6 ) := 31148 ;
V_ARRAY_SONG_IDS( 7 ) := 31150 ;
V_ARRAY_SONG_IDS( 8 ) := 31152 ;
V_ARRAY_SONG_IDS( 9 ) := 31154 ;
V_ARRAY_SONG_IDS( 10 ) := 31156 ;
V_ARRAY_SONG_IDS( 11 ) := 31158 ;
V_ARRAY_SONG_IDS( 12 ) := 31160 ;
V_ARRAY_SONG_IDS( 13 ) := 33598 ;
V_ARRAY_SONG_IDS( 14 ) := 33603 ;
V_ARRAY_SONG_IDS( 15 ) := 33605 ;
V_ARRAY_SONG_IDS( 16 ) := 33607 ;
V_ARRAY_SONG_IDS( 17 ) := 33609 ;
V_ARRAY_SONG_IDS( 18 ) := 33611 ;
V_ARRAY_SONG_IDS( 19 ) := 33613 ;
V_ARRAY_SONG_IDS( 20 ) := 33615 ;
V_ARRAY_SONG_IDS( 21 ) := 33617 ;
V_ARRAY_SONG_IDS( 22 ) := 33630 ;
V_ARRAY_SONG_IDS( 23 ) := 33632 ;
V_ARRAY_SONG_IDS( 24 ) := 33636 ;
V_ARRAY_SONG_IDS( 25 ) := 33638 ;
V_ARRAY_SONG_IDS( 26 ) := 33640 ;
V_ARRAY_SONG_IDS( 27 ) := 33642 ;
V_ARRAY_SONG_IDS( 28 ) := 33644 ;
V_ARRAY_SONG_IDS( 29 ) := 33646 ;
V_ARRAY_SONG_IDS( 30 ) := 33648 ;
V_ARRAY_SONG_IDS( 31 ) := 33662 ;
V_ARRAY_SONG_IDS( 32 ) := 33667 ;
V_ARRAY_SONG_IDS( 33 ) := 33669 ;
V_ARRAY_SONG_IDS( 34 ) := 33671 ;
V_ARRAY_SONG_IDS( 35 ) := 33673 ;
V_ARRAY_SONG_IDS( 36 ) := 33675 ;
V_ARRAY_SONG_IDS( 37 ) := 33677 ;
V_ARRAY_SONG_IDS( 38 ) := 33679 ;
V_ARRAY_SONG_IDS( 39 ) := 33681 ;
V_ARRAY_SONG_IDS( 40 ) := 33683 ;
V_ARRAY_SONG_IDS( 41 ) := 33685 ;
V_ARRAY_SONG_IDS( 42 ) := 33700 ;
V_ARRAY_SONG_IDS( 43 ) := 33702 ;
V_ARRAY_SONG_IDS( 44 ) := 33704 ;
V_ARRAY_SONG_IDS( 45 ) := 33706 ;
V_ARRAY_SONG_IDS( 46 ) := 33708 ;
V_ARRAY_SONG_IDS( 47 ) := 33710 ;
V_ARRAY_SONG_IDS( 48 ) := 33712 ;
V_ARRAY_SONG_IDS( 49 ) := 33723 ;
V_ARRAY_SONG_IDS( 50 ) := 33725 ;
V_ARRAY_SONG_IDS( 51 ) := 33727 ;
V_ARRAY_SONG_IDS( 52 ) := 33729 ;
V_ARRAY_SONG_IDS( 53 ) := 33731 ;
V_ARRAY_SONG_IDS( 54 ) := 33733 ;
V_ARRAY_SONG_IDS( 55 ) := 33735 ;
V_ARRAY_SONG_IDS( 56 ) := 33737 ;
V_ARRAY_SONG_IDS( 57 ) := 33749 ;
V_ARRAY_SONG_IDS( 58 ) := 33751 ;
V_ARRAY_SONG_IDS( 59 ) := 33753 ;
V_ARRAY_SONG_IDS( 60 ) := 33755 ;
V_ARRAY_SONG_IDS( 61 ) := 33757 ;
V_ARRAY_SONG_IDS( 62 ) := 33759 ;
V_ARRAY_SONG_IDS( 63 ) := 33761 ;
V_ARRAY_SONG_IDS( 64 ) := 33763 ;
V_ARRAY_SONG_IDS( 65 ) := 33775 ;
V_ARRAY_SONG_IDS( 66 ) := 33777 ;
V_ARRAY_SONG_IDS( 67 ) := 33779 ;
V_ARRAY_SONG_IDS( 68 ) := 33781 ;
V_ARRAY_SONG_IDS( 69 ) := 33783 ;
V_ARRAY_SONG_IDS( 70 ) := 33785 ;
V_ARRAY_SONG_IDS( 71 ) := 33787 ;
V_ARRAY_SONG_IDS( 72 ) := 33789 ;
V_ARRAY_SONG_IDS( 73 ) := 33791 ;
V_ARRAY_SONG_IDS( 74 ) := 33793 ;
V_ARRAY_SONG_IDS( 75 ) := 33807 ;
V_ARRAY_SONG_IDS( 76 ) := 33809 ;
V_ARRAY_SONG_IDS( 77 ) := 33811 ;
V_ARRAY_SONG_IDS( 78 ) := 33813 ;
V_ARRAY_SONG_IDS( 79 ) := 33815 ;
V_ARRAY_SONG_IDS( 80 ) := 33817 ;
V_ARRAY_SONG_IDS( 81 ) := 33819 ;
V_ARRAY_SONG_IDS( 82 ) := 33821 ;
V_ARRAY_SONG_IDS( 83 ) := 33823 ;
V_ARRAY_SONG_IDS( 84 ) := 33825 ;
V_ARRAY_SONG_IDS( 85 ) := 33839 ;
V_ARRAY_SONG_IDS( 86 ) := 33844 ;
V_ARRAY_SONG_IDS( 87 ) := 33846 ;
V_ARRAY_SONG_IDS( 88 ) := 33848 ;
V_ARRAY_SONG_IDS( 89 ) := 33850 ;
V_ARRAY_SONG_IDS( 90 ) := 33852 ;
V_ARRAY_SONG_IDS( 91 ) := 33854 ;
V_ARRAY_SONG_IDS( 92 ) := 33856 ;
V_ARRAY_SONG_IDS( 93 ) := 33858 ;
V_ARRAY_SONG_IDS( 94 ) := 33860 ;
V_ARRAY_SONG_IDS( 95 ) := 33874 ;
V_ARRAY_SONG_IDS( 96 ) := 33879 ;
V_ARRAY_SONG_IDS( 97 ) := 33881 ;
V_ARRAY_SONG_IDS( 98 ) := 33883 ;
V_ARRAY_SONG_IDS( 99 ) := 33885 ;
V_ARRAY_SONG_IDS(100 ) := 33889 ;
/* do stuff with data from Song and Pricing Database coming from the two
separate cursors, then continue processing more rows...
FOR i IN C_get_music_price_data( v_array_song_ids ) LOOP
. (this is the loop where I pass in v_array_song_ids
. populated with only 100 cells and it runs forever)
END LOOP;
END; -
BULK COLLECT in select query inside a function
Hi All,
My query is :
SELECT col1,col2,col3 FROM table_a; --( consider this is a long running query with lot of joins)
Need to know how can i get the output of the above query from a function using BULK COLLECT.
and i tried this:
CREATE OR REPLACE TYPE tab_a_row
AS OBJECT (
col1 number(20),
col2 number(20),
col2 number(20)) ;
create or replace type tab_a_nt as table of tab_a_row;
create or replace function get_table_a
return sys_refcursor
is
tab_a_recs tab_a_nt;
rv sys_refcursor;
begin
SELECT tab_a_row(col1,col2,col3) BULK COLLECT INTO tab_a_recs FROM table_a;
open rv for select * from table(tab_a_recs);
return rv;
end;
Function created successfully. and i exec this from sql plus using
SQL> var rc refcursor;
SQL> exec :rc := get_table_a;
BEGIN :rc := get_table_a; END;
ERROR at line 1:
ORA-22905: cannot access rows from a non-nested table item
ORA-06512: at "GET_TABLE_A", line 12
ORA-06512: at line 1
Kindly share your ideas on how to use bulk collect and get set of outputs from a function.
Edited by: 887268 on Apr 18, 2013 3:10 AM>
If i use refcursor , then the JAVA code needs to be changed accordinglyto get the refcursor output.
>
Well, of course. Java has to know what the sql projection is. How else will it know how many columns there are and their datatypes.
But that is true no matter what method you use.
>
But if i use a PLSQL COLLECTION TYPE (nested tables ) , then i can get output as ,
select * from table(function_name(input1,input2));
>
No - using the 'table' function mean you are calling a PIPELINED function.
This is a sample of a PIPELINED procedure.
-- type to match emp record
create or replace type emp_scalar_type as object
(EMPNO NUMBER(4) ,
ENAME VARCHAR2(10),
JOB VARCHAR2(9),
MGR NUMBER(4),
HIREDATE DATE,
SAL NUMBER(7, 2),
COMM NUMBER(7, 2),
DEPTNO NUMBER(2)
-- table of emp records
create or replace type emp_table_type as table of emp_scalar_type
-- pipelined function
create or replace function get_emp( p_deptno in number )
return emp_table_type
PIPELINED
as
TYPE EmpCurTyp IS REF CURSOR RETURN emp%ROWTYPE;
emp_cv EmpCurTyp;
l_rec emp%rowtype;
begin
open emp_cv for select * from emp where deptno = p_deptno;
loop
fetch emp_cv into l_rec;
exit when (emp_cv%notfound);
pipe row( emp_scalar_type( l_rec.empno, LOWER(l_rec.ename),
l_rec.job, l_rec.mgr, l_rec.hiredate, l_rec.sal, l_rec.comm, l_rec.deptno ) );
end loop;
return;
end;
select * from table(get_emp(20))Or your function could return a collection like the example from this thread this morning.
Example of Collection as datatype of a function’s return value
CREATE OR REPLACE TYPE enamelist as VARRAY(20) of VARCHAR2(20)
/* Formatted on 4/18/2013 4:06:47 PM (QP5 v5.126.903.23003) */
CREATE OR REPLACE FUNCTION ename_fn
RETURN enamelist
AS
v_cursor_main enamelist := enamelist ();
BEGIN
SELECT ename
BULK COLLECT
INTO v_cursor_main
FROM emp;
RETURN v_cursor_main;
EXCEPTION
WHEN OTHERS
THEN
RETURN v_cursor_main;
END;
select * from table(ename_fn()) from dual;
COLUMN_VALUE
SMITH
ALLEN
WARD
JONES
MARTIN
BLAKE
CLARK
SCOTT
KING
TURNER
ADAMS
JAMES
FORD
MILLER -
Calling Stored procedure which uses Bulk Collect
Hi All, I have Oracle stored procedure which uses Bulk Collect and returns table type parameter as output. Can anyone please help me how Can I call this kind of stored procedures which returns table type output using VB and Oracle's Driver. (I am successfully able to call using MS ODBC driver, but I want to use OraOLEDB driver.)
861412 wrote:
how Can I call this kind of stored procedures which returns table type output using VB and Oracle's Driver. This forum deals with the server-side languages SQL and PL/SQL.
Your question deals with the client side and Visual Basic language. -
Bulk collect into using vector in where clause
Hi,
I have a java stored procedure which takes an array as input and returns an array as output.
I want to select a column in to a collection of array, using array as the select criteria.
For example
create or replace type My_list as table of varchar(20);
create or replace type outarray as table of BLOB;
create or replace function myfunction ( inputlist My_list)
return outarray
AS
o_data := outarray();
BEGIN
SELECT mycolumn from mytable BULK COLLECT INTO o_data where my_creteria=inputlist
retrun o_data;
END;
Unless I use FORALL I can not retrieve the inputlist values. I am not sure how I can use the bulk collect and forall in the same statement?
My data is arranged such that , for each value in the inputlist one row will be fetched. I want to fetch all these rows at one go using the inputlist as select criteria,
I can use the for i in inputlist.first .. inputlist.LAST and iterate the inputlist but that process is quite expensive interms of the latency.
You help in optimizing my query would be highly appreciated.
Regards,
SyamThat only leaves ugly solutions, like:
<ul>
<li>Using dynamic SQL to create a temporary table, doing a bulk insert of the incoming array and then reading the table values using a subquery with the IN opeartor.</li>
<li>Building a dyanmic list, like ('value1','value2', ..) and use it with the IN operator, this is a variable list and requires DBMS_SQL package.</li>
<li>Ignore the bulk operations. :-(</li>
</ul> -
I have a 10.2.0.4 database that contains a PL/SQL procedure that copies data from a singe remote 10.2.0.4 database table. The procedure will return anywhere from 50,000 to 500,000 rows of data. In testing I have made this a pretty speedy process using BULK COLLECT and using FORALL to load them in 2000 row batches. It has now been requested that I include an additional column from my source table which happens to be a CLOB datatype. However, when I try to perform this with the extra column I get the standard "cannot select remote lob locators" or whatever. Does anyone know of a way to perform this using BULK COLLECT? I've seen countless examples of doing it using "INSERT INTO TABLEX SELECT COL1, COL2, COL3, etc FROM TABLE Y@DBLINK". I don't want to do it this way for performance reasons. Any suggestions would be greatly appreciated.
sjm133 wrote:
I've seen countless examples of doing it using "INSERT INTO TABLEX SELECT COL1, COL2, COL3, etc FROM TABLE Y@DBLINK". I don't want to do it this way for performance reasons.What performance reasons are those then?
Best thing to do would be to give it a go and see, and then if you find problems with it look for alternatives. Don't dismiss solutions without trying them. ;) -
Bulk collect / forall which collection type?
Hi I am trying to speed up the query below using bulk collect / forall:
SELECT h.cust_order_no AS custord, l.shipment_set AS sset
FROM info.tlp_out_messaging_hdr h, info.tlp_out_messaging_lin l
WHERE h.message_id = l.message_id
AND h.contract = '12384'
AND l.shipment_set IS NOT NULL
AND h.cust_order_no IS NOT NULL
GROUP BY h.cust_order_no, l.shipment_set
I would like to extract the 2 fields selected above into a new table as quickly as possible, but I’m fairly new to Oracle and I’m finding it difficult to sort out the best way to do it. The query below does not work (no doubt there are numerous issues) but hopefully it is sufficiently developed to shows the sort of thing I am trying to achieve:
DECLARE
TYPE xcustord IS TABLE OF info.tlp_out_messaging_hdr.cust_order_no%TYPE;
TYPE xsset IS TABLE OF info.tlp_out_messaging_lin.shipment_set%TYPE;
TYPE xarray IS TABLE OF tp_a1_tab%rowtype INDEX BY BINARY_INTEGER;
v_xarray xarray;
v_xcustord xcustord;
v_xsset xsset;
CURSOR cur IS
SELECT h.cust_order_no AS custord, l.shipment_set AS sset
FROM info.tlp_out_messaging_hdr h, info.tlp_out_messaging_lin l
WHERE h.message_id = l.message_id
AND h.contract = '1111'
AND l.shipment_set IS NOT NULL
AND h.cust_order_no IS NOT NULL;
BEGIN
OPEN cur;
LOOP
FETCH cur
BULK COLLECT INTO v_xarray LIMIT 10000;
EXIT WHEN v_xcustord.COUNT() = 0;
FORALL i IN 1 .. v_xarray.COUNT
INSERT INTO TP_A1_TAB (cust_order_no, shipment_set)
VALUES (v_xarray(i).cust_order_no,v_xarray(i).shipment_set);
commit;
END LOOP;
CLOSE cur;
END;
I am running on Oracle 9i release 2.Well I suppose I can share the whole query as it stands for context and information (see below).
This is a very ugly piece of code that I am trying to improve. The advantage it has currently is
that it works, the disadvantage it has is that it's very slow. My thoughts were that bulk collect
and forall might be useful tools here as part of the re-write hence my original question, but perhaps not.
So on a more general note any advice on how best speed up this code would be welcome:
CREATE OR REPLACE VIEW DLP_TEST AS
WITH aa AS
(SELECT h.cust_order_no AS c_ref,l.shipment_set AS shipset,
l.line_id, h.message_id, l.message_line_no,
l.vendor_part_no AS part, l.rqst_quantity AS rqst_qty, l.quantity AS qty,
l.status AS status, h.rowversion AS allocation_date
FROM info.tlp_in_messaging_hdr h
LEFT JOIN info.tlp_in_messaging_lin l
ON h.message_id = l.message_id
WHERE h.contract = '12384'
AND h.cust_order_no IS NOT NULL
UNION ALL
SELECT ho.cust_order_no AS c_ref, lo.shipment_set AS shipset,
lo.line_id, ho.message_id, lo.message_line_no,
lo.vendor_part_no AS part,lo.rqst_quantity AS rqst_qty, lo.quantity AS qty,
lo.status AS status, ho.rowversion AS allocation_date
FROM info.tlp_out_messaging_hdr ho, info.tlp_out_messaging_lin lo
WHERE ho.message_id = lo.message_id
AND ho.contract = '12384'
AND ho.cust_order_no IS NOT NULL),
a1 AS
(SELECT h.cust_order_no AS custord, l.shipment_set AS sset
FROM info.tlp_out_messaging_hdr h, info.tlp_out_messaging_lin l
WHERE h.message_id = l.message_id
AND h.contract = '12384'
AND l.shipment_set IS NOT NULL
AND h.cust_order_no IS NOT NULL
GROUP BY h.cust_order_no, l.shipment_set),
a2 AS
(SELECT ho.cust_order_no AS c_ref, lo.shipment_set AS shipset
FROM info.tlp_out_messaging_hdr ho, info.tlp_out_messaging_lin lo
WHERE ho.message_id = lo.message_id
AND ho.contract = '12384'
AND ho.message_type = '3B13'
AND lo.shipment_set IS NOT NULL
GROUP BY ho.cust_order_no, lo.shipment_set),
a3 AS
(SELECT a1.custord, a1.sset, CONCAT('SHIPSET',a1.sset) AS ssset
FROM a1
LEFT OUTER JOIN a2
ON a1.custord = a2.c_ref AND a1.sset = a2.shipset
WHERE a2.c_ref IS NULL),
bb AS
(SELECT so.contract, so.order_no, sr.service_request_no AS sr_no, sr.reference_no,
substr(sr.part_no,8) AS shipset,
substr(note_text,1,instr(note_text,'.',1,1)-1) AS Major_line,
note_text AS CISCO_line,ma.part_no,
(Select TO_DATE(TO_CHAR(TO_DATE(d.objversion,'YYYYMMDDHH24MISS'),'YYYY-MM-DD HH24:MI:SS'),'YYYY-MM-DD HH24:MI:SS')
FROM ifsapp.document_text d WHERE so.note_id = d.note_id AND so.contract = '12384') AS Print_Date,
ma.note_text
FROM (ifsapp.service_request sr
LEFT OUTER JOIN ifsapp.work_order_shop_ord ws
ON sr.service_request_no = ws.wo_no
LEFT OUTER JOIN ifsapp.shop_ord so
ON ws.order_no = so.order_no)
LEFT OUTER JOIN ifsapp.customer_order co
ON sr.reference_no = co.cust_ref
LEFT OUTER JOIN ifsapp.shop_material_alloc ma
ON so.order_no = ma.order_no
JOIN a3
ON a3.custord = sr.reference_no AND a3.sset = substr(sr.part_no,8)
WHERE sr.part_contract = '12384'
AND so.contract = '12384'
AND co.contract = '12384'
AND sr.reference_no IS NOT NULL
AND ma.part_no NOT LIKE 'SHIPSET%'),
cc AS
(SELECT
bb.reference_no,
bb.shipset,
bb.order_no,
bb.cisco_line,
aa.message_id,
aa.allocation_date,
row_number() over(PARTITION BY bb.reference_no, bb.shipset, aa.allocation_date
ORDER BY bb.reference_no, bb.shipset, aa.allocation_date, aa.message_id DESC) AS selector
FROM bb
LEFT OUTER JOIN aa
ON bb.reference_no = aa.c_ref AND bb.shipset = aa.shipset
WHERE aa.allocation_date <= bb.print_date
OR aa.allocation_date IS NULL
OR bb.print_date IS NULL),
dd AS
(SELECT
MAX(reference_no) AS reference_no,
MAX(shipset) AS shipset,
order_no,
MAX(allocation_date) AS allocation_date
FROM cc
WHERE selector = 1
GROUP BY order_no, selector),
ee AS
(SELECT
smx.order_no,
SUM(smx.qty_assigned) AS total_allocated,
SUM(smx.qty_issued) AS total_issued,
SUM(smx.qty_required) AS total_required
FROM ifsapp.shop_material_alloc smx
WHERE smx.contract = '12384'
AND smx.part_no NOT LIKE 'SHIPSET%'
GROUP BY smx.order_no),
ff AS
(SELECT
dd.reference_no,
dd.shipset,
dd.order_no,
MAX(allocation_date) AS last_allocation,
MAX(ee.total_allocated) AS total_allocated,
MAX(ee.total_issued) AS total_issued,
MAX(ee.total_required) AS total_required
FROM dd
LEFT OUTER JOIN ee
ON dd.order_no = ee.order_no
GROUP BY dd.reference_no, dd.shipset, dd.order_no),
base AS
(SELECT x.order_no, x.part_no, z.rel_no, MIN(x.dated) AS dated, MIN(y.cust_ref) AS cust_ref, MIN(z.line_no) AS line_no,
MIN(y.state) AS state, MIN(y.contract) AS contract, MIN(z.demand_order_ref1) AS demand_order_ref1
FROM ifsapp.inventory_transaction_hist x, ifsapp.customer_order y, ifsapp.customer_order_line z
WHERE x.contract = '12384'
AND x.order_no = y.order_no
AND y.order_no = z.order_no
AND x.transaction = 'REP-OESHIP'
AND x.part_no = z.part_no
AND TRUNC(x.dated) >= SYSDATE - 8
GROUP BY x.order_no, x.part_no, z.rel_no)
SELECT
DISTINCT
bb.contract,
bb.order_no,
bb.sr_no,
CAST('-' AS varchar2(40)) AS Usr,
CAST('-' AS varchar2(40)) AS Name,
CAST('01' AS number) AS Operation,
CAST('Last Reservation' AS varchar2(40)) AS Action,
CAST('-' AS varchar2(40)) AS Workcenter,
CAST('-' AS varchar2(40)) AS Next_Workcenter_no,
CAST('Print SO' AS varchar2(40)) AS Next_WC_Description,
ff.total_allocated,
ff.total_issued,
ff.total_required,
ff.shipset,
ff.last_allocation AS Action_date,
ff.reference_no
FROM ff
LEFT OUTER JOIN bb
ON bb.order_no = ff.order_no
WHERE bb.order_no IS NOT NULL
UNION ALL
SELECT
c.contract AS Site, c.order_no AS Shop_Order, b.wo_no AS SR_No,
CAST('-' AS varchar2(40)) AS UserID,
CAST('-' AS varchar2(40)) AS User_Name,
CAST('02' AS number) AS Operation,
CAST('SO Printed' AS varchar2(40)) AS Action,
CAST('SOPRINT' AS varchar2(40)) AS Workcenter,
CAST('PKRPT' AS varchar2(40)) AS Next_Workcenter_no,
CAST('Pickreport' AS varchar2(40)) AS Next_WC_Description,
CAST('0' AS number) AS Total_Allocated,
CAST('0' AS number) AS Total_Issued,
CAST('0' AS number) AS Total_Required,
e.part_no AS Ship_Set,
TO_DATE(TO_CHAR(TO_DATE(d.objversion,'YYYYMMDDHH24MISS'),'YYYY-MM-DD HH24:MI:SS'),'YYYY-MM-DD HH24:MI:SS')AS Action_Date,
f.cust_ref AS cust_ref
FROM ifsapp.shop_ord c
LEFT OUTER JOIN ifsapp.work_order_shop_ord b
ON b.order_no = c.order_no
LEFT OUTER JOIN ifsapp.document_text d
ON d.note_id = c.note_id
LEFT OUTER JOIN ifsapp.customer_order_line e
ON e.demand_order_ref1 = TRIM(to_char(b.wo_no))
LEFT OUTER JOIN ifsapp.customer_order f
ON f.order_no = e.order_no
JOIN a3
ON a3.custord = f.cust_ref AND a3.ssset = e.part_no
WHERE c.contract = '12384'
AND e.contract = '12384'
AND d.objversion IS NOT NULL
UNION ALL
SELECT
a.site AS Site, a.order_no AS Shop_Order, b.wo_no AS SR_No, a.userid AS UserID,
a.user_name AS Name, a.operation_no AS Operation, a.action AS Action,
a.workcenter_no AS Workcenter, a.next_work_center_no AS Next_Workcenter_no,
(SELECT d.description FROM ifsapp.work_center d WHERE a.next_work_center_no = d.work_center_no AND a.site = d.contract)
AS Next_WC_Description,
CAST('0' AS number) AS Total_Allocated,
CAST('0' AS number) AS Total_Issued,
CAST('0' AS number) AS Total_Required,
e.part_no AS Ship_set,
a.action_date AS Action_Date, f.cust_ref AS cust_ref
FROM ifsapp.shop_ord c
LEFT OUTER JOIN ifsapp.work_order_shop_ord b
ON b.order_no = c.order_no
LEFT OUTER JOIN ifsapp.customer_order_line e
ON e.demand_order_ref1 = to_char(b.wo_no)
LEFT OUTER JOIN ifsapp.customer_order f
ON f.order_no = e.order_no
LEFT OUTER JOIN info.tp_hvt_so_op_hist a
ON a.order_no = c.order_no
JOIN a3
ON a3.custord = f.cust_ref AND a3.ssset = e.part_no
WHERE a.site = '12384'
AND c.contract = '12384'
AND e.contract = '12384'
AND f.contract = '12384'
UNION ALL
SELECT so.contract AS Site, so.order_no AS Shop_Order_No, sr.service_request_no AS SR_No,
CAST('-' AS varchar2(40)) AS "User",
CAST('-' AS varchar2(40)) AS "Name",
CAST('999' AS number) AS "Operation",
CAST('Shipped' AS varchar2(40)) AS "Action",
CAST('SHIP' AS varchar2(40)) AS "Workcenter",
CAST('-' AS varchar2(40)) AS "Next_Workcenter_no",
CAST('-' AS varchar2(40)) AS "Next_WC_Description",
CAST('0' AS number) AS Total_Allocated,
CAST('0' AS number) AS Total_Issued,
CAST('0' AS number) AS Total_Required,
so.part_no AS ship_set, base.dated AS Action_Date,
sr.reference_no AS CUST_REF
FROM base
LEFT OUTER JOIN ifsapp.service_request sr
ON base.demand_order_ref1 = sr.service_request_no
LEFT OUTER JOIN ifsapp.work_order_shop_ord ws
ON sr.service_request_no = ws.wo_no
LEFT OUTER JOIN ifsapp.shop_ord so
ON ws.order_no = so.order_no
WHERE base.contract = '12384'; -
Problem with BULK COLLECT with million rows - Oracle 9.0.1.4
We have a requirement where are supposed to load 58 millions of rows into a FACT Table in our DATA WAREHOUSE. We initially planned to use Oracle Warehouse Builder but due to performance reasons, decided to write custom code. We wrote a custome procedure which opens a simple cursor and reads all the 58 million rows from the SOURCE Table and in a loop processes the rows and inserts the records into a TARGET Table. The logic works fine but it took 20hrs to complete the load.
We then tried to leverage the BULK COLLECT and FORALL and PARALLEL options and modified our PL/SQL code completely to reflect these. Our code looks very simple.
1. We declared PL/SQL BINARY_INDEXed Tables to store the data in memory.
2. We used BULK COLLECT into FETCH the data.
3. We used FORALL statement while inserting the data.
We did not introduce any of our transformation logic yet.
We tried with the 600,000 records first and it completed in 1 min and 29 sec with no problems. We then doubled the no. of rows to 1.2 million and the program crashed with the following error:
ERROR at line 1:
ORA-04030: out of process memory when trying to allocate 16408 bytes (koh-kghu
call ,pmucalm coll)
ORA-06512: at "VVA.BULKLOAD", line 66
ORA-06512: at line 1
We got the same error even with 1 million rows.
We do have the following configuration:
SGA - 8.2 GB
PGA
- Aggregate Target - 3GB
- Current Allocated - 439444KB (439 MB)
- Maximum allocated - 2695753 KB (2.6 GB)
Temp Table Space - 60.9 GB (Total)
- 20 GB (Available approximately)
I think we do have more than enough memory to process the 1 million rows!!
Also, some times the same program results in the following error:
SQL> exec bulkload
BEGIN bulkload; END;
ERROR at line 1:
ORA-03113: end-of-file on communication channel
We did not even attempt the full load. Also, we are not using the PARALLEL option yet.
Are we hitting any bug here? Or PL/SQL is not capable of mass loads? I would appreciate any thoughts on this?
Thanks,
Haranadh
Following is the code:
set echo off
set timing on
create or replace procedure bulkload as
-- SOURCE --
TYPE src_cpd_dt IS TABLE OF ima_ama_acct.cpd_dt%TYPE;
TYPE src_acqr_ctry_cd IS TABLE OF ima_ama_acct.acqr_ctry_cd%TYPE;
TYPE src_acqr_pcr_ctry_cd IS TABLE OF ima_ama_acct.acqr_pcr_ctry_cd%TYPE;
TYPE src_issr_bin IS TABLE OF ima_ama_acct.issr_bin%TYPE;
TYPE src_mrch_locn_ref_id IS TABLE OF ima_ama_acct.mrch_locn_ref_id%TYPE;
TYPE src_ntwrk_id IS TABLE OF ima_ama_acct.ntwrk_id%TYPE;
TYPE src_stip_advc_cd IS TABLE OF ima_ama_acct.stip_advc_cd%TYPE;
TYPE src_authn_resp_cd IS TABLE OF ima_ama_acct.authn_resp_cd%TYPE;
TYPE src_authn_actvy_cd IS TABLE OF ima_ama_acct.authn_actvy_cd%TYPE;
TYPE src_resp_tm_id IS TABLE OF ima_ama_acct.resp_tm_id%TYPE;
TYPE src_mrch_ref_id IS TABLE OF ima_ama_acct.mrch_ref_id%TYPE;
TYPE src_issr_pcr IS TABLE OF ima_ama_acct.issr_pcr%TYPE;
TYPE src_issr_ctry_cd IS TABLE OF ima_ama_acct.issr_ctry_cd%TYPE;
TYPE src_acct_num IS TABLE OF ima_ama_acct.acct_num%TYPE;
TYPE src_tran_cnt IS TABLE OF ima_ama_acct.tran_cnt%TYPE;
TYPE src_usd_tran_amt IS TABLE OF ima_ama_acct.usd_tran_amt%TYPE;
src_cpd_dt_array src_cpd_dt;
src_acqr_ctry_cd_array src_acqr_ctry_cd;
src_acqr_pcr_ctry_cd_array src_acqr_pcr_ctry_cd;
src_issr_bin_array src_issr_bin;
src_mrch_locn_ref_id_array src_mrch_locn_ref_id;
src_ntwrk_id_array src_ntwrk_id;
src_stip_advc_cd_array src_stip_advc_cd;
src_authn_resp_cd_array src_authn_resp_cd;
src_authn_actvy_cd_array src_authn_actvy_cd;
src_resp_tm_id_array src_resp_tm_id;
src_mrch_ref_id_array src_mrch_ref_id;
src_issr_pcr_array src_issr_pcr;
src_issr_ctry_cd_array src_issr_ctry_cd;
src_acct_num_array src_acct_num;
src_tran_cnt_array src_tran_cnt;
src_usd_tran_amt_array src_usd_tran_amt;
j number := 1;
CURSOR c1 IS
SELECT
cpd_dt,
acqr_ctry_cd ,
acqr_pcr_ctry_cd,
issr_bin,
mrch_locn_ref_id,
ntwrk_id,
stip_advc_cd,
authn_resp_cd,
authn_actvy_cd,
resp_tm_id,
mrch_ref_id,
issr_pcr,
issr_ctry_cd,
acct_num,
tran_cnt,
usd_tran_amt
FROM ima_ama_acct ima_ama_acct
ORDER BY issr_bin;
BEGIN
OPEN c1;
FETCH c1 bulk collect into
src_cpd_dt_array ,
src_acqr_ctry_cd_array ,
src_acqr_pcr_ctry_cd_array,
src_issr_bin_array ,
src_mrch_locn_ref_id_array,
src_ntwrk_id_array ,
src_stip_advc_cd_array ,
src_authn_resp_cd_array ,
src_authn_actvy_cd_array ,
src_resp_tm_id_array ,
src_mrch_ref_id_array ,
src_issr_pcr_array ,
src_issr_ctry_cd_array ,
src_acct_num_array ,
src_tran_cnt_array ,
src_usd_tran_amt_array ;
CLOSE C1;
FORALL j in 1 .. src_cpd_dt_array.count
INSERT INTO ima_dly_acct (
CPD_DT,
ACQR_CTRY_CD,
ACQR_TIER_CD,
ACQR_PCR_CTRY_CD,
ACQR_PCR_TIER_CD,
ISSR_BIN,
OWNR_BUS_ID,
USER_BUS_ID,
MRCH_LOCN_REF_ID,
NTWRK_ID,
STIP_ADVC_CD,
AUTHN_RESP_CD,
AUTHN_ACTVY_CD,
RESP_TM_ID,
PROD_REF_ID,
MRCH_REF_ID,
ISSR_PCR,
ISSR_CTRY_CD,
ACCT_NUM,
TRAN_CNT,
USD_TRAN_AMT)
VALUES (
src_cpd_dt_array(j),
src_acqr_ctry_cd_array(j),
null,
src_acqr_pcr_ctry_cd_array(j),
null,
src_issr_bin_array(j),
null,
null,
src_mrch_locn_ref_id_array(j),
src_ntwrk_id_array(j),
src_stip_advc_cd_array(j),
src_authn_resp_cd_array(j),
src_authn_actvy_cd_array(j),
src_resp_tm_id_array(j),
null,
src_mrch_ref_id_array(j),
src_issr_pcr_array(j),
src_issr_ctry_cd_array(j),
src_acct_num_array(j),
src_tran_cnt_array(j),
src_usd_tran_amt_array(j));
COMMIT;
END bulkload;
SHOW ERRORS
-----------------------------------------------------------------------------do you have a unique key available in the rows you are fetching?
It seems a cursor with 20 million rows that is as wide as all the columnsyou want to work with is a lot of memory for the server to use at once. You may be able to do this with parallel processing (dop over 8) and a lot of memory for the warehouse box (and the box you are extracting data from)...but is this the most efficient (and thereby fastest) way to do it?
What if you used a cursor to select a unique key only, and then during the cursor loop fetch each record, transform it, and insert it into the target?
Its a different way to do a lot at once, but it cuts down on the overall memory overhead for the process.
I know this isnt as elegant as a single insert to do it all at once, but sometimes trimming a process down so it takes less resources at any given moment is much faster than trying to do the whole thing at once.
My solution is probably biased by transaction systems, so I would be interested in what the data warehouse community thinks of this.
For example:
source table my_transactions (tx_seq_id number, tx_fact1 varchar2(10), tx_fact2 varchar2(20), tx_fact3 number, ...)
select a cursor of tx_seq_id only (even at 20 million rows this is not much)
you could then either use a for loop or even bulk collect into a plsql collection or table
then process individually like this:
procedure process_a_tx(p_tx_seq_id in number)
is
rTX my_transactions%rowtype;
begin
select * into rTX from my_transactions where tx_seq_id = p_tx_seq_id;
--modify values as needed
insert into my_target(a, b, c) values (rtx.fact_1, rtx.fact2, rtx.fact3);
commit;
exception
when others
rollback;
--write to a log or raise and exception
end process_a_tx;
procedure collect_tx
is
cursor tx is
select tx_seq_id from my_transactions;
begin
for rTx in cTx loop
process_a_tx(rtx.tx_seq_id);
end loop;
end collect_tx;
Maybe you are looking for
-
How to track the Inbound child IDOcs in PI 7.4 Single stack
Hi Friends, I would like to know the procedure of tracking the Inbound child IDocs in PI 7.4 Single stack. Here is the detailed description. Say we are getting the data from third party system and the message may have multiple IDOcs in it. The messag
-
My iPhone 4 rebooting while hooked up to charge after fall.
My iPhone 4 recetly fell from pocket to my foot and turned off right away, now it's only rebooting while connected to a charging source; but only continuous reboots. We can't connect to restore it to iTunes because the home button doesn't work as wel
-
How can I recover these saved emails???
-
Downloading nano CD messed up ITunes!
I have a mini ipod & ibook g4. Bought a nano tonight. Didn't download new nano cd, went to updatd nano to my libary (660 songs), but it crashed half way through. So I inserted nano cd, but it wouldn't allow me to install (said components were disable
-
Hello Everyone, We present our latest LCCS SDK 1.1.1 release. Our primary goal for this release has been consolidating on big features we released in mid April as we move forward. We took special care about some user issues raised in last one month.