10g: parallel pipelined table func. using table(cast(SQL collect.))?

Question

Hi,
i try to distribute SQL data objects - stored in a SQL data type TABLE OF <object-Type> - to multiple (parallel) instances of a table function,
by passing a CURSOR(...) to the table function, which selects from the SQL TABLE OF storage via "select * from TABLE(CAST(<storage> as <storage-type>)".
But oracle always only uses a single table function instance :-(
whatever hints i provide or setting i use for the parallel table function (parallel_enable ...)
Could it be, that this is due to the fact, that my data are not
globally available, but only in the main thread data?
Can someone confirm, that it's not possible to start multiple parallel table functions
for selecting on SQL data type TABLE OF <object>storages?
Here's an example sqlplus program to show the issue:
-------------------- snip ---------------------------------------------
set serveroutput on;
drop table test_table;
drop type ton_t;
drop type test_list;
drop type test_obj;
create table test_table
     a number(19,0),
     b timestamp with time zone,
     c varchar2(256)
create or replace type test_obj as object(
     a number(19,0),
     b timestamp with time zone,
     c varchar2(256)
create or replace type test_list as table of test_obj;
create or replace type ton_t as table of number;
create or replace package test_pkg
as
     type test_rec is record (
          a number(19,0),
          b timestamp with time zone,
          c varchar2(256)
     type test_tab is table of test_rec;
     type test_cur is ref cursor return test_rec;
     function TF(mycur test_cur)
return test_list pipelined
parallel_enable(partition mycur by hash(a));
end;
create or replace package body test_pkg
as
     function TF(mycur test_cur)
return test_list pipelined
parallel_enable(partition mycur by hash(a))
is
          sid number;
          counter number(19,0) := 0;
          myrec test_rec;
          mytab test_tab;
          mytab2 test_list := test_list();
     begin
          select userenv('SID') into sid from dual;
          dbms_output.put_line('test_pkg.TF( sid => '''|| sid || ''' ): enter');
          loop
               fetch mycur into myRec;
               exit when mycur%NOTFOUND;
               mytab2.extend;
               mytab2(mytab2.last) := test_obj(myRec.a, myRec.b, myRec.c);
          end loop;
          for i in mytab2.first..mytab2.last loop
               -- attention: saves own SID in test_obj.a for indication to caller
               --     how many sids have been involved
               pipe row(test_obj(sid, mytab2(i).b, mytab2(i).c));
               counter := counter + 1;
          end loop;
          dbms_output.put_line('test_pkg.TF( sid => '''|| sid || ''' ): exit, piped #' || counter || ' records');
     end;
end;
declare
     myList test_list := test_list();
     myList2 test_list := test_list();
     sids ton_t := ton_t();
begin
     for i in 1..10000 loop
          myList.extend; myList(myList.last) := test_obj(i, sysdate, to_char(i+2));
     end loop;
     -- save into the real table
     insert into test_table select * from table(cast (myList as test_list));
     dbms_output.put_line(chr(10) || 'copy ''mylist'' to ''mylist2'' by streaming via table function...');
     select test_obj(a, b, c) bulk collect into myList2
     from table(test_pkg.TF(CURSOR(select /*+ parallel(tab,10) */ * from table(cast (myList as test_list)) tab)));
     dbms_output.put_line('... saved #' || myList2.count || ' records');
     select distinct(tab.a) bulk collect into sids from table(cast (myList2 as test_list)) tab;
     dbms_output.put_line('worker thread''s sid list:');
     for i in sids.first..sids.last loop
          dbms_output.put_line('sid #' || sids(i));
     end loop;
     dbms_output.put_line(chr(10) || 'copy physical ''test_table'' to ''mylist2'' by streaming via table function:');
     select test_obj(a, b, c) bulk collect into myList2
     from table(test_pkg.TF(CURSOR(select /*+ parallel(tab,10) */ * from test_table tab)));
     dbms_output.put_line('... saved #' || myList2.count || ' records');
     select distinct(tab.a) bulk collect into sids from table(cast (myList2 as test_list)) tab;
     dbms_output.put_line('worker thread''s sid list:');
     for i in sids.first..sids.last loop
          dbms_output.put_line('sid #' || sids(i));
     end loop;
end;
-------------------- snap ---------------------------------------------
Here's the output:
-------------------- snip ---------------------------------------------
copy 'mylist' to 'mylist2' by streaming via table function...
test_pkg.TF( sid => '98' ): enter
test_pkg.TF( sid => '98' ): exit, piped #10000 records
... saved #10000 records
worker thread's sid list:
sid #98 -- ONLY A SINGLE SID HERE!
copy physical 'test_table' to 'mylist2' by streaming via table function:
... saved #10000 records
worker thread's sid list:
sid #128 -- A LIST OF SIDS HERE!
sid #141
sid #85
sid #125
sid #254
sid #101
sid #124
sid #109
sid #142
sid #92
PL/SQL procedure successfully completed.
-------------------- snap ---------------------------------------------
I posted it to newsgroup comp.databases.oracle.server.
(summary: "10g: parallel pipelined table functions with cursor selecting from table(cast(SQL collection)) doesn't work ")
But i didn't get a response.
There i also wrote some background information about my application:
-------------------- snip ---------------------------------------------
My application has a #2 steps/stages data selection.
A 1st select for minimal context base data
- mainly to evaluate for due driving data records.
And a 2nd select for all the "real" data to process a context
(joining much more other tables here, which i don't want to do for non-due records).
So it's doing stage #1 select first, then stage #2 select - based on stage #1 results - next.
The first implementation of the application did the stage #1 select in the main session of the pl/sql code.
And for the stage #2 select there was done a dispatch to multiple parallel table functions (in multiple worker sessions) for the "real work".
That worked.
However there was a flaw:
Between records from stage #1 selection and records from stage #2 selection there is a 1:n relation (via key / foreign key relation).
Means, for #1 resulting record from stage #1 selection, there are #x records from stage #2 selection.
That forced me to use "cluster curStage2 by (theKey)".
Because the worker sessions need to evaluate the all-over status for a context of #1 record from stage #1 and #x records from stage #2
(so it needs to have #x records of stage #2 together).
This then resulted in delay for starting up the worker sessions (i didn't find a way to get rid of this).
So i wanted to shift the invocation of the worker sessions to the stage #1 selection.
Then i don't need the "cluster curStage2 by (theKey)" anymore!
But: i also need to do an update of the primary driving data!
So the stage #1 select is a 'select ... for update ...'.
But you can't use such in CURSOR for table functions (which i can understand, why it's not possible).
So i have to do my stage #1 selection in two steps:
1. 'select for update' by main session and collect result in SQL collection.
2. pass collected data to parallel table functions
And for 2. i recognized, that it doesn't start up multiple parallel table function instances.
As a work-around
- if it's just not possible to start multiple parallel pipelined table functions for dispatching from 'select * from TABLE(CAST(... as ...))' -
i need to select again on the base tables - driven by the SQL collection data.
But before i do so, i wanted to verify, if it's really not possible.
Maybe i just miss a special oracle hint or whatever you can get "out of another box" :-)
-------------------- snap ---------------------------------------------
- many thanks!
rgds,
Frank

Answer 1

Hi,
i try to distribute SQL data objects - stored in a SQL data type TABLE OF <object-Type> - to multiple (parallel) instances of a table function,
by passing a CURSOR(...) to the table function, which selects from the SQL TABLE OF storage via "select * from TABLE(CAST(<storage> as <storage-type>)".
But oracle always only uses a single table function instance :-(
whatever hints i provide or setting i use for the parallel table function (parallel_enable ...)
Could it be, that this is due to the fact, that my data are not
globally available, but only in the main thread data?
Can someone confirm, that it's not possible to start multiple parallel table functions
for selecting on SQL data type TABLE OF <object>storages?
Here's an example sqlplus program to show the issue:
-------------------- snip ---------------------------------------------
set serveroutput on;
drop table test_table;
drop type ton_t;
drop type test_list;
drop type test_obj;
create table test_table
     a number(19,0),
     b timestamp with time zone,
     c varchar2(256)
create or replace type test_obj as object(
     a number(19,0),
     b timestamp with time zone,
     c varchar2(256)
create or replace type test_list as table of test_obj;
create or replace type ton_t as table of number;
create or replace package test_pkg
as
     type test_rec is record (
          a number(19,0),
          b timestamp with time zone,
          c varchar2(256)
     type test_tab is table of test_rec;
     type test_cur is ref cursor return test_rec;
     function TF(mycur test_cur)
return test_list pipelined
parallel_enable(partition mycur by hash(a));
end;
create or replace package body test_pkg
as
     function TF(mycur test_cur)
return test_list pipelined
parallel_enable(partition mycur by hash(a))
is
          sid number;
          counter number(19,0) := 0;
          myrec test_rec;
          mytab test_tab;
          mytab2 test_list := test_list();
     begin
          select userenv('SID') into sid from dual;
          dbms_output.put_line('test_pkg.TF( sid => '''|| sid || ''' ): enter');
          loop
               fetch mycur into myRec;
               exit when mycur%NOTFOUND;
               mytab2.extend;
               mytab2(mytab2.last) := test_obj(myRec.a, myRec.b, myRec.c);
          end loop;
          for i in mytab2.first..mytab2.last loop
               -- attention: saves own SID in test_obj.a for indication to caller
               --     how many sids have been involved
               pipe row(test_obj(sid, mytab2(i).b, mytab2(i).c));
               counter := counter + 1;
          end loop;
          dbms_output.put_line('test_pkg.TF( sid => '''|| sid || ''' ): exit, piped #' || counter || ' records');
     end;
end;
declare
     myList test_list := test_list();
     myList2 test_list := test_list();
     sids ton_t := ton_t();
begin
     for i in 1..10000 loop
          myList.extend; myList(myList.last) := test_obj(i, sysdate, to_char(i+2));
     end loop;
     -- save into the real table
     insert into test_table select * from table(cast (myList as test_list));
     dbms_output.put_line(chr(10) || 'copy ''mylist'' to ''mylist2'' by streaming via table function...');
     select test_obj(a, b, c) bulk collect into myList2
     from table(test_pkg.TF(CURSOR(select /*+ parallel(tab,10) */ * from table(cast (myList as test_list)) tab)));
     dbms_output.put_line('... saved #' || myList2.count || ' records');
     select distinct(tab.a) bulk collect into sids from table(cast (myList2 as test_list)) tab;
     dbms_output.put_line('worker thread''s sid list:');
     for i in sids.first..sids.last loop
          dbms_output.put_line('sid #' || sids(i));
     end loop;
     dbms_output.put_line(chr(10) || 'copy physical ''test_table'' to ''mylist2'' by streaming via table function:');
     select test_obj(a, b, c) bulk collect into myList2
     from table(test_pkg.TF(CURSOR(select /*+ parallel(tab,10) */ * from test_table tab)));
     dbms_output.put_line('... saved #' || myList2.count || ' records');
     select distinct(tab.a) bulk collect into sids from table(cast (myList2 as test_list)) tab;
     dbms_output.put_line('worker thread''s sid list:');
     for i in sids.first..sids.last loop
          dbms_output.put_line('sid #' || sids(i));
     end loop;
end;
-------------------- snap ---------------------------------------------
Here's the output:
-------------------- snip ---------------------------------------------
copy 'mylist' to 'mylist2' by streaming via table function...
test_pkg.TF( sid => '98' ): enter
test_pkg.TF( sid => '98' ): exit, piped #10000 records
... saved #10000 records
worker thread's sid list:
sid #98 -- ONLY A SINGLE SID HERE!
copy physical 'test_table' to 'mylist2' by streaming via table function:
... saved #10000 records
worker thread's sid list:
sid #128 -- A LIST OF SIDS HERE!
sid #141
sid #85
sid #125
sid #254
sid #101
sid #124
sid #109
sid #142
sid #92
PL/SQL procedure successfully completed.
-------------------- snap ---------------------------------------------
I posted it to newsgroup comp.databases.oracle.server.
(summary: "10g: parallel pipelined table functions with cursor selecting from table(cast(SQL collection)) doesn't work ")
But i didn't get a response.
There i also wrote some background information about my application:
-------------------- snip ---------------------------------------------
My application has a #2 steps/stages data selection.
A 1st select for minimal context base data
- mainly to evaluate for due driving data records.
And a 2nd select for all the "real" data to process a context
(joining much more other tables here, which i don't want to do for non-due records).
So it's doing stage #1 select first, then stage #2 select - based on stage #1 results - next.
The first implementation of the application did the stage #1 select in the main session of the pl/sql code.
And for the stage #2 select there was done a dispatch to multiple parallel table functions (in multiple worker sessions) for the "real work".
That worked.
However there was a flaw:
Between records from stage #1 selection and records from stage #2 selection there is a 1:n relation (via key / foreign key relation).
Means, for #1 resulting record from stage #1 selection, there are #x records from stage #2 selection.
That forced me to use "cluster curStage2 by (theKey)".
Because the worker sessions need to evaluate the all-over status for a context of #1 record from stage #1 and #x records from stage #2
(so it needs to have #x records of stage #2 together).
This then resulted in delay for starting up the worker sessions (i didn't find a way to get rid of this).
So i wanted to shift the invocation of the worker sessions to the stage #1 selection.
Then i don't need the "cluster curStage2 by (theKey)" anymore!
But: i also need to do an update of the primary driving data!
So the stage #1 select is a 'select ... for update ...'.
But you can't use such in CURSOR for table functions (which i can understand, why it's not possible).
So i have to do my stage #1 selection in two steps:
1. 'select for update' by main session and collect result in SQL collection.
2. pass collected data to parallel table functions
And for 2. i recognized, that it doesn't start up multiple parallel table function instances.
As a work-around
- if it's just not possible to start multiple parallel pipelined table functions for dispatching from 'select * from TABLE(CAST(... as ...))' -
i need to select again on the base tables - driven by the SQL collection data.
But before i do so, i wanted to verify, if it's really not possible.
Maybe i just miss a special oracle hint or whatever you can get "out of another box" :-)
-------------------- snap ---------------------------------------------
- many thanks!
rgds,
Frank