10g: parallel pipelined table func - distributing DISTINCT data sets

Question

Hi,
i want to distribute data records, selected from cursor, via parallel pipelined table function to multiple worker threads for processing and returning result records.
The tables, where i am selecting data from, are partitioned and subpartitioned.
All tables share the same partitioning/subpartitioning schema.
Each table has a column 'Subpartition_Key', which is hashed to a physical subpartition.
E.g. the Subpartition_Key ranges from 000...999, but we have only 10 physical subpartitions.
The select of records is done partition-wise - one partition after another (in bulks).
The parallel running worker threads select more data from other tables for their processing (2nd level select)
Now my goal is to distribute initial records to the worker threads in a way, that they operate on distinct subpartitions - to decouple the access to resources (for the 2nd level select)
But i cannot just use 'parallel_enable(partition curStage1 by hash(subpartition_key))' for the distribution.
hash(subpartition_key) (hashing A) does not match with the hashing B used to assign the physical subpartition for the INSERT into the tables.
Even when i remodel the hashing B, calculate some SubPartNo(subpartition_key) and use that for 'parallel_enable(partition curStage1 by hash(SubPartNo))' it doesn't work.
Also 'parallel_enable(partition curStage1 by range(SubPartNo))' doesn't help. The load distribution is unbalanced - some worker threads get data of one subpartition, some of multiple subpartitions, some are idle.
How can i distribute the records to the worker threads according a given subpartition-schema?
+[amendment:+
Actually the hashing for the parallel_enable is counterproductive here - it would be better to have some 'parallel_enable(partition curStage1 by SubPartNo)'.]
- many thanks!
best regards,
Frank
Edited by: user8704911 on Jan 12, 2012 2:51 AM

Answer 1

Hello
A couple of things to note. 1, when you use partition by hash(or range) on 10gr2 and above, there is an additional BUFFER SORT operation vs using partition by ANY. For small datasets this is not necessarily an issue, but the temp space used by this stage can be significant for larger data sets. So be sure to check temp space usage for this process or you could run into problems later.
| Id  | Operation                             | Name     | Rows  | Bytes | Cost (%CPU)| Time     | Pstart| Pstop |    TQ  |IN-OUT| PQ Distrib |
|   0 | SELECT STATEMENT                      |          |  8168 |  1722K|    24   (0)| 00:00:01 |       |       |        |      |            |
|   1 |  PX COORDINATOR                       |          |       |       |            |          |       |       |        |      |            |
|   2 |   PX SEND QC (RANDOM)                 | :TQ10001 |  8168 |  1722K|    24   (0)| 00:00:01 |       |       |  Q1,01 | P->S | QC (RAND)  |
|   3 |****BUFFER SORT****                    |          |  8168 |  1722K|            |          |       |       |  Q1,01 | PCWP |            |
|   4 |     VIEW                              |          |  8168 |  1722K|    24   (0)| 00:00:01 |       |       |  Q1,01 | PCWP |            |
|   5 |      COLLECTION ITERATOR PICKLER FETCH| TF       |       |       |            |          |       |       |  Q1,01 | PCWP |            |
|   6 |       PX RECEIVE                      |          |   100 |  4800 |     2   (0)| 00:00:01 |       |       |  Q1,01 | PCWP |            |
|   7 |        PX SEND HASH                   | :TQ10000 |   100 |  4800 |     2   (0)| 00:00:01 |       |       |  Q1,00 | P->P | HASH       |
|   8 |         PX BLOCK ITERATOR             |          |   100 |  4800 |     2   (0)| 00:00:01 |     1 |    10 |  Q1,00 | PCWC |            |
|   9 |          TABLE ACCESS FULL            | TEST_TAB |   100 |  4800 |     2   (0)| 00:00:01 |     1 |    20 |  Q1,00 | PCWP |            |
-----------------------------------------------------------------------------------------------------------------------------------------------It may be in this case that you can use clustering with partition by any to achieve your goal...
create or replace package test_pkg as
     type Test_Tab_Rec_t is record (
          Tracking_ID                 number(19),
          Partition_Key               date,
          Subpartition_Key            number(3),
          sid                    number
     type Test_Tab_Rec_Tab_t is table of Test_Tab_Rec_t;
     type Test_Tab_Rec_Hash_t is table of Test_Tab_Rec_t index by binary_integer;
     type Test_Tab_Rec_HashHash_t is table of Test_Tab_Rec_Hash_t index by binary_integer;
     type Cur_t is ref cursor return Test_Tab_Rec_t;
     procedure populate;
     procedure report;
     function tf(cur in Cur_t)
     return test_list pipelined
     parallel_enable(partition cur by hash(subpartition_key));
     function tf_any(cur in Cur_t)
     return test_list PIPELINED
    CLUSTER cur BY (Subpartition_Key)
     parallel_enable(partition cur by ANY);   
end;
create or replace package body test_pkg as
     procedure populate
     is
          Tracking_ID number(19) := 1;
          Partition_Key date := current_timestamp;
          Subpartition_Key number(3) := 1;
     begin
          dbms_output.put_line(chr(10) || 'populate data into Test_Tab...');
          for Subpartition_Key in 0..99
          loop
               for ctr in 1..1
               loop
                    insert into test_tab (tracking_id, partition_key, subpartition_key)
                    values (Tracking_ID, Partition_Key, Subpartition_Key);
                    Tracking_ID := Tracking_ID + 1;
               end loop;
          end loop;
          dbms_output.put_line('...done (populate data into Test_Tab)');
     end;
     procedure report
     is
          recs Test_Tab_Rec_Tab_t;
     begin
          dbms_output.put_line(chr(10) || 'list data per partition/subpartition...');
          for item in (select partition_name, subpartition_name from user_tab_subpartitions where table_name='TEST_TAB' order by partition_name, subpartition_name)
          loop
               dbms_output.put_line('partition/subpartition = '  || item.partition_name || '/' || item.subpartition_name || ':');
               execute immediate 'select * from test_tab SUBPARTITION(' || item.subpartition_name || ')' bulk collect into recs;
               if recs.count > 0
               then
                    for i in recs.first..recs.last
                    loop
                         dbms_output.put_line('...' || recs(i).Tracking_ID || ', ' || recs(i).Partition_Key  || ', ' || recs(i).Subpartition_Key);
                    end loop;
               end if;
          end loop;
          dbms_output.put_line('... done (list data per partition/subpartition)');
     end;
     function tf(cur in Cur_t)
     return test_list pipelined
     parallel_enable(partition cur by hash(subpartition_key))
     is
          sid number;
          input Test_Tab_Rec_t;
          output test_object;
     begin
          select userenv('SID') into sid from dual;
          loop
               fetch cur into input;
               exit when cur%notfound;
               output := test_object(input.tracking_id, input.partition_key, input.subpartition_key,sid);
               pipe row(output);
          end loop;
     end;
     function tf_any(cur in Cur_t)
     return test_list PIPELINED
    CLUSTER cur BY (Subpartition_Key)
     parallel_enable(partition cur by ANY)
     is
          sid number;
          input Test_Tab_Rec_t;
          output test_object;
     begin
          select userenv('SID') into sid from dual;
          loop
               fetch cur into input;
               exit when cur%notfound;
               output := test_object(input.tracking_id, input.partition_key, input.subpartition_key,sid);
               pipe row(output);
          end loop;
     end;
end;
XXXX> with parts as (
  2  select --+ materialize
  3      data_object_id,
  4      subobject_name
  5  FROM
  6      user_objects
  7  WHERE
  8      object_name = 'TEST_TAB'
  9  and
10      object_type = 'TABLE SUBPARTITION'
11  )
12  SELECT
13        COUNT(*),
14        parts.subobject_name,
15        target.sid
16  FROM
17        parts,
18        test_tab tt,
19        test_tab_part_hash target
20  WHERE
21        tt.tracking_id = target.tracking_id
22  and
23        parts.data_object_id = DBMS_MView.PMarker(tt.rowid)
24  GROUP BY
25        parts.subobject_name,
26        target.sid
27  ORDER BY
28        target.sid,
29        parts.subobject_name
30  /
XXXX> INSERT INTO test_tab_part_hash select * from table(test_pkg.tf(CURSOR(select * from test_tab)))
  2  /
100 rows created.
Elapsed: 00:00:00.14
XXXX>
XXXX> INSERT INTO test_tab_part_any_cluster select * from table(test_pkg.tf_any(CURSOR(select * from test_tab)))
  2  /
100 rows created.
--using partition by hash
XXXX> with parts as (
  2  select --+ materialize
  3      data_object_id,
  4      subobject_name
  5  FROM
  6      user_objects
  7  WHERE
  8      object_name = 'TEST_TAB'
  9  and
10      object_type = 'TABLE SUBPARTITION'
11  )
12  SELECT
13        COUNT(*),
14        parts.subobject_name,
15        target.sid
16  FROM
17        parts,
18        test_tab tt,
19        test_tab_part_hash target
20  WHERE
21        tt.tracking_id = target.tracking_id
22  and
23        parts.data_object_id = DBMS_MView.PMarker(tt.rowid)
24  GROUP BY
25        parts.subobject_name,
26        target.sid
27  /
  COUNT(*) SUBOBJECT_NAME                        SID
         3 SYS_SUBP31                           1272
         1 SYS_SUBP32                           1272
         1 SYS_SUBP33                           1272
         3 SYS_SUBP34                           1272
         1 SYS_SUBP36                           1272
         1 SYS_SUBP37                           1272
         3 SYS_SUBP38                           1272
         1 SYS_SUBP39                           1272
         1 SYS_SUBP32                           1280
         2 SYS_SUBP33                           1280
         2 SYS_SUBP34                           1280
         1 SYS_SUBP35                           1280
         2 SYS_SUBP36                           1280
         1 SYS_SUBP37                           1280
         2 SYS_SUBP38                           1280
         1 SYS_SUBP40                           1280
         2 SYS_SUBP33                           1283
         2 SYS_SUBP34                           1283
         2 SYS_SUBP35                           1283
         2 SYS_SUBP36                           1283
         1 SYS_SUBP37                           1283
         1 SYS_SUBP38                           1283
         2 SYS_SUBP39                           1283
         1 SYS_SUBP40                           1283
         1 SYS_SUBP32                           1298
         1 SYS_SUBP34                           1298
         1 SYS_SUBP36                           1298
         2 SYS_SUBP37                           1298
         4 SYS_SUBP38                           1298
         2 SYS_SUBP40                           1298
         1 SYS_SUBP31                           1313
         1 SYS_SUBP33                           1313
         1 SYS_SUBP39                           1313
         1 SYS_SUBP40                           1313
         1 SYS_SUBP32                           1314
         1 SYS_SUBP35                           1314
         1 SYS_SUBP38                           1314
         1 SYS_SUBP40                           1314
         2 SYS_SUBP33                           1381
         1 SYS_SUBP34                           1381
         1 SYS_SUBP35                           1381
         3 SYS_SUBP36                           1381
         3 SYS_SUBP37                           1381
         1 SYS_SUBP38                           1381
         2 SYS_SUBP36                           1531
         1 SYS_SUBP37                           1531
         2 SYS_SUBP38                           1531
         1 SYS_SUBP39                           1531
         1 SYS_SUBP40                           1531
         2 SYS_SUBP33                           1566
         1 SYS_SUBP34                           1566
         1 SYS_SUBP35                           1566
         1 SYS_SUBP37                           1566
         1 SYS_SUBP38                           1566
         2 SYS_SUBP39                           1566
         3 SYS_SUBP40                           1566
         1 SYS_SUBP32                           1567
         3 SYS_SUBP33                           1567
         3 SYS_SUBP35                           1567
         3 SYS_SUBP36                           1567
         1 SYS_SUBP37                           1567
         2 SYS_SUBP38                           1567
62 rows selected.
--using partition by any cluster by subpartition_key
Elapsed: 00:00:00.26
XXXX> with parts as (
  2  select --+ materialize
  3      data_object_id,
  4      subobject_name
  5  FROM
  6      user_objects
  7  WHERE
  8      object_name = 'TEST_TAB'
  9  and
10      object_type = 'TABLE SUBPARTITION'
11  )
12  SELECT
13        COUNT(*),
14        parts.subobject_name,
15        target.sid
16  FROM
17        parts,
18        test_tab tt,
19        test_tab_part_any_cluster target
20  WHERE
21        tt.tracking_id = target.tracking_id
22  and
23        parts.data_object_id = DBMS_MView.PMarker(tt.rowid)
24  GROUP BY
25        parts.subobject_name,
26        target.sid
27  ORDER BY
28        target.sid,
29        parts.subobject_name
30  /
  COUNT(*) SUBOBJECT_NAME                        SID
        11 SYS_SUBP37                           1253
        10 SYS_SUBP34                           1268
         4 SYS_SUBP31                           1289
        10 SYS_SUBP40                           1314
         7 SYS_SUBP39                           1367
         9 SYS_SUBP35                           1377
        14 SYS_SUBP36                           1531
         5 SYS_SUBP32                           1572
        13 SYS_SUBP33                           1577
        17 SYS_SUBP38                           1609
10 rows selected.Bear in mind though that this does require a sort of the incomming dataset but does not require buffering of the output...
PLAN_TABLE_OUTPUT
Plan hash value: 2570087774
| Id  | Operation                            | Name     | Rows  | Bytes | Cost (%CPU)| Time     | Pstart| Pstop |    TQ  |IN-OUT| PQ Distrib |
|   0 | SELECT STATEMENT                     |          |  8168 |  1722K|    24   (0)| 00:00:01 |       |       |        |      |            |
|   1 |  PX COORDINATOR                      |          |       |       |            |          |       |       |        |      |            |
|   2 |   PX SEND QC (RANDOM)                | :TQ10000 |  8168 |  1722K|    24   (0)| 00:00:01 |       |       |  Q1,00 | P->S | QC (RAND)  |
|   3 |    VIEW                              |          |  8168 |  1722K|    24   (0)| 00:00:01 |       |       |  Q1,00 | PCWP |            |
|   4 |     COLLECTION ITERATOR PICKLER FETCH| TF_ANY   |       |       |            |          |       |       |  Q1,00 | PCWP |            |
|   5 |      SORT ORDER BY                   |          |       |       |            |          |       |       |  Q1,00 | PCWP |            |
|   6 |       PX BLOCK ITERATOR              |          |   100 |  4800 |     2   (0)| 00:00:01 |     1 |    10 |  Q1,00 | PCWC |            |
|   7 |        TABLE ACCESS FULL             | TEST_TAB |   100 |  4800 |     2   (0)| 00:00:01 |     1 |    20 |  Q1,00 | PCWP |            |
----------------------------------------------------------------------------------------------------------------------------------------------HTH
David