Insert using parallel

Question

what does this last statement mean?  it is as though the query runs just like without any hints.
oracle doc:
Using Parallel Execution
Examples of Distributed Transaction Parallelization
This section contains several examples of distributed transaction processing.
Example 1 Distributed Transaction Parallelization
In this example, the DML statement queries a remote object:
INSERT /* APPEND PARALLEL (t3,2) */ INTO t3 SELECT * FROM t4@dblink;
The query operation is executed serially without notification because it references a remote object.

Answer 1

Randolf,
As far as I have a real db link why not test it myself  (see my questions at the end of this thread)
SQL> insert /*+ append parallel(lcl) */ into local_tab lcl select /*+ parralel(dst) */ dst.* from distant_tab dst;
51 rows created.
SQL> select * from table(dbms_xplan.display_cursor);
PLAN_TABLE_OUTPUT
SQL_ID  1nhmuzb4ayq7x, child number 0
insert /*+ append parallel(lcl) */ into local_tab lcl select /*+
parralel(dst) */ dst.* from distant_tab dst
Plan hash value: 2098243032
| Id  | Operation        | Name          | Rows  | Bytes | Cost (%CPU)| Time     | Inst   |IN-OUT|
|   0 | INSERT STATEMENT |               |       |       |    57 (100)|          |        |      |
|   1 |  LOAD AS SELECT  |               |       |       |            |          |        |      |
|   2 |   REMOTE         | distant_tab   |    51 |  3009 |    57   (0)| 00:00:01 | XLCL~  | R->S |
Remote SQL Information (identified by operation id):
   2 - SELECT /*+ OPAQUE_TRANSFORM */ "DST_ID","NAME_FR","NAME_NL","DST_UIC_CODE","DST_VOI
       C_CODE","BUR_UIC_CODE","BUR_VOIC_CODE","INFO_NEEDED","TRANSFERRED","VALID_FROM_DATE",
    "VALID_TO_DATE","SORTING","SO_NEEDED" FROM "distant_tab" "DST" (accessing'XLCL_XDST.WORLD' )
Let enable parallel DML and repeat the same insert
SQL> alter session enable parallel dml;
Session altered.
SQL> insert /*+ append parallel(lcl) */ into local_tab lcl select /*+ parralel(dst) */ dst.* from distant_tab dst;
51 rows created.
SQL> select * from table(dbms_xplan.display_cursor);
PLAN_TABLE_OUTPUT
SQL_ID  1nhmuzb4ayq7x, child number 1
insert /*+ append parallel(lcl) */ into local_tab lcl select /*+ parralel(dst) */ dst.* from distant_tab dst
Plan hash value: 2511483212
| Id  | Operation               | Name          | Rows  | Bytes | Cost (%CPU)| Time     | TQ/Ins |IN-OUT| PQ Distrib |
|   0 | INSERT STATEMENT        |               |       |       |    57 (100)|          |        |      |            |
|   1 |  PX COORDINATOR         |               |       |       |            |          |        |      |            |
|   2 |   PX SEND QC (RANDOM)   | :TQ10001      |    51 |  3009 |    57   (0)| 00:00:01 |  Q1,01 | P->S | QC (RAND)  |
|   3 |    LOAD AS SELECT       |               |       |       |            |          |  Q1,01 | PCWP |            |
|   4 |     PX RECEIVE          |               |    51 |  3009 |    57   (0)| 00:00:01 |  Q1,01 | PCWP |            |
|   5 |      PX SEND ROUND-ROBIN| :TQ10000      |    51 |  3009 |    57   (0)| 00:00:01 |        | S->P | RND-ROBIN  |
|   6 |       REMOTE            | distant_tab   |    51 |  3009 |    57   (0)| 00:00:01 | XLCL~  | R->S |            |
Remote SQL Information (identified by operation id):
   6 - SELECT /*+ OPAQUE_TRANSFORM */ "DST_ID","NAME_FR","NAME_NL","DST_UIC_CODE","DST_VOIC_CODE","BUR_UIC_COD
       E","BUR_VOIC_CODE","INFO_NEEDED","TRANSFERRED","VALID_FROM_DATE","VALID_TO_DATE","SORTING","SO_NEEDED"
       FROM "distant_tab" "DST" (accessing 'XLCL_XDST.WORLD' )
SQL> select * from local_tab;
select * from local_tab
ERROR at line 1:
ORA-12838: cannot read/modify an object after modifying it in parallel
SQL> select
  2      dfo_number,
  3      tq_id,
  4      server_type,
  5      process,
  6      num_rows,
  7      bytes,
  8      waits,
  9      timeouts,
10      avg_latency,
11      instance
12  from
13      v$pq_tqstat
14  order by
15      dfo_number,
16      tq_id,
17      server_type desc,
18      process
19  ;
DFO_NUMBER      TQ_ID SERVER_TYP PROCES   NUM_ROWS      BYTES      WAITS   TIMEOUTS AVG_LATENCY   INSTANCE
         1          0 Producer   QC             51       4451          0          0           0          1
         1          1 Consumer   QC              1        683         14          6           0          1
This time parallel DML has been used
What If I create a trigger on the local_tab table and repeat the insert?
SQL> create or replace trigger local_tab_trg
  2  before insert on local_tab
  3  for each row
  4  begin
  5   null;
  6  end;
  7  /
Trigger created.
SQL> insert /*+ append parallel(lcl) */ into local_tab lcl select /*+ parralel(dst) */ dst.* from distant_tab dst;
51 rows created.
SQL> select * from table(dbms_xplan.display_cursor);
PLAN_TABLE_OUTPUT
SQL_ID  1nhmuzb4ayq7x, child number 1
insert /*+ append parallel(lcl) */ into local_tab lcl select /*+
parralel(dst) */ dst.* from distant_tab dst
Plan hash value: 1788691278
| Id  | Operation                | Name          | Rows  | Bytes | Cost (%CPU)| Time     | Inst   |IN-OUT|
|   0 | INSERT STATEMENT         |               |       |       |    57 (100)|          |        |      |
|   1 |  LOAD TABLE CONVENTIONAL |               |       |       |            |          |        |      |
|   2 |   REMOTE                 | distant_tab |    51 |  3009 |    57   (0)| 00:00:01 | XLCL~ | R->S |
Remote SQL Information (identified by operation id):
   2 - SELECT /*+ OPAQUE_TRANSFORM */ "DST_ID","NAME_FR","NAME_NL","DST_UIC_CODE","DST_VOIC_CODE",
       "BUR_UIC_CODE","BUR_VOIC_CODE","INFO_NEEDED","TRANSFERRED","VALID_FROM_DATE","VALID_TO_DATE","S
       ORTING","SO_NEEDED" FROM "distant_tab" "DST" (accessing 'XLCL_XDST.WORLD' )
Parallel run has been disabled by the existence of this trigger in both distant and local database
SQL> drop trigger local_tab_trg;
Trigger dropped.
Now I want to test an insert using only the append hint as shown below
SQL> insert /*+ append */ into local_tab lcl select /*+ parralel(dst) */ dst.* from distant_tab dst;
51 rows created.
SQL> select * from table(dbms_xplan.display_cursor);
PLAN_TABLE_OUTPUT
SQL_ID  4pkxbmy8410s9, child number 0
insert /*+ append */ into local_tab lcl select /*+ parralel(dst) */
dst.* from distant_tab dst
Plan hash value: 2098243032
| Id  | Operation        | Name          | Rows  | Bytes | Cost (%CPU)| Time     | Inst   |IN-OUT|
|   0 | INSERT STATEMENT |               |       |       |    57 (100)|          |        |      |
|   1 |  LOAD AS SELECT  |               |       |       |            |          |        |      |
|   2 |   REMOTE         | distant_tab |    51 |  3009 |    57   (0)| 00:00:01 | XLCL~    | R->S |
Remote SQL Information (identified by operation id):
   2 - SELECT /*+ OPAQUE_TRANSFORM */ "DST_ID","NAME_FR","NAME_NL","DST_UIC_CODE","DST_VOI
       C_CODE","BUR_UIC_CODE","BUR_VOIC_CODE","INFO_NEEDED","TRANSFERRED","VALID_FROM_DATE","V
       ALID_TO_DATE","SORTING","SO_NEEDED" FROM "distant_tab" "DST" (accessing
       'XLCL_XDST.WORLD' )
SQL> select * from local_tab;
select * from local_tab
ERROR at line 1:
ORA-12838: cannot read/modify an object after modifying it in parallel
Question 1 : What does this last ORA-128838 means if the execution plan is not showing a parallel DML insert?
Particularly when I repeat the same insert using a parallel hint without the append hint
SQL> insert /*+ parallel(lcl) */ into local_tab lcl select /*+ parralel(dst) */ dst.* from distant_tab dst;
51 rows created.
SQL> select * from table(dbms_xplan.display_cursor);
PLAN_TABLE_OUTPUT
SQL_ID  40uqkc82n1mqn, child number 0
insert /*+ parallel(lcl) */ into local_tab lcl select /*+ parralel(dst)
*/ dst.* from distant_tab dst
Plan hash value: 2511483212
| Id  | Operation               | Name          | Rows  | Bytes | Cost (%CPU)| Time     | TQ/Ins |IN-OUT| PQ Distrib |
|   0 | INSERT STATEMENT        |               |       |       |    57 (100)|          |        |      |            |
|   1 |  PX COORDINATOR         |               |       |       |            |          |        |      |            |
|   2 |   PX SEND QC (RANDOM)   | :TQ10001      |    51 |  3009 |    57   (0)| 00:00:01 |  Q1,01 | P->S | QC (RAND)  |
|   3 |    LOAD AS SELECT       |               |       |       |            |          |  Q1,01 | PCWP |            |
|   4 |     PX RECEIVE          |               |    51 |  3009 |    57   (0)| 00:00:01 |  Q1,01 | PCWP |            |
|   5 |      PX SEND ROUND-ROBIN| :TQ10000      |    51 |  3009 |    57   (0)| 00:00:01 |        | S->P | RND-ROBIN  |
|   6 |       REMOTE            | distant_tab   |    51 |  3009 |    57   (0)| 00:00:01 | A1124~ | R->S |            |
Remote SQL Information (identified by operation id):
   6 - SELECT /*+ OPAQUE_TRANSFORM */ "DST_ID","NAME_FR","NAME_NL","DST_UIC_CODE","DST_VOI
       C_CODE","BUR_UIC_CODE","BUR_VOIC_CODE","INFO_NEEDED","TRANSFERRED","VALID_FROM_DATE","V
       ALID_TO_DATE","SORTING","SO_NEEDED" FROM "distant_tab" "DST" (accessing'XLCL_XDST.WORLD' )
SQL> select * from local_tab;
select * from local_tab
ERROR at line 1:
ORA-12838: cannot read/modify an object after modifying it in parallel
"in that particular case the REMOTE data was joined to some local data and there was an additional BUFFER SORT operation in the execution plan, which (in most cases) shows up when a serial to parallel distribution (S->P) is involved in an operation with further re-distribution of data, and the BUFFER SORT had to buffer all the data from remote before the local operation could continue - defeating any advantage of speeding up the local write operation by parallel DML. A serial direct path insert was actually faster in that particular case."
Question 2 : Isn’t this a normal behavior of distributed DML where the driving site is always the site where the DML operation is done? And this is why the BUFFER SORT had to buffer all the data from remote to the driving site (local site here)?
http://jonathanlewis.wordpress.com/2008/12/05/distributed-dml/
Best regards
Mohamed Houri