Double WINDOW SORT Operation

Question

Please review following SQL and it's execution plan. Why am I seeing 2 WINDOW SORT operations even though in sql . analytic function "row_number" has been used only once?
Also, In step 3 of the plan, why "bytes" goes from 35 GB(4th step) to 88GB when row count remains the same. In fact , since I'm selecting just 1st row , both row count as well as "bytes" should have gone down. Shouldn't it?
  SELECT orddtl.ord_dtl_key, orddtl.ld_nbr, orddtl.actv_flg,
         orddtl.ord_nbr
     FROM (SELECT /*+ parallel(od, 8) parallel(sc,8) */  od.ord_dtl_key, od.ld_nbr, od.actv_flg,
                  od.ord_nbr,
                  ROW_NUMBER () OVER (PARTITION BY od.ord_dtl_key, od.START_TS ORDER BY sc.START_TS DESC)
                                                                      rownbr
             FROM edw.order_detail od LEFT OUTER JOIN edw.srvc_code sc
                  ON (    sc.srvc_cd_key = od.srvc_cd_key
                      AND od.part_nbr = sc.part_nbr
                      AND od.item_cre_dt >= sc.START_TS
                      AND od.item_cre_dt < sc.END_TS
            WHERE od.part_nbr = 11 ) orddtl
    WHERE orddtl.rownbr = 1;Execution Plan
| Id  | Operation                      | Name              | Rows  | Bytes |TempSpc| Cost (%CPU)| Time     | Pstart| Pstop |    TQ  |IN-OUT| PQ Distrib |
|   0 | SELECT STATEMENT               |                   |    88M|   121G|       |  2353K (65)| 00:33:07 |       |       |        |      |            |
|   1 |  PX COORDINATOR                |                   |       |       |       |            |          |       |       |        |      |            |
|   2 |   PX SEND QC (RANDOM)          | :TQ10002          |    88M|   121G|       |  2353K (65)| 00:33:07 |       |       |  Q1,02 | P->S | QC (RAND)  |
|*  3 |    VIEW                        |                   |    88M|   121G|       |  2353K (65)| 00:33:07 |       |       |  Q1,02 | PCWP |            |
|*  4 |     WINDOW SORT PUSHED RANK    |                   |    88M|    35G|    75G|  2353K (65)| 00:33:07 |       |       |  Q1,02 | PCWP |            |
|   5 |      PX RECEIVE                |                   |    88M|    35G|       |  2353K (65)| 00:33:07 |       |       |  Q1,02 | PCWP |            |
|   6 |       PX SEND HASH             | :TQ10001          |    88M|    35G|       |  2353K (65)| 00:33:07 |       |       |  Q1,01 | P->P | HASH       |
|*  7 |        WINDOW CHILD PUSHED RANK|                   |    88M|    35G|       |  2353K (65)| 00:33:07 |       |       |  Q1,01 | PCWP |            |
|*  8 |         HASH JOIN RIGHT OUTER  |                   |    88M|    35G|       |  1610K (92)| 00:22:39 |       |       |  Q1,01 | PCWP |            |
|   9 |          PX RECEIVE            |                   |  1133K|    32M|       |  1197  (20)| 00:00:02 |       |       |  Q1,01 | PCWP |            |
|  10 |           PX SEND BROADCAST    | :TQ10000          |  1133K|    32M|       |  1197  (20)| 00:00:02 |       |       |  Q1,00 | P->P | BROADCAST  |
|  11 |            PX BLOCK ITERATOR   |                   |  1133K|    32M|       |  1197  (20)| 00:00:02 |   KEY |   KEY |  Q1,00 | PCWC |            |
|  12 |             TABLE ACCESS FULL  |  SRVC_CODE        |  1133K|    32M|       |  1197  (20)| 00:00:02 |     1 |     1 |  Q1,00 | PCWP |            |
|  13 |          PX BLOCK ITERATOR     |                   |    88M|    32G|       |   188K (27)| 00:02:39 |   KEY |   KEY |  Q1,01 | PCWC |            |
|  14 |           TABLE ACCESS FULL    |    ORDER_DETAIL   |    88M|    32G|       |   188K (27)| 00:02:39 |     1 |     1 |  Q1,01 | PCWP |            |
Predicate Information (identified by operation id):
   3 - filter("orddtl"."rownbr"=1)
   4 - filter(ROW_NUMBER() OVER ( PARTITION BY "od"."ORD_DTL_KEY","od"."START_TS" ORDER BY INTERNAL_FUNCTION("SC"."START_TS"(+))
              DESC )<=1)
   7 - filter(ROW_NUMBER() OVER ( PARTITION BY "od"."ORD_DTL_KEY","od"."START_TS" ORDER BY INTERNAL_FUNCTION("SC"."START_TS"(+))
              DESC )<=1)
   8 - access("od"."part_nbr"="SC"."part_nbr"(+) AND "SC"."SRVC_CD_KEY"(+)="od"."SRVC_CD_KEY")
       filter("od"."ITEM_CRE_DT"<"SC"."END_TS"(+) AND "od"."ITEM_CRE_DT">="SC"."START_TS"(+))

Answer 1

Thanks Jonathan for your reply.
This type of pattern happens quite frequently in parallel execution with aggregation. A layer of slave processes can do partial aggregation before passing a reduced result set to the query co-ordinator to finish the job.
I wouldn't be 100% sure without building a model to check, but I think the logic of your quer allows the eight slaves to identify each "rownumber() = 1" for the data set they have collected, and the allows the query coordinator to do the window sort on the eight incoming rows (for each key) and determine which one of the eight is ultimate the highest date.So is it a normal pattern? Will step#7 & #4 do the same amount work as stated in PREDICATE information part of execution plan.?
You’re correct! There are 8 slave processes appears to be performing WINDOW PUSHED RANK ( Step#7 in Execution Plan ) as you see in following output. Per execution plan and your comment, each one appears to be finding partial set of rows row_num <= 1. It’s apparently doing lots of work and very slow even with 8 processes. So not sure , how slow would be QC doing the same work just by itself.
And as you see below , it’s [Step#7 ] very slow and half of the slaves performing multi pass sort operation. Even though , it was estimated 35GB for that operation, why it’s estimating work area size of only 6-14MB only? Also, It’s allocating so low amount of PGA than expected. P_A_T was set to approx 11 GB. Currently this was the only query/operation on the Instance.
Why it’s not allocating more PGA for that operation? [My apologies for diverting from my original question ].
I have included PGA stats as well here which was taken 5-10 minutes later than other PQ session information. It’s still shows that there is no shortage of PGA.
Moreover, I have observed this behavior (under allocation of PGA) especially for WINDOWS SORT operations for other SQLs too. Is it normal behavior ? I’m on 10.2.0.4.
select
decode(px.qcinst_id,NULL,username,
' - '||lower(substr(pp.SERVER_NAME,
length(pp.SERVER_NAME)-4,4) ) )"Username",
decode(px.qcinst_id,NULL, 'QC', '(Slave)') "QC/Slave" ,
to_char( px.server_set) "SlaveSet",
to_char(s.sid) "SID",
to_char(px.inst_id) "Slave INST",
decode(sw.state,'WAITING', 'WAIT', 'NOT WAIT' ) as STATE,
case  sw.state WHEN 'WAITING' THEN substr(sw.event,1,30) ELSE NULL end as wait_event ,
to_char(s.ROW_WAIT_OBJ#)  wait_OBID,
decode(px.qcinst_id, NULL ,to_char(s.sid) ,px.qcsid) "QC SID",
to_char(px.qcinst_id) "QC INST",
px.req_degree "Req. DOP",
px.degree "Actual DOP"
from gv$px_session px,
gv$session s ,
gv$px_process pp,
gv$session_wait sw
where px.sid=s.sid (+)
and px.serial#=s.serial#(+)
and px.inst_id = s.inst_id(+)
and px.sid = pp.sid (+)
and px.serial#=pp.serial#(+)
and sw.sid = s.sid
and sw.inst_id = s.inst_id
order by
  decode(px.QCINST_ID,  NULL, px.INST_ID,  px.QCINST_ID),
  px.QCSID,
  decode(px.SERVER_GROUP, NULL, 0, px.SERVER_GROUP),
  px.SERVER_SET,
  px.INST_ID
UNAME        QC/Slave SlaveSet SID       Slave INS STATE    WAIT_EVENT                      WAIT_OBID QC SID QC INS Req. DOP Actual DOP
APPS_ORD     QC                1936      2         WAIT     PX Deq: Execute Reply          71031      1936
- p006      (Slave)  1        1731      2         WAIT     PX Deq: Execution Msg          71021      1936   2             8          8
- p007      (Slave)  1        2159      2         WAIT     PX Deq: Execution Msg          71021      1936   2             8          8
- p002      (Slave)  1        2090      2         WAIT     PX Deq: Execution Msg          71021      1936   2             8          8
- p005      (Slave)  1        1965      2         WAIT     PX Deq: Execution Msg          71021      1936   2             8          8
- p001      (Slave)  1        1934      2         WAIT     PX Deq: Execution Msg          71021      1936   2             8          8
- p004      (Slave)  1        1843      2         WAIT     PX Deq: Execution Msg          71021      1936   2             8          8
- p000      (Slave)  1        1778      2         WAIT     PX Deq: Execution Msg          71021      1936   2             8          8
- p003      (Slave)  1        1751      2         WAIT     PX Deq: Execution Msg          71021      1936   2             8          8
- p009      (Slave)  2        2138      2         NOT WAIT                                71031      1936   2             8          8
- p012      (Slave)  2        1902      2         NOT WAIT                                71031      1936   2             8          8
- p008      (Slave)  2        1921      2         NOT WAIT                                71031      1936   2             8          8
- p013      (Slave)  2        2142      2         NOT WAIT                                71031      1936   2             8          8
- p015      (Slave)  2        2091      2         NOT WAIT                                71031      1936   2             8          8
- p014      (Slave)  2        2122      2         NOT WAIT                                71031      1936   2             8          8
- p010      (Slave)  2        2146      2         NOT WAIT                                71031      1936   2             8          8
- p011      (Slave)  2        1754      2         NOT WAIT                                71031      1936   2             8          8
SELECT operation_type AS type                      ,
        workarea_address WADDR,
        operation_id as OP_ID,
        policy                                      ,
        vwa.sql_id,
        vwa.inst_id i#,
        vwa.sid                                     ,
        vwa.qcsid   QCsID,
        vwa.QCINST_ID  QC_I#,
        s.username uname,
        ROUND(active_time    /1000000,2)   AS a_sec ,
        ROUND(work_area_size /1024/1024,2) AS wsize ,
        ROUND(expected_size  /1024/1024,2) AS exp   ,
        ROUND(actual_mem_used/1024/1024,2) AS act   ,
        ROUND(max_mem_used   /1024/1024,2) AS MAX   ,
        number_passes                      AS p#,
        ROUND(tempseg_size/1024/1024,2)    AS temp
FROM   gv$sql_workarea_active vwa ,
        gv$session s
where  vwa.sid = s.sid
and    vwa.inst_id = s.inst_id
order by vwa.sql_id, operation_id, vwa.inst_id, username, vwa.qcsid
TYPE            WADDR            OP_ID POLI SQL_ID         I#    SID  QCSID QC_I# UNAME                A_SEC      WSIZE        EXP        ACT        MAX   P#       TEMP
WINDOW (SORT)   07000003D2B03F90     7 AUTO 8z5s5wdy94ty3   2   2146   1936     2 APPS_ORD            1181.22      13.59      13.59       7.46      90.98    1        320
WINDOW (SORT)   07000003D2B03F90     7 AUTO 8z5s5wdy94ty3       2142   1936     2 APPS_ORD            1181.07       7.03       7.03       4.02      90.98    0        288
WINDOW (SORT)   07000003D2B03F90     7 AUTO 8z5s5wdy94ty3       2091   1936     2 APPS_ORD            1181.06       7.03       7.03        4.5      90.98    0        288
WINDOW (SORT)   07000003D2B03F90     7 AUTO 8z5s5wdy94ty3       1921   1936     2 APPS_ORD            1181.09      13.59      13.59       2.24      90.98    1        320
WINDOW (SORT)   07000003D2B03F90     7 AUTO 8z5s5wdy94ty3       2138   1936     2 APPS_ORD            1181.16       7.03       7.03       1.34      90.98    0        288
WINDOW (SORT)   07000003D2B03F90     7 AUTO 8z5s5wdy94ty3       1754   1936     2 APPS_ORD            1181.09      14.06      14.06       5.77      90.98    1        320
WINDOW (SORT)   07000003D2B03F90     7 AUTO 8z5s5wdy94ty3       2122   1936     2 APPS_ORD            1181.15       6.56       6.56        .24      90.98    0        288
WINDOW (SORT)   07000003D2B03F90     7 AUTO 8z5s5wdy94ty3       1902   1936     2 APPS_ORD            1181.12      14.06      14.06       9.12      90.98    1        320
HASH-JOIN       07000003D2B03F28     8 AUTO 8z5s5wdy94ty3       2142   1936     2 APPS_ORD            1183.24      98.64      98.64     100.44     100.44    0
HASH-JOIN       07000003D2B03F28     8 AUTO 8z5s5wdy94ty3       2138   1936     2 APPS_ORD            1183.24      98.64      98.64     100.44     100.44    0
HASH-JOIN       07000003D2B03F28     8 AUTO 8z5s5wdy94ty3       2122   1936     2 APPS_ORD            1183.24      98.64      98.64     100.44     100.44    0
HASH-JOIN       07000003D2B03F28     8 AUTO 8z5s5wdy94ty3       2091   1936     2 APPS_ORD            1183.24      98.64      98.64     100.44     100.44    0
HASH-JOIN       07000003D2B03F28     8 AUTO 8z5s5wdy94ty3       1921   1936     2 APPS_ORD            1183.24      98.64      98.64     100.44     100.44    0
HASH-JOIN       07000003D2B03F28     8 AUTO 8z5s5wdy94ty3       1902   1936     2 APPS_ORD            1183.24      98.64      98.64     100.44     100.44    0
HASH-JOIN       07000003D2B03F28     8 AUTO 8z5s5wdy94ty3       2146   1936     2 APPS_ORD            1183.24      98.64      98.64     100.44     100.44    0
HASH-JOIN       07000003D2B03F28     8 AUTO 8z5s5wdy94ty3       1754   1936     2 APPS_ORD            1183.24      98.64      98.64     100.44     100.44    0
                                                          sum                                                872.07            838.21
PGA Stats – taken 5-10 minutes later than above.
select name, decode(unit,'bytes',round(value/1048576,2)||' MB', value) value from v$pgastat
NAME                                               VALUE
aggregate PGA target parameter                     11264 MB
aggregate PGA auto target                          9554.7 MB
global memory bound                                1024 MB
total PGA inuse                                    902.21 MB
total PGA allocated                                3449.64 MB
maximum PGA allocated                              29155.44 MB
total freeable PGA memory                          2140.56 MB
process count                                      107
max processes count                                379
PGA memory freed back to OS                        77240169.56 MB
total PGA used for auto workareas                  254.14 MB
maximum PGA used for auto workareas                22797.02 MB
total PGA used for manual workareas                0 MB
maximum PGA used for manual workareas              16.41 MB
over allocation count                              0
bytes processed                                    323796668.77 MB
extra bytes read/written                           183362312.02 MB
cache hit percentage                               63.84
recompute count (total)                            2054320
SELECT
PGA_TARGET_FOR_ESTIMATE/1048576 ESTMTD_PGA_MB,
   PGA_TARGET_FACTOR PGA_TGT_FCTR,
   ADVICE_STATUS ADV_STS,
   BYTES_PROCESSED/1048576 ESTMTD_MB_PRCD,
   ESTD_EXTRA_BYTES_RW/1048576 ESTMTD_XTRA_MB,
   ESTD_PGA_CACHE_HIT_PERCENTAGE ESTMTD_CHIT_PCT,
   ESTD_OVERALLOC_COUNT O_ALOC_CNT
FROM V$PGA_TARGET_ADVICE
ESTMTD_PGA_MB PGA_TGT_FCTR ADV ESTMTD_MB_PRCD ESTMTD_XTRA_MB ESTMTD_CHIT_PCT O_ALOC_CNT
        1,408         .125 ON     362,905,053    774,927,577              32      19973
        2,816          .25 ON     362,905,053    571,453,995              39        709
        5,632           .5 ON     362,905,053    249,201,001              59          5
        8,448          .75 ON     362,905,053    216,717,381              63          0
       11,264            1 ON     362,905,053    158,762,256              70          0
       13,517          1.2 ON     362,905,053    153,025,642              70          0
       15,770          1.4 ON     362,905,053    153,022,337              70          0
       18,022          1.6 ON     362,905,053    153,022,337              70          0
       20,275          1.8 ON     362,905,053    153,022,337              70          0
       22,528            2 ON     362,905,053    153,022,337              70          0
       33,792            3 ON     362,905,053    153,022,337              70          0
       45,056            4 ON     362,905,053    153,022,337              70          0
       67,584            6 ON     362,905,053    153,022,337              70          0
       90,112            8 ON     362,905,053    153,022,337              70          0