Same query has vastly different run times on different DBs

Question

I have a query that is run on two different Oracle DBs (located on separate computers). One finishes in about 45 minutes; the other takes over two hours (how much longer, I can't tell you, as Linux keeps dropping the connection at the two-hour mark, but that's another story).
The tables on each space have identical data; the only significant difference is, one uses tables, while the other uses materialized views. The one with the tables is the faster one.
Both databases are running 11.2.0.2.0 Enterprise Edition 64-bit.
I ran an Explain Plan on the queries, and noticed that the faster one had an additional couple of lines in its plan.
Here is the query:
select pay_plan
     , case when salary < 10000
            then 0
            when salary >= 150000
            then 15
            else floor(salary / 10000)
       end as salary_group
     , pay_grade
     , pay_step
     , fy
     , count(employee_id) as employee_count
from
    select dcps.yr + 1 as fy
         , dcps.employee_id
         , dcpds.pay_plan
         , dcpds.pay_grade
         , dcpds.pay_step
         , sum(dcps.salary) as salary
    from
      select /*+ index(dcps dcps_location) */
             employee_id
           , extract(year from pay_date) as yr
           , sum(
                  case when grc like 'O%'
                       then 0
                       else amt_eec * 26
                  end
                ) as salary
      from dcps
      where location like 'W%'
      and   to_number(to_char(pay_date, 'MMDD')) between 1014 and 1027
      and   substr(grc, 1, 1) IN ('B', 'C', 'D', 'E', 'F', 'H', 'L', 'O', 'R', 'S', 'T')
      group by employee_id, extract(year from pay_date)
    ) dcps
    join
      select employee_id
           , pay_plan
           , pay_grade
           , pay_step
           , file_date
      from
        select /*+ index(dcpds dcpds_location) */
               employee_id
             , pay_plan
             , pay_grade
             , decode(pay_plan, 'YA', 0, pay_step) as pay_step
             , file_date
             , max(file_date)
               over (partition by extract(year from (file_date + 61)))
               as last_file_date
        from dcpds
        where location like 'W%'
        and   pay_plan in ('GS', 'YA')
      where file_date = last_file_date
    ) dcpds
    on (
         dcpds.employee_id = dcps.employee_id
         and dcps.yr = extract(year from dcpds.file_date)
    group by dcps.yr, dcps.employee_id, dcpds.pay_plan, dcpds.pay_grade, dcpds.pay_step
group by pay_plan
    , case when salary < 10000
           then 0
           when salary >= 150000
           then 15
           else floor(salary / 10000)
      end
    , pay_grade
    , pay_step
    , fy;Here is the "faster" plan:
(sorry about the formatting - it's taken from an XML version generated in Toad)
id="0" operation="SELECT STATEMENT" optimizer="ALL_ROWS" cost="10,604,695" cardinality="46" bytes="2,346" cpu_cost="369,545,379,847" io_cost="10,595,408" time="148,466"
    id="1" operation="HASH" option="GROUP BY" cost="10,604,695" cardinality="46" bytes="2,346" cpu_cost="369,545,379,847" io_cost="10,595,408" qblock_name="SEL$1" time="148,466"
        id="2" operation="VIEW" object_owner="DDELGRANDE_DBA" object_instance="1" cost="10,604,694" cardinality="41,337" bytes="2,108,187" cpu_cost="369,477,028,079" io_cost="10,595,408" qblock_name="SEL$103D06FF" time="148,466"
            id="3" operation="HASH" option="GROUP BY" cost="10,604,694" cardinality="41,337" bytes="3,348,297" cpu_cost="369,477,028,079" io_cost="10,595,408" temp_space="4,178,000" qblock_name="SEL$103D06FF" time="148,466"
                id="4" operation="HASH JOIN" cost="10,604,203" cardinality="41,337" bytes="3,348,297" cpu_cost="369,396,215,555" io_cost="10,594,919" temp_space="4,211,000" access_predicates="&quot;EMPLOYEE_ID&quot;=&quot;ITEM_2&quot; AND &quot;ITEM_1&quot;=EXTRACT(YEAR FROM INTERNAL_FUNCTION(&quot;FILE_DATE&quot;))" time="148,459"
                    object_ID="0" id="5" operation="VIEW" object_owner="SYS" object_name="VW_GBC_6" object_type="VIEW" object_instance="39" cost="2,195,131" cardinality="87,663" bytes="3,155,868" cpu_cost="241,010,751,843" io_cost="2,189,074" qblock_name="SEL$2EE98332" time="30,732"
                        id="6" operation="HASH" option="GROUP BY" cost="2,195,131" cardinality="87,663" bytes="3,155,868" cpu_cost="241,010,751,843" io_cost="2,189,074" temp_space="4,424,000" qblock_name="SEL$2EE98332" time="30,732"
                            id="7" operation="VIEW" object_owner="DDELGRANDE_DBA" object_instance="2" cost="2,194,600" cardinality="91,299" bytes="3,286,764" cpu_cost="240,889,683,025" io_cost="2,188,546" qblock_name="SEL$3" time="30,725"
                                id="8" operation="HASH" option="GROUP BY" cost="2,194,600" cardinality="91,299" bytes="3,012,867" cpu_cost="240,889,683,025" io_cost="2,188,546" temp_space="4,424,000" qblock_name="SEL$3" time="30,725"
                                    object_ID="1" id="9" operation="TABLE ACCESS" option="BY INDEX ROWID" optimizer="ANALYZED" object_owner="CORP_FIN" object_name="DCPS" object_type="TABLE" object_instance="3" cost="2,194,088" cardinality="91,299" bytes="3,012,867" cpu_cost="240,769,155,979" io_cost="2,188,037" qblock_name="SEL$3" filter_predicates="TO_NUMBER(TO_CHAR(INTERNAL_FUNCTION(&quot;DTE_PPE_END&quot;),'MMDD'))&gt;=1014 AND TO_NUMBER(TO_CHAR(INTERNAL_FUNCTION(&quot;DTE_PPE_END&quot;),'MMDD'))&lt;=1027 AND (SUBSTR(&quot;GRC&quot;,1,1)='B' OR SUBSTR(&quot;GRC&quot;,1,1)='C' OR SUBSTR(&quot;GRC&quot;,1,1)='D' OR SUBSTR(&quot;GRC&quot;,1,1)='E' OR SUBSTR(&quot;GRC&quot;,1,1)='F' OR SUBSTR(&quot;GRC&quot;,1,1)='H' OR SUBSTR(&quot;GRC&quot;,1,1)='L' OR SUBSTR(&quot;GRC&quot;,1,1)='O' OR SUBSTR(&quot;GRC&quot;,1,1)='R' OR SUBSTR(&quot;GRC&quot;,1,1)='S' OR SUBSTR(&quot;GRC&quot;,1,1)='T')" time="30,718"
                                        object_ID="2" id="10" operation="INDEX" option="RANGE SCAN" optimizer="ANALYZED" object_owner="CORP_FIN" object_name="DCPS_LOCATION" object_type="INDEX" search_columns="1" cost="153,659" cardinality="348,929,550" cpu_cost="22,427,363,111" io_cost="153,095" qblock_name="SEL$3" access_predicates="&quot;LOCATION&quot; LIKE 'W%'" filter_predicates="&quot;LOCATION&quot; LIKE 'W%'" time="2,152"/
                    id="11" operation="VIEW" object_owner="DDELGRANDE_DBA" object_instance="5" cost="8,354,912" cardinality="23,219,146" bytes="1,044,861,570" cpu_cost="123,043,653,827" io_cost="8,351,820" qblock_name="SEL$5" filter_predicates="&quot;FILE_DATE&quot;=&quot;LAST_FILE_DATE&quot;" time="116,969"
                        id="12" operation="WINDOW" option="SORT" cost="8,354,912" cardinality="23,219,146" bytes="766,231,818" cpu_cost="123,043,653,827" io_cost="8,351,820" temp_space="1,211,565,000" qblock_name="SEL$5" time="116,969"
                            object_ID="3" id="13" operation="TABLE ACCESS" option="BY INDEX ROWID" optimizer="ANALYZED" object_owner="CORP_FIN" object_name="DCPDS" object_type="TABLE" object_instance="6" cost="8,225,535" cardinality="23,219,146" bytes="766,231,818" cpu_cost="94,120,935,947" io_cost="8,223,170" qblock_name="SEL$5" filter_predicates="&quot;PAY_PLAN&quot;='GS' OR &quot;PAY_PLAN&quot;='YA'" time="115,158"
                                object_ID="4" id="14" operation="INDEX" option="RANGE SCAN" optimizer="ANALYZED" object_owner="DDELGRANDE_DBA" object_name="DCPDS_LOCATION" object_type="INDEX" search_columns="1" cost="19,848" cardinality="44,080,322" cpu_cost="2,837,503,343" io_cost="19,777" qblock_name="SEL$5" access_predicates="&quot;LOCATION&quot; LIKE 'W%'" filter_predicates="&quot;LOCATION&quot; LIKE 'W%'" time="278"/Here is the "slower" one:
id="0" operation="SELECT STATEMENT" optimizer="ALL_ROWS" cost="28,025,223" cardinality="104,755" bytes="5,552,015" cpu_cost="806,125,131,535" io_cost="27,983,186" time="392,354"
    id="1" operation="HASH" option="GROUP BY" cost="28,025,223" cardinality="104,755" bytes="5,552,015" cpu_cost="806,125,131,535" io_cost="27,983,186" qblock_name="SEL$1" time="392,354"
        id="2" operation="VIEW" object_owner="DDELGRANDE_DBA" object_instance="1" cost="28,025,218" cardinality="104,755" bytes="5,552,015" cpu_cost="806,027,246,428" io_cost="27,983,186" qblock_name="SEL$1D90FC22" time="392,354"
            id="3" operation="HASH" option="GROUP BY" cost="28,025,218" cardinality="104,755" bytes="8,275,645" cpu_cost="806,027,246,428" io_cost="27,983,186" qblock_name="SEL$1D90FC22" time="392,354"
                id="4" operation="HASH JOIN" cost="28,025,213" cardinality="104,755" bytes="8,275,645" cpu_cost="805,929,361,321" io_cost="27,983,186" temp_space="481,887,000" access_predicates="&quot;EMPLOYEE_ID&quot;=&quot;DCPS&quot;.&quot;EMPLOYEE_ID&quot; AND &quot;DCPS&quot;.&quot;YR&quot;=EXTRACT(YEAR FROM INTERNAL_FUNCTION(&quot;FILE_DATE&quot;))" time="392,353"
                    id="5" operation="VIEW" object_owner="DDELGRANDE_DBA" object_instance="2" cost="2,823,626" cardinality="10,475,527" bytes="356,167,918" cpu_cost="487,845,223,357" io_cost="2,798,186" qblock_name="SEL$3" time="39,531"
                        id="6" operation="HASH" option="GROUP BY" cost="2,823,626" cardinality="10,475,527" bytes="398,070,026" cpu_cost="487,845,223,357" io_cost="2,798,186" qblock_name="SEL$3" time="39,531"
                            object_ID="0" id="7" operation="MAT_VIEW ACCESS" option="BY INDEX ROWID" object_owner="ARMYMP" object_name="DCPS" object_type="MAT_VIEW" object_instance="3" cost="2,823,051" cardinality="10,475,527" bytes="398,070,026" cpu_cost="476,819,453,647" io_cost="2,798,186" qblock_name="SEL$3" filter_predicates="TO_NUMBER(TO_CHAR(INTERNAL_FUNCTION(&quot;DTE_PPE_END&quot;),'MMDD'))&gt;=1014 AND TO_NUMBER(TO_CHAR(INTERNAL_FUNCTION(&quot;DTE_PPE_END&quot;),'MMDD'))&lt;=1027 AND (SUBSTR(&quot;GRC&quot;,1,1)='B' OR SUBSTR(&quot;GRC&quot;,1,1)='C' OR SUBSTR(&quot;GRC&quot;,1,1)='D' OR SUBSTR(&quot;GRC&quot;,1,1)='E' OR SUBSTR(&quot;GRC&quot;,1,1)='F' OR SUBSTR(&quot;GRC&quot;,1,1)='H' OR SUBSTR(&quot;GRC&quot;,1,1)='L' OR SUBSTR(&quot;GRC&quot;,1,1)='O' OR SUBSTR(&quot;GRC&quot;,1,1)='R' OR SUBSTR(&quot;GRC&quot;,1,1)='S' OR SUBSTR(&quot;GRC&quot;,1,1)='T')" time="39,523"
                                object_ID="1" id="8" operation="INDEX" option="RANGE SCAN" optimizer="ANALYZED" object_owner="ARMYMP" object_name="DCPS_LOCATION" object_type="INDEX" search_columns="1" cost="281,465" cardinality="215,251,937" cpu_cost="46,870,067,520" io_cost="279,021" qblock_name="SEL$3" access_predicates="&quot;LOCATION&quot; LIKE 'W%'" filter_predicates="&quot;LOCATION&quot; LIKE 'W%'" time="3,941"/
                    id="9" operation="VIEW" object_owner="DDELGRANDE_DBA" object_instance="5" cost="25,134,240" cardinality="20,437,108" bytes="919,669,860" cpu_cost="311,591,056,432" io_cost="25,117,991" qblock_name="SEL$5" filter_predicates="&quot;FILE_DATE&quot;=&quot;LAST_FILE_DATE&quot;" time="351,880"
                        id="10" operation="WINDOW" option="SORT" cost="25,134,240" cardinality="20,437,108" bytes="633,550,348" cpu_cost="311,591,056,432" io_cost="25,117,991" temp_space="984,859,000" qblock_name="SEL$5" time="351,880"
                            object_ID="2" id="11" operation="MAT_VIEW ACCESS" option="BY INDEX ROWID" object_owner="ARMYMP" object_name="DCPDS" object_type="MAT_VIEW" object_instance="6" cost="25,024,511" cardinality="20,437,108" bytes="633,550,348" cpu_cost="286,442,201,519" io_cost="25,009,574" qblock_name="SEL$5" filter_predicates="&quot;PAY_PLAN&quot;='GS' OR &quot;PAY_PLAN&quot;='YA'" time="350,344"
                                object_ID="3" id="12" operation="INDEX" option="RANGE SCAN" optimizer="ANALYZED" object_owner="ARMYMP" object_name="DCPDS_LOCATION" object_type="INDEX" search_columns="1" cost="52,686" cardinality="34,054,388" cpu_cost="8,896,424,679" io_cost="52,222" qblock_name="SEL$5" access_predicates="&quot;LOCATION&quot; LIKE 'W%'" filter_predicates="&quot;LOCATION&quot; LIKE 'W%'" time="738"/Notice the faster one has two extra lines in it; it is creating a SYS-based view with an additional "Group By" hash.
Also, the faster one's Table Access By Rowid lines are marked as "Analyzed", while the slower one's Materialized View Access By Index Rowid lines are not.
Any idea why this would happen?
(And yes, I do notice that the cpu_cost values for the slower one tend to be 2-4x as high as for the faster one.)

Answer 1

Also, the faster one's Table Access By Rowid lines are marked as "Analyzed", while the slower one's Materialized View Access By Index Rowid lines are not.Have you gathered stats on MVs table?