Baffled by performance

Question

Oracle 11.1.0.6.0 client, 10.2.0.3 database, Visual Studio 2005.
I have a C# program that had a pretty involved update in it. There were two bind variables, one used several times. When I ran the update as a fat statement within the program, it took hours to run. When I ran it with SQL Developer (with the bind variables in the same places) it took a minute. So I extracted the fat update, put it in a PL/SQL procedure, called it...
  OleDbCommand cmd = CommonSupport._CSIADO.getOleDbForProc("MFIVE.CALC_DOWNTIME.INSERT_PDS");
  OleDbParameter resultParm = cmd.Parameters.Add("p_records_affected", OleDbType.Integer);
  resultParm.Direction = ParameterDirection.Output;
  OleDbParameter downtimeCalcDtParm =
     cmd.Parameters.AddWithValue("p_downtime_calc_dt", un_downtime_calc_dt);
  downtimeCalcDtParm.Direction = ParameterDirection.Input;
  cmd.ExecuteNonQuery();...and it still takes hours. I ran the PL/SQL procedure from within SQL Developer (again, passing in the parameters) and it still takes a minute.
If I look at Enterprise Manager, it says that it's using the same plan as SQL Developer says it will use. Doing a tkprof of a trace is rather different, though, leaving out the work being done in the subselects and just... looking different.
I'll post the tkprof output here for the bored, but a general question: does OLEDB make Oracle stupid under some circumstances?
SQL Developer calling PL/SQL procedure:from SQL Developer (about 100 seconds):
insert into pd_dwn_stat
     (company, unit_id, period, oper_dt, maint_dt)
    with d as
     (select c.company, c.unit_id, min(c.start_dt) as start_dt,
        min(c.shift_code) as shift_code, c.end_dt
      from
         (select a.company, a.unit_id, a.start_dt, g.shift_code as shift_code,
            max(coalesce(b.end_dt, sysdate)) as end_dt
          from unit_downtime a
            join unit_downtime b
              on a.company = b.company
             and a.unit_id = b.unit_id
            left outer join o_wo o
              on o.company = a.company
             and o.wo_no = a.wo_no
            left outer join loc_maint g
              on g.company = o.company
             and g.location = o.location
          where a.start_dt >= b.start_dt
            and a.start_dt < coalesce(a.end_dt, sysdate)
            and b.start_dt < coalesce(b.end_dt, sysdate)
            and b.start_dt >=any
          (select z.start_dt
           from unit_downtime z
           where z.downtime_calc_dt = :p_downtime_calc_dt
             and z.unit_id = b.unit_id)
          group by a.company, a.unit_id, a.start_dt, g.shift_code) c
      group by c.company, c.unit_id, c.end_dt),
       f as
     (select company, fisc_pd, start_date,
        lead(start_date) over (partition by company order by fisc_pd) as end_date
      from fiscal_cal)
    select u.company, u.unit_id, f.fisc_pd, 
      (select sum((least(f.end_date, :p_downtime_calc_dt, d.end_dt, s.end_dt) -
         greatest(f.start_date, d.start_dt, s.start_dt))) * 24 * 60 * 60 *1000
       from shift s
         join d
           on s.company = d.company
       where d.unit_id = u.unit_id
         and s.start_dt >= f.start_date
         and s.end_dt < f.end_date
         and d.start_dt <= s.end_dt
         and d.end_dt > s.start_dt
         and s.company = u.company
         and s.shift_code = u.shift_code) as calc_oper_dt,
      (select sum((least(f.end_date, :p_downtime_calc_dt, d.end_dt, s.end_dt) -
         greatest(f.start_date, d.start_dt, s.start_dt))) * 24 * 60 * 60 *1000
       from shift s
         join d
           on s.company = d.company
       where d.unit_id = u.unit_id
         and f.fisc_pd = f.fisc_pd
         and s.start_dt >= f.start_date
         and s.end_dt < f.end_date
         and d.start_dt <= s.end_dt
         and d.end_dt > s.start_dt
         and d.company = u.company
         and s.shift_code = d.shift_code) as calc_maint_dt
    from unit_dept_comp_main u
      join f
        on u.company = f.company
    where u.company = :g_company
      and f.start_date < sysdate
      and f.end_date >=any
       (select e.start_dt
        from unit_downtime e
        where e.company = u.company
          and e.unit_id = u.unit_id
          and e.downtime_calc_dt = :p_downtime_calc_dt)
call     count       cpu    elapsed       disk      query    current        rows
Parse        1      0.01       0.01          0          0          0           0
Execute      1    110.31     114.34       1358    2033031      74796       35662
Fetch        0      0.00       0.00          0          0          0           0
total        2    110.32     114.35       1358    2033031      74796       35662
Misses in library cache during parse: 1
Misses in library cache during execute: 1
Optimizer mode: ALL_ROWS
Parsing user id: 32 
Rows     Row Source Operation
  35662  TEMP TABLE TRANSFORMATION  (cr=15420 pr=909 pw=4 time=1416590 us)
      1   LOAD AS SELECT  (cr=14243 pr=737 pw=4 time=897903 us)
    797    HASH GROUP BY (cr=14243 pr=737 pw=0 time=863230 us)
    834     VIEW  (cr=14243 pr=737 pw=0 time=860507 us)
    834      SORT GROUP BY (cr=14243 pr=737 pw=0 time=858836 us)
   1006       NESTED LOOPS OUTER (cr=14243 pr=737 pw=0 time=2222986 us)
   1006        NESTED LOOPS OUTER (cr=13235 pr=733 pw=0 time=2195945 us)
   1006         NESTED LOOPS  (cr=11221 pr=626 pw=0 time=879284 us)
    839          HASH JOIN  (cr=1312 pr=411 pw=0 time=30725 us)
    801           TABLE ACCESS BY INDEX ROWID UNIT_DOWNTIME (cr=745 pr=0 pw=0 time=4057 us)
    801            INDEX RANGE SCAN UNIT_DOWNTIME_DMWIX3 (cr=6 pr=0 pw=0 time=844 us)(object id 98246)
  34925           TABLE ACCESS FULL UNIT_DOWNTIME (cr=567 pr=411 pw=0 time=503836 us)
   1006          TABLE ACCESS BY INDEX ROWID UNIT_DOWNTIME (cr=9909 pr=215 pw=0 time=316547 us)
   9992           INDEX RANGE SCAN UNIT_DOWNTIME_IX1 (cr=904 pr=177 pw=0 time=244588 us)(object id 13200)
   1006         TABLE ACCESS BY INDEX ROWID O_WO (cr=2014 pr=107 pw=0 time=332243 us)
   1006          INDEX UNIQUE SCAN O_WO_IX0 (cr=1008 pr=59 pw=0 time=170868 us)(object id 12787)
   1006        TABLE ACCESS BY INDEX ROWID LOC_MAINT (cr=1008 pr=4 pw=0 time=38706 us)
   1006         INDEX UNIQUE SCAN LOC_MAINT_IX0 (cr=2 pr=1 pw=0 time=8583 us)(object id 12574)
  35662   HASH JOIN RIGHT SEMI (cr=1177 pr=172 pw=0 time=411534 us)
    801    TABLE ACCESS BY INDEX ROWID UNIT_DOWNTIME (cr=745 pr=0 pw=0 time=4055 us)
    801     INDEX RANGE SCAN UNIT_DOWNTIME_DMWIX3 (cr=6 pr=0 pw=0 time=843 us)(object id 98246)
879375    HASH JOIN  (cr=432 pr=172 pw=0 time=896062 us)
    125     VIEW  (cr=1 pr=0 pw=0 time=1419 us)
    156      WINDOW SORT (cr=1 pr=0 pw=0 time=947 us)
    156       INDEX RANGE SCAN FISCAL_CAL_IX1 (cr=1 pr=0 pw=0 time=170 us)(object id 12304)
   7035     TABLE ACCESS FULL UNIT_DEPT_COMP_MAIN (cr=431 pr=172 pw=0 time=458753 us)
Elapsed times include waiting on following events:
  Event waited on                             Times   Max. Wait  Total Waited
  ----------------------------------------   Waited  ----------  ------------
  control file sequential read                    5        0.00          0.02
  db file sequential read                       780        0.04          2.11
  db file scattered read                         67        0.01          0.16
  direct path write temp                          1        0.00          0.00
  log file switch completion                      3        0.08          0.15
  SQL*Net message to client                       1        0.00          0.00
  virtual circuit status                         17       29.99        496.46
  SQL*Net message from client                     1      496.46        496.46-----
From C# program calling PL/SQL procedure (about three hours):
INSERT INTO PD_DWN_STAT (COMPANY, UNIT_ID, PERIOD, OPER_DT, MAINT_DT) WITH D
  AS (SELECT C.COMPANY, C.UNIT_ID, MIN(C.START_DT) AS START_DT,
  MIN(C.SHIFT_CODE) AS SHIFT_CODE, C.END_DT FROM (SELECT A.COMPANY, A.UNIT_ID,
   A.START_DT, G.SHIFT_CODE AS SHIFT_CODE, MAX(COALESCE(B.END_DT, SYSDATE))
  AS END_DT FROM UNIT_DOWNTIME A JOIN UNIT_DOWNTIME B ON A.COMPANY =
  B.COMPANY AND A.UNIT_ID = B.UNIT_ID LEFT OUTER JOIN O_WO O ON O.COMPANY =
  A.COMPANY AND O.WO_NO = A.WO_NO LEFT OUTER JOIN LOC_MAINT G ON G.COMPANY =
  O.COMPANY AND G.LOCATION = O.LOCATION WHERE A.START_DT >= B.START_DT AND
  A.START_DT < COALESCE(A.END_DT, SYSDATE) AND B.START_DT < COALESCE(B.END_DT,
   SYSDATE) AND B.START_DT >=ANY (SELECT Z.START_DT FROM UNIT_DOWNTIME Z
  WHERE Z.DOWNTIME_CALC_DT = :B1 AND Z.UNIT_ID = B.UNIT_ID) GROUP BY
  A.COMPANY, A.UNIT_ID, A.START_DT, G.SHIFT_CODE) C GROUP BY C.COMPANY,
  C.UNIT_ID, C.END_DT), F AS (SELECT COMPANY, FISC_PD, START_DATE,
  LEAD(START_DATE) OVER (PARTITION BY COMPANY ORDER BY FISC_PD) AS END_DATE
  FROM FISCAL_CAL) SELECT U.COMPANY, U.UNIT_ID, F.FISC_PD, (SELECT
  SUM((LEAST(F.END_DATE, :B1 , D.END_DT, S.END_DT) - GREATEST(F.START_DATE,
  D.START_DT, S.START_DT))) * 24 * 60 * 60 *1000 FROM SHIFT S JOIN D ON
  S.COMPANY = D.COMPANY WHERE D.UNIT_ID = U.UNIT_ID AND S.START_DT >=
  F.START_DATE AND S.END_DT < F.END_DATE AND D.START_DT <= S.END_DT AND
  D.END_DT > S.START_DT AND S.COMPANY = U.COMPANY AND S.SHIFT_CODE =
  U.SHIFT_CODE) AS CALC_OPER_DT, (SELECT SUM((LEAST(F.END_DATE, :B1 ,
  D.END_DT, S.END_DT) - GREATEST(F.START_DATE, D.START_DT, S.START_DT))) * 24
  * 60 * 60 *1000 FROM SHIFT S JOIN D ON S.COMPANY = D.COMPANY WHERE
  D.UNIT_ID = U.UNIT_ID AND F.FISC_PD = F.FISC_PD AND S.START_DT >=
  F.START_DATE AND S.END_DT < F.END_DATE AND D.START_DT <= S.END_DT AND
  D.END_DT > S.START_DT AND D.COMPANY = U.COMPANY AND S.SHIFT_CODE =
  D.SHIFT_CODE) AS CALC_MAINT_DT FROM UNIT_DEPT_COMP_MAIN U JOIN F ON
  U.COMPANY = F.COMPANY WHERE U.COMPANY = :B2 AND F.START_DATE < SYSDATE AND
  F.END_DATE >=ANY (SELECT E.START_DT FROM UNIT_DOWNTIME E WHERE E.COMPANY =
  U.COMPANY AND E.UNIT_ID = U.UNIT_ID AND E.DOWNTIME_CALC_DT = :B1 )
call     count       cpu    elapsed       disk      query    current        rows
Parse        1      0.00       0.00          0          0          0           0
Execute      1   7566.98    1870.53   15834828  137765256      75235       35662
Fetch        0      0.00       0.00          0          0          0           0
total        2   7566.98    1870.54   15834828  137765256      75235       35662
Misses in library cache during parse: 1
Misses in library cache during execute: 1
Optimizer mode: ALL_ROWS
Parsing user id: 32     (recursive depth: 1)
Rows     Row Source Operation
  35662  HASH JOIN RIGHT SEMI (cr=1177 pr=391 pw=0 time=521648 us)
    801   TABLE ACCESS BY INDEX ROWID UNIT_DOWNTIME (cr=745 pr=0 pw=0 time=4842 us)
    801    INDEX RANGE SCAN UNIT_DOWNTIME_DMWIX3 (cr=6 pr=0 pw=0 time=829 us)(object id 98246)
879375   HASH JOIN  (cr=432 pr=391 pw=0 time=1778117 us)
    125    VIEW  (cr=1 pr=0 pw=0 time=2041 us)
    156     WINDOW SORT (cr=1 pr=0 pw=0 time=1812 us)
    156      FILTER  (cr=1 pr=0 pw=0 time=493 us)
    156       INDEX RANGE SCAN FISCAL_CAL_IX1 (cr=1 pr=0 pw=0 time=332 us)(object id 12304)
   7035    TABLE ACCESS FULL UNIT_DEPT_COMP_MAIN (cr=431 pr=391 pw=0 time=734863 us)
-----

Answer 1

I tried the DistribTx=0 option which had the strange effect of causing the update to blow up with a "ORA-28115: policy with check option violation". The table does indeed have a DBMS_RLS policy on it, but why DistribTx=0 started making it fail I don't know. Anyway, that led me to use an Oracle user that wasn't subject to the policy rules and the query was fast again. In other words, trying the DistribTx was helpful in that it led me to discover that something in the DBMS_RLS setup was making the optimizer go mad. It wasn't OLEDB per se that was the problem; using the right (wrong) Oracle user, I was able to make the performance stink in SQL Developer too.