Select after upgrade 9i - 10g runs slowly thousand times - urgent

Question

I have select which runs around one second on Oracle 9. After upgrade to 10g this select runs more than two hours. I compute statistics on all tables, all columns and all indexes. This select consist from two parts - every part run on oracle 10g around 0.5 second but together more than 2 hours. When I rewrite it using WITH clausule, it runs 2 second. But I don't want to rewrite selects .. I want to find why it runs slow on 10g.
Below are original and rewrited select with their execution plans.
Any idea or recomendation?
Select:
SELECT * FROM (
SELECT DISTINCT from_fix_ident ident, from_ident_icao icao
     , latitude1 latitude, longitude1 longitude, from_fix_fea_pk src
     , -1 mslink
  FROM l_sky_airway, l_dgn_airway
WHERE l_sky_airway.mslink=l_dgn_airway.mslink
UNION ALL
SELECT DISTINCT to_fix_ident ident, to_ident_icao icao
     , latitude2 latitude, longitude2 longitude, to_fix_fea_pk src
     , -2 mslink
  FROM l_sky_airway, l_dgn_airway
WHERE l_sky_airway.mslink=l_dgn_airway.mslink
UNION ALL
SELECT ident, icao, latitude, longitude, 5 src, mslink FROM l_sky_navaid
UNION ALL
SELECT ident, icao, latitude, longitude, 6 src, mslink FROM l_sky_waypoint)
WHERE ident||';'||icao||';'||src IN (
SELECT ident||';'||icao||';'||src FROM (
SELECT from_fix_ident ident
     , from_ident_icao icao
     , latitude1 latitude
     , longitude1 longitude
     , from_fix_fea_pk src
  FROM l_sky_airway, l_dgn_airway
WHERE l_sky_airway.mslink = l_dgn_airway.mslink
UNION ALL
SELECT to_fix_ident ident
     , to_ident_icao icao
     , latitude2 latitude
     , longitude2 longitude
     , to_fix_fea_pk src
  FROM l_sky_airway, l_dgn_airway
WHERE l_sky_airway.mslink = l_dgn_airway.mslink
MINUS
SELECT ident
     , icao
     , latitude
     , longitude
     , 5 src
  FROM l_sky_navaid
MINUS
SELECT ident
     , icao
     , latitude
     , longitude
     , 6 src
  FROM l_sky_waypoint
ORDER BY ident, icao, src, mslink, latitude, longitude;Execution plan:
Plan
SELECT STATEMENT  ALL_ROWSCost: 2 003  Bytes: 1 572 402 240  Cardinality: 24 568 785                                               
     29 SORT ORDER BY  Cost: 2 003  Bytes: 1 572 402 240  Cardinality: 24 568 785                                          
          28 FILTER                                     
               12 VIEW EUS. Cost: 825  Bytes: 3 522 880  Cardinality: 55 045                                
                    11 UNION-ALL                           
                         4 HASH UNIQUE  Cost: 398  Bytes: 981 948  Cardinality: 22 317                      
                              3 HASH JOIN  Cost: 142  Bytes: 981 948  Cardinality: 22 317                 
                                   1 TABLE ACCESS FULL TABLE EUS.L_SKY_AIRWAY Cost: 85  Bytes: 290 121  Cardinality: 22 317            
                                   2 TABLE ACCESS FULL TABLE EUS.L_DGN_AIRWAY Cost: 56  Bytes: 691 827  Cardinality: 22 317            
                         8 HASH UNIQUE  Cost: 398  Bytes: 981 948  Cardinality: 22 317                      
                              7 HASH JOIN  Cost: 143  Bytes: 981 948  Cardinality: 22 317                 
                                   5 TABLE ACCESS FULL TABLE EUS.L_SKY_AIRWAY Cost: 85  Bytes: 290 121  Cardinality: 22 317            
                                   6 TABLE ACCESS FULL TABLE EUS.L_DGN_AIRWAY Cost: 56  Bytes: 691 827  Cardinality: 22 317            
                         9 TABLE ACCESS FULL TABLE EUS.L_SKY_NAVAID Cost: 6  Bytes: 57 225  Cardinality: 1 635                      
                         10 TABLE ACCESS FULL TABLE EUS.L_SKY_WAYPOINT Cost: 22  Bytes: 324 712  Cardinality: 8 776                      
               27 VIEW EUS. Cost: 325  Bytes: 12 042  Cardinality: 446                                
                    26 MINUS                           
                         23 MINUS                      
                              20 SORT UNIQUE  Cost: 325  Bytes: 23 128  Cardinality: 446                 
                                   19 UNION-ALL            
                                        15 HASH JOIN  Cost: 145  Bytes: 9 812  Cardinality: 223       
                                             13 TABLE ACCESS FULL TABLE EUS.L_SKY_AIRWAY Cost: 89  Bytes: 2 899  Cardinality: 223 
                                             14 TABLE ACCESS FULL TABLE EUS.L_DGN_AIRWAY Cost: 56  Bytes: 691 827  Cardinality: 22 317 
                                        18 HASH JOIN  Cost: 146  Bytes: 9 812  Cardinality: 223       
                                             16 TABLE ACCESS FULL TABLE EUS.L_SKY_AIRWAY Cost: 89  Bytes: 2 899  Cardinality: 223 
                                             17 TABLE ACCESS FULL TABLE EUS.L_DGN_AIRWAY Cost: 56  Bytes: 691 827  Cardinality: 22 317 
                              22 SORT UNIQUE  Bytes: 512  Cardinality: 16                 
                                   21 TABLE ACCESS FULL TABLE EUS.L_SKY_NAVAID Cost: 6  Bytes: 512  Cardinality: 16            
                         25 SORT UNIQUE  Bytes: 2 992  Cardinality: 88                      
                              24 TABLE ACCESS FULL TABLE EUS.L_SKY_WAYPOINT Cost: 24  Bytes: 2 992  Cardinality: 88                 Rewrited select which run fast:
WITH inselect AS
     (SELECT ident || ';' || icao || ';' || src
        FROM (SELECT from_fix_ident ident, from_ident_icao icao,
                     latitude1 latitude, longitude1 longitude,
                     from_fix_fea_pk src
                FROM l_sky_airway, l_dgn_airway
               WHERE l_sky_airway.mslink = l_dgn_airway.mslink
              UNION ALL
              SELECT to_fix_ident ident, to_ident_icao icao,
                     latitude2 latitude, longitude2 longitude,
                     to_fix_fea_pk src
                FROM l_sky_airway, l_dgn_airway
               WHERE l_sky_airway.mslink = l_dgn_airway.mslink
              MINUS
              SELECT ident, icao, latitude, longitude, 5 src
                FROM l_sky_navaid
              MINUS
              SELECT ident, icao, latitude, longitude, 6 src
                FROM l_sky_waypoint)),
     mainselect AS
     (SELECT DISTINCT from_fix_ident ident, from_ident_icao icao,
                      latitude1 latitude, longitude1 longitude,
                      from_fix_fea_pk src, -1 mslink
                 FROM l_sky_airway, l_dgn_airway
                WHERE l_sky_airway.mslink = l_dgn_airway.mslink
      UNION ALL
      SELECT DISTINCT to_fix_ident ident, to_ident_icao icao,
                      latitude2 latitude, longitude2 longitude,
                      to_fix_fea_pk src, -2 mslink
                 FROM l_sky_airway, l_dgn_airway
                WHERE l_sky_airway.mslink = l_dgn_airway.mslink
      UNION ALL
      SELECT ident, icao, latitude, longitude, 5 src, mslink
        FROM l_sky_navaid
      UNION ALL
      SELECT ident, icao, latitude, longitude, 6 src, mslink
        FROM l_sky_waypoint)
SELECT   *
    FROM mainselect
   WHERE ident || ';' || icao || ';' || src IN (SELECT *
                                                  FROM inselect)
ORDER BY ident, icao, src, mslink, latitude, longitude;
Plan
SELECT STATEMENT  ALL_ROWSCost: 550 336  Bytes: 2 383 172 145  Cardinality: 24 568 785                                             
    31 SORT ORDER BY  Cost: 550 336  Bytes: 2 383 172 145  Cardinality: 24 568 785                                         
        30 HASH JOIN  Cost: 2 647  Bytes: 2 383 172 145  Cardinality: 24 568 785                                     
            17 VIEW VIEW SYS.VW_NSO_1 Cost: 1 173  Bytes: 1 472 922  Cardinality: 44 634                                 
                16 HASH UNIQUE  Cost: 1 173  Bytes: 1 205 118  Cardinality: 44 634                             
                    15 VIEW EUS. Cost: 828  Bytes: 1 205 118  Cardinality: 44 634                         
                        14 MINUS                     
                            11 MINUS                 
                                8 SORT UNIQUE  Cost: 828  Bytes: 2 314 600  Cardinality: 44 634             
                                    7 UNION-ALL         
                                        3 HASH JOIN  Cost: 142  Bytes: 981 948  Cardinality: 22 317     
                                            1 TABLE ACCESS FULL TABLE EUS.L_SKY_AIRWAY Cost: 85  Bytes: 290 121  Cardinality: 22 317 
                                            2 TABLE ACCESS FULL TABLE EUS.L_DGN_AIRWAY Cost: 56  Bytes: 691 827  Cardinality: 22 317 
                                        6 HASH JOIN  Cost: 143  Bytes: 981 948  Cardinality: 22 317     
                                            4 TABLE ACCESS FULL TABLE EUS.L_SKY_AIRWAY Cost: 85  Bytes: 290 121  Cardinality: 22 317 
                                            5 TABLE ACCESS FULL TABLE EUS.L_DGN_AIRWAY Cost: 56  Bytes: 691 827  Cardinality: 22 317 
                                10 SORT UNIQUE  Bytes: 52 320  Cardinality: 1 635             
                                    9 TABLE ACCESS FULL TABLE EUS.L_SKY_NAVAID Cost: 6  Bytes: 52 320  Cardinality: 1 635         
                            13 SORT UNIQUE  Bytes: 298 384  Cardinality: 8 776                 
                                12 TABLE ACCESS FULL TABLE EUS.L_SKY_WAYPOINT Cost: 22  Bytes: 298 384  Cardinality: 8 776             
            29 VIEW EUS. Cost: 825  Bytes: 3 522 880  Cardinality: 55 045                                 
                28 UNION-ALL                             
                    21 HASH UNIQUE  Cost: 398  Bytes: 981 948  Cardinality: 22 317                         
                        20 HASH JOIN  Cost: 142  Bytes: 981 948  Cardinality: 22 317                     
                            18 TABLE ACCESS FULL TABLE EUS.L_SKY_AIRWAY Cost: 85  Bytes: 290 121  Cardinality: 22 317                 
                            19 TABLE ACCESS FULL TABLE EUS.L_DGN_AIRWAY Cost: 56  Bytes: 691 827  Cardinality: 22 317                 
                    25 HASH UNIQUE  Cost: 398  Bytes: 981 948  Cardinality: 22 317                         
                        24 HASH JOIN  Cost: 143  Bytes: 981 948  Cardinality: 22 317                     
                            22 TABLE ACCESS FULL TABLE EUS.L_SKY_AIRWAY Cost: 85  Bytes: 290 121  Cardinality: 22 317                 
                            23 TABLE ACCESS FULL TABLE EUS.L_DGN_AIRWAY Cost: 56  Bytes: 691 827  Cardinality: 22 317                 
                    26 TABLE ACCESS FULL TABLE EUS.L_SKY_NAVAID Cost: 6  Bytes: 57 225  Cardinality: 1 635                         
                    27 TABLE ACCESS FULL TABLE EUS.L_SKY_WAYPOINT Cost: 22  Bytes: 324 712  Cardinality: 8 776                          Reformated
Message was edited by:
Vlada
Message was edited by:
Vlada

Answer 1

Vlada,
could you please post an properly formatted explain plan output using DBMS_XPLAN.DISPLAY including the "Predicate Information" section below the plan to provide more details regarding your statement. Please use the \[code\] and \[code\] tags to enhance readability of the output provided:
In SQL*Plus:
SET LINESIZE 130
EXPLAIN PLAN FOR <your statement>;
SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY);In order to get a better understanding where your statement spends the time you might want to turn on SQL trace as described here:
[When your query takes too long|http://forums.oracle.com/forums/thread.jspa?threadID=501834]
and post the "tkprof" output here, too.
Could you also provide the information which version of 10g you're currently using? (4-digit version number). Note that 10g introduced a couple of significant changes including CPU costing enabled by default and a different default setting of DBMS_STATS regarding column histograms.
So you might want to re-gather the statistics using the method_opt parameter of the DBMS_STATS.GATHER__STATS procedures explicitly defined as "FOR ALL COLUMNS SIZE 1" to mimic the 9i behaviour.
You also might want to try the "OPTIMIZER_FEATURES_ENABLE" session setting set to "9.2.0" in order to find out if any of the new optimizer features could be causing the issue if re-gathering the statistics as suggested above doesn't make a difference.
Do you know if the tables/indexes in your 9i database also had reasonable statistics?
Regards,
Randolf
Oracle related stuff blog:
http://oracle-randolf.blogspot.com/
SQLTools++ for Oracle:
http://www.sqltools-plusplus.org:7676/
http://sourceforge.net/projects/sqlt-pp/
Edited by: Randolf Geist on Sep 8, 2008 9:32 AM
Added some specific suggestions