Query goes from 3.5 min to 150 min after switch to 64 bit

Question

I had Oracle 10.1.0.3.0 (32 bit) on Linux running a simple query, once per weekday, accessing tables in DB2 v8 on AIX M/F via Oracle Transparent Gateway (OTG) 9.2. The query took ~ 3.5 minutes to complete.
We have moved to Oracle 10.2.0.1.0 (64 bit) on 64 bit Linux. Now the same query takes ~ 150 min to complete. There are no errors or message I can find that tell me something is wrong. We fired the older 32 bit box back up and ran the query. It completed in 3 min 24 sec.
I suspect a compatibility issue between 64 bit Oracle and the older OTG but I don't even know where to start to look for that. Anyone have a suggestion or a similar tale of woe?

Answer 1

Thanks for the tip. I did get to the explain plans for both of these machines and while the order of execution was slightly different on each machine, I did not see where the CPU cost was different or where things were hanging up. I have included the reports for your perusal. The report for each box begins with the machine name, DT09FU04, DT09FU02 respectively:
DT09FU04 - over 3 hours to run
SQL> explain plan into mdm.plan_table
  2  for (
  3  SELECT
  4                      sidr_prdcr_id prod_id,
  5            agql_latitude latitude,
  6            agql_longitude longitude,
  7            s.sidr_inspctn_type,
  8            insp_last_name,
  9            insp_first_name,
10            insp_ph_num,
11            insp_dist,
12                      sidr_actl_cmp_date cmpdate,
13                      sidr_tst_rslt result,
14                      prdr_full_name full_name,
15                      prdr_lcl_name lcl_name,
16                      prdr_legal_desc legal,
17                      agql_smplid cms_id,
18                      agql_ft_snd_ratio snd,
19                      agql_kt6i_spgv_dy p1_dry,
20                      agql_kt6i_spgv_st p1_sat,
21                      agql_kt6i_spgv_ap p1_app,
22                      agql_kt6i_abs p1_abs,
23                      agql_kt6ii_spgv_dy p2_dry,
24                      agql_kt6ii_spgv_st p2_sat,
25                      agql_kt6ii_spgv_ap p2_app,
26                      agql_kt6ii_abs p2_abs,
27                      agql_kt6iii_wear wear,
28                      agql_rmrks_01 rmk1,
29                      agql_rmrks_02 rmk2,
30                      agql_rmrks_03 rmk3,
31                      agql_rmrks_04 rmk4,
32                      agql_rmrks_05 rmk5,
33                      sidr_rmrks srmk,
34                      sidr_rmrks2 srmk2,
35                      sidr_rmrks3 srmk3,
36            sysdate datestamp
37                    FROM
38                      dtcmspd.t_magql@cmscpms_p q,
39                      dtcmspd.t_msidr@cmscpms_p s,
40                      dtcmspd.t_mprdr@cmscpms_p p,
41                      dtcmspd.t_minsp@cmscpms_p i,
42            dual
43                    WHERE
44                      q.agql_smplid = s.sidr_smpl_id AND
45                      i.insp_id_num = s.sidr_inspctr_id AND
46                      p.prdr_prdcr_id = s.sidr_prdcr_id)
47  /
Explained.
SQL> select * from table(dbms_xplan.display);Plan hash value: 2317258752
| Id  | Operation        | Name    | Rows  | Bytes | Cost (%CPU)| Time     | Inst   |IN-OUT|
|   0 | SELECT STATEMENT |         |  2000 |  1794K|   213   (2)| 00:00:03 |        |      |
|*  1 |  HASH JOIN       |         |  2000 |  1794K|   213   (2)| 00:00:03 |        |      |
|   2 |   REMOTE         | T_MPRDR |  2000 |   210K|    52   (0)| 00:00:01 | CMSCP~ | R->S |
|*  3 |   HASH JOIN      |         |  2000 |  1583K|   160   (2)| 00:00:02 |        |      |
|*  4 |    HASH JOIN     |         |  2000 |   603K|   107   (1)| 00:00:02 |        |      |
|   5 |     NESTED LOOPS |         |  2000 |   109K|    61  (12)| 00:00:01 |        |      |
|   6 |      FAST DUAL   |         |     1 |       |     2   (0)| 00:00:01 |        |      |
|   7 |      REMOTE      | T_MINSP |  2000 |   109K|    52   (0)| 00:00:01 | CMSCP~ | R->S |
|   8 |     REMOTE       | T_MSIDR |  2000 |   494K|    52   (0)| 00:00:01 | CMSCP~ | R->S |
|   9 |    REMOTE        | T_MAGQL |  2000 |   980K|    52   (0)| 00:00:01 | CMSCP~ | R->S |
Predicate Information (identified by operation id):
   1 - access("P"."PRDR_PRDCR_ID"="S"."SIDR_PRDCR_ID")
   3 - access("Q"."AGQL_SMPLID"="S"."SIDR_SMPL_ID")
   4 - access("I"."INSP_ID_NUM"="S"."SIDR_INSPCTR_ID")
Remote SQL Information (identified by operation id):
   2 - SELECT "PRDR_PRDCR_ID", "PRDR_FULL_NAME", "PRDR_LCL_NAME", "PRDR_LEGAL_DESC"
       FROM "DTCMSPD"."T_MPRDR" (accessing 'CMSCPMS_P.REGRESS.RDBMS.DEV.US.ORACLE.COM' )
   7 - SELECT "INSP_ID_NUM", "INSP_LAST_NAME", "INSP_FIRST_NAME", "INSP_PH_NUM",
       "INSP_DIST" FROM "DTCMSPD"."T_MINSP" (accessing
       'CMSCPMS_P.REGRESS.RDBMS.DEV.US.ORACLE.COM' )
   8 - SELECT "SIDR_SMPL_ID", "SIDR_INSPCTR_ID", "SIDR_INSPCTN_TYPE",
       "SIDR_PRDCR_ID", "SIDR_ACTL_CMP_DATE", "SIDR_TST_RSLT", "SIDR_RMRKS",
       "SIDR_RMRKS2", "SIDR_RMRKS3" FROM "DTCMSPD"."T_MSIDR" (accessing
       'CMSCPMS_P.REGRESS.RDBMS.DEV.US.ORACLE.COM' )
   9 - SELECT "AGQL_SMPLID", "AGQL_LATITUDE", "AGQL_LONGITUDE",
       "AGQL_FT_SND_RATIO", "AGQL_KT6I_SPGV_DY", "AGQL_KT6I_SPGV_ST", "AGQL_KT6I_SPGV_AP",
       "AGQL_KT6I_ABS", "AGQL_KT6II_SPGV_DY", "AGQL_KT6II_SPGV_ST", "AGQL_KT6II_SPGV_AP",
       "AGQL_KT6II_ABS", "AGQL_KT6III_WEAR", "AGQL_RMRKS_01", "AGQL_RMRKS_02",
       "AGQL_RMRKS_03", "AGQL_RMRKS_04", "AGQL_RMRKS_05" FROM "DTCMSPD"."T_MAGQL"
       (accessing 'CMSCPMS_P.REGRESS.RDBMS.DEV.US.ORACLE.COM' )
46 rows selected.
DT09FU02 - 3 min. 24 sec.
SQL> explain plan into mdm.plan_table
  2  for (
  3  SELECT
  4                      sidr_prdcr_id prod_id,
  5            agql_latitude latitude,
  6            agql_longitude longitude,
  7            s.sidr_inspctn_type,
  8            insp_last_name,
  9            insp_first_name,
10            insp_ph_num,
11            insp_dist,
12                      sidr_actl_cmp_date cmpdate,
13                      sidr_tst_rslt result,
14                      prdr_full_name full_name,
15                      prdr_lcl_name lcl_name,
16                      prdr_legal_desc legal,
17                      agql_smplid cms_id,
18                      agql_ft_snd_ratio snd,
19                      agql_kt6i_spgv_dy p1_dry,
20                      agql_kt6i_spgv_st p1_sat,
21                      agql_kt6i_spgv_ap p1_app,
22                      agql_kt6i_abs p1_abs,
23                      agql_kt6ii_spgv_dy p2_dry,
24                      agql_kt6ii_spgv_st p2_sat,
25                      agql_kt6ii_spgv_ap p2_app,
26                      agql_kt6ii_abs p2_abs,
27                      agql_kt6iii_wear wear,
28                      agql_rmrks_01 rmk1,
29                      agql_rmrks_02 rmk2,
30                      agql_rmrks_03 rmk3,
31                      agql_rmrks_04 rmk4,
32                      agql_rmrks_05 rmk5,
33                      sidr_rmrks srmk,
34                      sidr_rmrks2 srmk2,
35                      sidr_rmrks3 srmk3,
36            sysdate datestamp
37                    FROM
38                      dtcmspd.t_magql@cmscpms_p q,
39                      dtcmspd.t_msidr@cmscpms_p s,
40                      dtcmspd.t_mprdr@cmscpms_p p,
41                      dtcmspd.t_minsp@cmscpms_p i,
42            dual
43                    WHERE
44                      q.agql_smplid = s.sidr_smpl_id AND
45                      i.insp_id_num = s.sidr_inspctr_id AND
46                      p.prdr_prdcr_id = s.sidr_prdcr_id)
47  /
Explained.
SQL> select * from table(dbms_xplan.display);
Plan hash value: 3130898149
| Id  | Operation               | Name | Rows  | Bytes | Cost (%CPU)| Time     | Inst   |IN-OUT|
|   0 | SELECT STATEMENT        |      |  2000 |   695K|   213   (2)| 00:00:03 |        |      |
|*  1 |  HASH JOIN              |      |  2000 |   695K|   213   (2)| 00:00:03 |        |      |
|   2 |   REMOTE                |      |  2000 |   195K|    52   (0)| 00:00:01 | CMSCP~ | R->S |
|*  3 |   HASH JOIN             |      |  2000 |   500K|   160   (2)| 00:00:02 |        |      |
|   4 |    REMOTE               |      |  2000 |   195K|    52   (0)| 00:00:01 | CMSCP~ | R->S |
|*  5 |    HASH JOIN            |      |  2000 |   304K|   107   (1)| 00:00:02 |        |      |
|   6 |     MERGE JOIN CARTESIAN|      |  2000 |   109K|    61  (12)| 00:00:01 |        |      |
|   7 |      FAST DUAL          |      |     1 |       |     2   (0)| 00:00:01 |        |      |
|   8 |      REMOTE             |      |  2000 |   109K|    52   (0)| 00:00:01 | CMSCP~ | R->S |
|   9 |     REMOTE              |      |  2000 |   195K|    52   (0)| 00:00:01 | CMSCP~ | R->S |
Predicate Information (identified by operation id):
   1 - access("P"."PRDR_PRDCR_ID"="S"."SIDR_PRDCR_ID")
   3 - access("Q"."AGQL_SMPLID"="S"."SIDR_SMPL_ID")
   5 - access("I"."INSP_ID_NUM"="S"."SIDR_INSPCTR_ID")23 rows selected.