DB link performance.

Question

HI Experts
One of our developers is facing performance issue on one query using dblinks.
When we run the query in remote db it is taking 4-5 mins.
When we execute the query from his local db using db links it is never ending.
Looking at this initially we are suspecting to be network issue.
we did some testing and found
Scenario 1
When we execute the query from his local db which is using some of his local tables and some remote tables it never ends.
We did session tracing and found that it waiting on
WAIT #1: nam='SQL*Net message from dblink' ela= 43367 driver id=675562835 #bytes=1 p3=0 obj#=-1 tim=10635854911875
WAIT #1: nam='SQL*Net message to dblink' ela= 1 driver id=675562835 #bytes=1 p3=0 obj#=-1 tim=10635854911949
WAIT #1: nam='SQL*Net message from dblink' ela= 42156 driver id=675562835 #bytes=1 p3=0 obj#=-1 tim=10635854954117
WAIT #1: nam='SQL*Net message to dblink' ela= 0 driver id=675562835 #bytes=1 p3=0 obj#=-1 tim=10635854954193
Scenario 2
When we execute the query from his local db using all the tables in the remote db using dblinks it is executing in 6-7 mins.
Problem is occuring when we use some of the tables in his local schema and some table in his remote db schema .
Tables that exist in his local schema are very small.
At this stage we are not sure whether this could be due to network or some other issue.
Any Ideas?
Thanks in Advance.
Mehtab

Answer 1

You didn't post a 4 digit database version local 10.2.0.3 remote is 9.2.0.7
- You didn't post a version
- You didn't post the actual statement
SELECT /*+DRIVING_SITE( )*/ DISTINCT dsnh.notif_nr
,dsnh.sales_org_id
,dsnh.sales_org_name
,dsnh.fiscal_year
,dsnh.fiscal_period
,dsnh.fiscal_week_period
,dsnh.notif_type
,dsnh.notif_type_desc
,dsnh.repair_scenario
,dsnh.product_class_code
,dsnh.shipto_partner_nr
,dsnh.sales_channel
,dsnh.channel_type233 characters
,dsnh.header_warranty_code
,dsnh.depot_partner_name
,dsnh.shipto_partner_name
FROM dn_service_notification_hdr@dblink dsnh,
sap_customer@dblink sc,
sys_statuses@dblink systbl,
external_internal_tbl@dblink exttbl
WHERE dsnh.shipto_partner_nr = sc.sap_cust_no
AND dsnh.notif_nr = systbl.doc_nr
AND systbl.sys_status = exttbl.ext_status
AND UPPER(dsnh.notif_type)
IN (SELECT UPPER(notif_type)
FROM opc_notif_type_1
WHERE report_name = 'NEW REPAIR VOLUME'
AND service_description = 'Carry In'
AND UPPER(dsnh.notif_type)||' - '||UPPER(exttbl.ext_status)
NOT IN (SELECT UPPER(notif_type)||' - '||UPPER(cancel_flag_value)
FROM opc_notif_type_1
WHERE report_name = 'NEW REPAIR VOLUME'
AND service_description = 'Carry In'
AND dsnh.repair_scenario IN (SELECT repair_scenario
FROM opc_repair_scenario_1
WHERE report_name = 'NEW REPAIR VOLUME'
AND service_description = 'Carry In'
AND UPPER(sc.sales_district) IN (SELECT UPPER(sales_district)
FROM opc_sales_district_1
WHERE report_name = 'NEW REPAIR VOLUME'
AND service_description = 'Carry In'
AND dsnh.sales_org_id IN (SELECT sales_org_id
FROM opc_sales_org_1 )
AND dsnh.header_warranty_code <>'OO'
AND TO_DATE(TO_CHAR(dsnh.notification_creation_dt,'MM/DD/RRRR'),'MM/DD/RRRR')
BETWEEN TO_DATE('03/02/2008','MM/DD/RRRR')
AND TO_DATE('03/02/2008','MM/DD/RRRR')
AND UPPER(dsnh.product_class_code) <> UPPER('3P')
- You didn't post an EXPLAIN PLAN
SQL> /
PLAN_TABLE_OUTPUT
Plan hash value: 1508264907
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | Inst |IN-OUT|
| 0 | SELECT STATEMENT | | 1 | 750 | 354K (1)| 01:10:54 | | |
| 1 | HASH UNIQUE | | 1 | 750 | 354K (1)| 01:10:54 | | |
|* 2 | FILTER | | | | | | | |
|* 3 | HASH JOIN | | 1 | 714 | 351K (1)| 01:10:24 | | |
| 4 | MERGE JOIN CARTESIAN | | 1 | 95 | 12 (0)| 00:00:01 | | |
| 5 | MERGE JOIN CARTESIAN | | 1 | 64 | 9 (0)| 00:00:01 | | |
PLAN_TABLE_OUTPUT
| 6 | MERGE JOIN CARTESIAN| | 1 | 60 | 6 (0)| 00:00:01 | | |
|* 7 | TABLE ACCESS FULL | OPC_REPAIR_SCENARIO_1 | 1 | 27 | 3 (0)| 00:00:01 | | |
| 8 | BUFFER SORT | | 1 | 33 | 3 (0)| 00:00:01 | | |
|* 9 | TABLE ACCESS FULL | OPC_NOTIF_TYPE_1 | 1 | 33 | 3 (0)| 00:00:01 | | |
| 10 | BUFFER SORT | | 2 | 8 | 6 (0)| 00:00:01 | | |
| 11 | TABLE ACCESS FULL | OPC_SALES_ORG_1 | 2 | 8 | 3 (0)| 00:00:01 | | |
| 12 | BUFFER SORT | | 2 | 62 | 9 (0)| 00:00:01 | | |
|* 13 | TABLE ACCESS FULL | OPC_SALES_DISTRICT_1 | 2 | 62 | 3 (0)| 00:00:01 | | |
|* 14 | FILTER | | 2273 | 1374K| 351K (1)| 01:10:24 | | |
| 15 | REMOTE | | | | | | ASRDMP | R->S |
|* 16 | TABLE ACCESS FULL | OPC_NOTIF_TYPE_1 | 1 | 39 | 3 (0)| 00:00:01 | | |
PLAN_TABLE_OUTPUT
Predicate Information (identified by operation id):
2 - filter( NOT EXISTS (SELECT /*+ */ 0 FROM "OPC_NOTIF_TYPE_1" "OPC_NOTIF_TYPE_1" WHERE
"SERVICE_DESCRIPTION"='Carry In' AND "REPORT_NAME"='NEW REPAIR VOLUME' AND LNNVL(UPPER(:B1)||' -
'||UPPER(:B2)<>UPPER("NOTIF_TYPE")||' - '||UPPER("CANCEL_FLAG_VALUE"))))
3 - access(UPPER("DSNH"."NOTIF_TYPE")=UPPER("NOTIF_TYPE") AND
"REPAIR_SCENARIO"=TO_NUMBER("DSNH"."REPAIR_SCENARIO") AND "SALES_ORG_ID"=TO_NUMBER("DSNH"."SALES_ORG_ID")
AND UPPER("SC"."SALES_DISTRICT")=UPPER("SALES_DISTRICT"))
PLAN_TABLE_OUTPUT
7 - filter("REPORT_NAME"='NEW REPAIR VOLUME' AND "SERVICE_DESCRIPTION"='Carry In')
9 - filter("SERVICE_DESCRIPTION"='Carry In' AND "REPORT_NAME"='NEW REPAIR VOLUME')
13 - filter("SERVICE_DESCRIPTION"='Carry In' AND "REPORT_NAME"='NEW REPAIR VOLUME')
14 - filter(TO_DATE(TO_CHAR(INTERNAL_FUNCTION("DSNH"."NOTIFICATION_CREATION_DT"),'MM/DD/RRRR'),'MM/DD/RRR
R')<=TO_DATE('2008-03-02 00:00:00', 'yyyy-mm-dd hh24:mi:ss') AND
TO_DATE(TO_CHAR(INTERNAL_FUNCTION("DSNH"."NOTIFICATION_CREATION_DT"),'MM/DD/RRRR'),'MM/DD/RRRR')>=TO_DATE('
2008-03-02 00:00:00', 'yyyy-mm-dd hh24:mi:ss'))
16 - filter("SERVICE_DESCRIPTION"='Carry In' AND "REPORT_NAME"='NEW REPAIR VOLUME' AND
LNNVL(UPPER(:B1)||' - '||UPPER(:B2)<>UPPER("NOTIF_TYPE")||' - '||UPPER("CANCEL_FLAG_VALUE")))
Remote SQL Information (identified by operation id):
PLAN_TABLE_OUTPUT
15 - SELECT "A1"."NOTIF_NR","A1"."SALES_ORG_ID","A1"."SALES_ORG_NAME","A1"."FISCAL_YEAR","A1"."FISCAL_PER
IOD","A1"."FISCAL_WEEK_PERIOD","A1"."NOTIF_TYPE","A1"."NOTIF_TYPE_DESC","A1"."REPAIR_SCENARIO","A1"."PRODUC
T_CLASS_CODE","A1"."SHIPTO_PARTNER_NR","A1"."SALES_CHANNEL","A1"."CHANNEL_TYPE","A1"."HEADER_WARRANTY_CODE"
,"A1"."DEPOT_PARTNER_NAME","A1"."SHIPTO_PARTNER_NAME","A1"."SHIPTO_PARTNER_NR","A1"."NOTIF_NR","A1"."NOTIF_
TYPE","A1"."NOTIF_TYPE","A1"."REPAIR_SCENARIO","A1"."SALES_ORG_ID","A1"."HEADER_WARRANTY_CODE","A1"."NOTIFI
CATION_CREATION_DT","A1"."PRODUCT_CLASS_CODE","A2"."SAP_CUST_NO","A2"."SALES_DISTRICT","A3"."DOC_NR","A3"."
SYS_STATUS","A4"."EXT_STATUS","A4"."EXT_STATUS" FROM "DN_SERVICE_NOTIFICATION_HDR" "A1","SAP_CUSTOMER"
"A2","SYS_STATUSES" "A3","EXTERNAL_INTERNAL_TBL" "A4" WHERE "A3"."SYS_STATUS"="A4"."EXT_STATUS" AND
"A1"."NOTIF_NR"="A3"."DOC_NR" AND "A1"."SHIPTO_PARTNER_NR"="A2"."SAP_CUST_NO" AND
PLAN_TABLE_OUTPUT
UPPER("A1"."PRODUCT_CLASS_CODE")<>'3P' AND "A1"."HEADER_WARRANTY_CODE"<>'OO' (accessing 'ASRDMP' )
- you did post only a very small section of the wait analysis.