After 10gto11g upgd, 'Planning Data Pull Worker' perf is very slow

Question

Hi all,
We upgraded database from 10g to 11gR2. We have 3 prod instances , they work together. They are ebiz, planning, and configurator. Some of you might be familiar with this kind of setup.
We are running 'Planning Data Pull Worker' in APS instance (planning) instance. This program "SELECTs" (and gets data) from EBIZ instance and "INSERTS" into planning instance.
But after upgrade, this works fine in one instance (PERF6) (half hr), and takes very long like 6+ hrs in another instacen ARCH6.
We figured where the problem is. A select statement.
"select count(1) from apps.MRP_AP3_SALES_ORDERS_V ;"
Eventhough this is not exactly the select been used. We simplified by just trying "count(1)" of the main view which as the culprit.
I am pasting trace of both instance where working fine (PERF6) and where it is taking too long (ARCH6).
Please have a look and help. We opened a sev-1 tar as well , still problem persists.
Thanks,
Gopal

Answer 1

=======================================
___PERF6 instance trace; Where it works fine._
=======================================
SQL> SELECT * FROM table(DBMS_XPLAN.DISPLAY_AWR(('87dtkqstb5adq'),'961766442',null,'ADVANCED'));
PLAN_TABLE_OUTPUT
SQL_ID 87dtkqstb5adq
select count(1) from apps.MRP_AP3_SALES_ORDERS_V
Plan hash value: 961766442
| Id | Operation | Name | Row
s | Bytes | Cost (%CPU)| Time |
PLAN_TABLE_OUTPUT
| 0 | SELECT STATEMENT | |
| | 29253 (100)| |
| 1 | SORT AGGREGATE | |
1 | 189 | | |
| 2 | NESTED LOOPS | |
PLAN_TABLE_OUTPUT
1 | 189 | 29253 (1)| 00:05:52 |
| 3 | HASH JOIN | |
1 | 185 | 29253 (1)| 00:05:52 |
| 4 | NESTED LOOPS OUTER | |
1 | 176 | 29246 (1)| 00:05:51 |
| 5 | NESTED LOOPS OUTER | |
1 | 170 | 29245 (1)| 00:05:51 |
PLAN_TABLE_OUTPUT
| 6 | NESTED LOOPS | |
1 | 161 | 29245 (1)| 00:05:51 |
| 7 | HASH JOIN | | 60
90 | 654K| 19090 (1)| 00:03:50 |
| 8 | TABLE ACCESS BY INDEX ROWID | FND_LANGUAGES |
20 | 100 | 2 (0)| 00:00:01 |
| 9 | INDEX RANGE SCAN | FND_LANGUAGES_N1 |
20 | | 1 (0)| 00:00:01 |
PLAN_TABLE_OUTPUT
| 10 | HASH JOIN | | 62
46 | 640K| 19088 (1)| 00:03:50 |
| 11 | TABLE ACCESS FULL | OE_TRANSACTION_TYPES_TL | 12
97 | 37613 | 10 (0)| 00:00:01 |
| 12 | NESTED LOOPS | |
| | | |
| 13 | NESTED LOOPS | | 41
PLAN_TABLE_OUTPUT
03 | 304K| 19077 (1)| 00:03:49 |
| 14 | NESTED LOOPS | | 41
03 | 220K| 14517 (1)| 00:02:55 |
| 15 | HASH JOIN | | 41
00 | 104K| 9959 (1)| 00:02:00 |
| 16 | TABLE ACCESS FULL | MRP_DERIVED_SO_DEMANDS | 41
00 | 65600 | 17 (0)| 00:00:01 |
PLAN_TABLE_OUTPUT
| 17 | INDEX FAST FULL SCAN | MTL_SYS_ITEMS_SN_N2 | 15
59K| 14M| 1184 (1)| 00:00:15 |
| 18 | TABLE ACCESS BY INDEX ROWID| OE_ORDER_LINES_ALL |
1 | 29 | 2 (0)| 00:00:01 |
| 19 | INDEX UNIQUE SCAN | OE_ORDER_LINES_U1 |
1 | | 1 (0)| 00:00:01 |
| 20 | INDEX UNIQUE SCAN | OE_ORDER_HEADERS_U1 |
1 | | 1 (0)| 00:00:01 |
PLAN_TABLE_OUTPUT
| 21 | TABLE ACCESS BY INDEX ROWID | OE_ORDER_HEADERS_ALL |
1 | 21 | 2 (0)| 00:00:01 |
| 22 | TABLE ACCESS BY INDEX ROWID | MTL_SALES_ORDERS |
1 | 51 | 3 (0)| 00:00:01 |
| 23 | INDEX RANGE SCAN | MTL_SALES_ORDERS_N1 |
1 | | 2 (0)| 00:00:01 |
| 24 | INDEX UNIQUE SCAN | PJM_PROJECT_PARAMETERS_U1 |
PLAN_TABLE_OUTPUT
1 | 9 | 0 (0)| |
| 25 | INDEX RANGE SCAN | OE_ODR_LINES_SN_N2 |
1 | 6 | 2 (0)| 00:00:01 |
| 26 | TABLE ACCESS FULL | AR_SYSTEM_PARAMETERS_ALL |
43 | 387 | 6 (0)| 00:00:01 |
| 27 | INDEX UNIQUE SCAN | GL_SETS_OF_BOOKS_U2 |
1 | 4 | 0 (0)| |
PLAN_TABLE_OUTPUT
Query Block Name / Object Alias (identified by operation id):
1 - SEL$15993197
8 - SEL$15993197 / FNL@SEL$2
9 - SEL$15993197 / FNL@SEL$2
11 - SEL$15993197 / OTTL@SEL$2
PLAN_TABLE_OUTPUT
16 - SEL$15993197 / MDSD@SEL$2
17 - SEL$15993197 / MSIK@SEL$2
18 - SEL$15993197 / OOL@SEL$2
19 - SEL$15993197 / OOL@SEL$2
20 - SEL$15993197 / OOHA@SEL$2
21 - SEL$15993197 / OOHA@SEL$2
22 - SEL$15993197 / MSO@SEL$2
23 - SEL$15993197 / MSO@SEL$2
24 - SEL$15993197 / MPP@SEL$2
25 - SEL$15993197 / OOL1@SEL$2
26 - SEL$15993197 / ASPA@SEL$2
PLAN_TABLE_OUTPUT
27 - SEL$15993197 / GSB@SEL$2
Outline Data
/*+
BEGIN_OUTLINE_DATA
IGNORE_OPTIM_EMBEDDED_HINTS
OPTIMIZER_FEATURES_ENABLE('11.2.0.2')
DB_VERSION('11.2.0.2')
OPT_PARAM('optimizer_dynamic_sampling' 6)
PLAN_TABLE_OUTPUT
ALL_ROWS
OUTLINE_LEAF(@"SEL$15993197")
ELIMINATE_JOIN(@"SEL$5C160134" "MTL_SALES_ORDERS"@"SEL$3")
OUTLINE(@"SEL$5C160134")
MERGE(@"SEL$335DD26A")
OUTLINE(@"SEL$1")
OUTLINE(@"SEL$335DD26A")
MERGE(@"SEL$3")
OUTLINE(@"SEL$2")
OUTLINE(@"SEL$3")
INDEX_FFS(@"SEL$15993197" "MSIK"@"SEL$2" ("MRP_SN_SYS_ITEMS"."INVENTORY_IT
PLAN_TABLE_OUTPUT
EM_ID"
"MRP_SN_SYS_ITEMS"."ORGANIZATION_ID"))
FULL(@"SEL$15993197" "MDSD"@"SEL$2")
INDEX_RS_ASC(@"SEL$15993197" "OOL"@"SEL$2" ("OE_ORDER_LINES_ALL"."LINE_ID"
INDEX(@"SEL$15993197" "OOHA"@"SEL$2" ("OE_ORDER_HEADERS_ALL"."HEADER_ID"))
FULL(@"SEL$15993197" "OTTL"@"SEL$2")
INDEX_RS_ASC(@"SEL$15993197" "FNL"@"SEL$2" ("FND_LANGUAGES"."INSTALLED_FLA
G"))
PLAN_TABLE_OUTPUT
INDEX_RS_ASC(@"SEL$15993197" "MSO"@"SEL$2" ("MTL_SALES_ORDERS"."SEGMENT1")
INDEX(@"SEL$15993197" "MPP"@"SEL$2" ("PJM_PROJECT_PARAMETERS"."PROJECT_ID"
"PJM_PROJECT_PARAMETERS"."ORGANIZATION_ID"))
INDEX(@"SEL$15993197" "OOL1"@"SEL$2" ("MRP_SN_ODR_LINES"."LINE_ID"))
FULL(@"SEL$15993197" "ASPA"@"SEL$2")
INDEX(@"SEL$15993197" "GSB"@"SEL$2" ("GL_SETS_OF_BOOKS"."SET_OF_BOOKS_ID")
PLAN_TABLE_OUTPUT
LEADING(@"SEL$15993197" "MSIK"@"SEL$2" "MDSD"@"SEL$2" "OOL"@"SEL$2" "OOHA"
@"SEL$2" "OTTL"@"SEL$2"
"FNL"@"SEL$2" "MSO"@"SEL$2" "MPP"@"SEL$2" "OOL1"@"SEL$2" "ASPA"@"S
EL$2" "GSB"@"SEL$2")
USE_HASH(@"SEL$15993197" "MDSD"@"SEL$2")
USE_NL(@"SEL$15993197" "OOL"@"SEL$2")
USE_NL(@"SEL$15993197" "OOHA"@"SEL$2")
PLAN_TABLE_OUTPUT
NLJ_BATCHING(@"SEL$15993197" "OOHA"@"SEL$2")
USE_HASH(@"SEL$15993197" "OTTL"@"SEL$2")
USE_HASH(@"SEL$15993197" "FNL"@"SEL$2")
USE_NL(@"SEL$15993197" "MSO"@"SEL$2")
USE_NL(@"SEL$15993197" "MPP"@"SEL$2")
USE_NL(@"SEL$15993197" "OOL1"@"SEL$2")
USE_HASH(@"SEL$15993197" "ASPA"@"SEL$2")
USE_NL(@"SEL$15993197" "GSB"@"SEL$2")
SWAP_JOIN_INPUTS(@"SEL$15993197" "MDSD"@"SEL$2")
SWAP_JOIN_INPUTS(@"SEL$15993197" "OTTL"@"SEL$2")
SWAP_JOIN_INPUTS(@"SEL$15993197" "FNL"@"SEL$2")
PLAN_TABLE_OUTPUT
END_OUTLINE_DATA
110 rows selected.
SQL> spool off