Performance problems between dev and prod

Question

I run the same query with identical data and indexes, but one system takes a 0.01 seconds to run while the production system takes 1.0 seconds to run. TKprof for dev is:
Rows Row Source Operation
1 TABLE ACCESS BY INDEX ROWID VAP_BANDVALUE
3 NESTED LOOPS
1 NESTED LOOPS
41 NESTED LOOPS
41 NESTED LOOPS
1 TABLE ACCESS BY INDEX ROWID VAP_PACKAGE
1 INDEX UNIQUE SCAN SYS_C0032600 (object id 51356)
41 TABLE ACCESS BY INDEX ROWID VAP_BANDELEMENT
41 AND-EQUAL
82 INDEX RANGE SCAN IDX_BE2 (object id 53559)
41 INDEX RANGE SCAN IDX_BE1 (object id 53558)
41 TABLE ACCESS BY INDEX ROWID VAP_BAND
41 INDEX UNIQUE SCAN SYS_C0034599 (object id 53556)
1 INDEX UNIQUE SCAN SYS_C0032549 (object id 51335)
1 INDEX RANGE SCAN IDX_BV1 (object id 53557)Tkprof for Prod is :
Rows Execution Plan
0 SELECT STATEMENT MODE: ALL_ROWS
1 TABLE ACCESS MODE: ANALYZED (BY INDEX ROWID) OF 'VAP_BANDVALUE' (TABLE)
52001 NESTED LOOPS
26000 NESTED LOOPS
26000 NESTED LOOPS
26000 NESTED LOOPS
1 TABLE ACCESS MODE: ANALYZED (BY INDEX ROWID) OF 'VAP_PACKAGE' (TABLE)
1 INDEX MODE: ANALYZED (UNIQUE SCAN) OF 'SYS_C0018725' (INDEX (UNIQUE))
26000 TABLE ACCESS MODE: ANALYZED (BY INDEX ROWID) OF 'VAP_BANDELEMENT' (TABLE)
26000 INDEX MODE: ANALYZED (RANGE SCAN) OF 'IDX_BE2' (INDEX)
26000 TABLE ACCESS MODE: ANALYZED (BY INDEX ROWID) OF 'VAP_BAND' (TABLE)
26000 INDEX MODE: ANALYZED (UNIQUE SCAN) OF 'SYS_C0030648' (INDEX (UNIQUE))
26000 INDEX MODE: ANALYZED (UNIQUE SCAN) OF 'SYS_C0018674' (INDEX (UNIQUE))
26000 INDEX MODE: ANALYZED (RANGE SCAN) OF 'IDX_BV1' (INDEX).The row count varies greatly. But it shouldn't as the data is the same.
Any ideas?

Answer 1

From DEV you show the Row Source Operations for the query. The column named "Rows" signifies the actual number of rows processed with each step.
From PROD you show the Execution Plan for the query; that is, tkprof was executed with the EXPLAIN option which generates the execution plan as of the time when tkprof was run. The "Rows" column in the Explain Plan output comes from the PLAN_TABLE.CARDINALITY, which represents an estimate by the CBO for the number of rows [expected to be] processed with each step.
So, if by <quote>The row count varies greatly</quote> you meant these "Rows" columns outputs then you're are comparing actuals from a database with estimates from another. Get the Row Source Operations from both.
"Identical data and indexes":
1. data may be the same, but it is not necessarily stored physically the same way.
2. Indexes being the same means their definitions are the same; again, physically they are not necessarily identical
In other words, data in PROD (the way it is stored on disk) may have evolved as a result of discrete deletes/updates/inserts ... in DEV it could be, for example, stored more compact if you took a copy of PROD and moved into DEV. So, the number of blocks for your segments will likely be different between PROD and DEV, the clustering factor for your indexes are likely different, etc ... things which could [and do] influence the CBO. The statistics may be different.
I guess what I'm saying is ... it is quite hard, if not outright impossible, to get two identical databases/instances/load ... hence, don't expect the executions to be 100% identical, even if you have "identical data and indexes". By all means compare between DEV and PROD (make sure you compare the same thing though) and use the observed differences as an indicator for further investigation ... don't chase the goal of 100% identical behavior.
Now, by all means look at that query taking 1 second in PROD ... I have only addressed <quote>The row count varies greatly. But it shouldn't as the data is the same.</quote>