DB Performance after upgrade.

Question

All,
We have upgraded our application in a test environment which has the production data. The Database was upgraded from oracle 8.1.7.4 to 10.2.0.3 and the application was upgraded to its latest version.
Prior to upgrade,the database was set to RBO and now after upgrade,we moved to CBO.
There are tables with 1m to 25m rows in the database.Some of the tables have upto 500 columns in it. we do have custom columns added to tables
We're seeing severe performance problem with the application .NET interfaces.
Prior to upgrade,the interface takes .065secs to complete a single transaction and now it is taking upto 1.5sec to process.
ADDM/AWR reports shows:
event waits Time(s) %total call time
db file sequential read 1,071,878 5,605 5 94.8 User I/O
CPU time 386 6.5
log file parallel write 849 10 12 .2 System I/O
control file parallel write 3,600 6 2 .1 System I/O
latch free 668 3 5 .1 Other
enabled trace at session level and generated tkprof which has wait events on 'db file sequential read' and 'SQL*net message from client';
OVERALL TOTALS FOR ALL NON-RECURSIVE STATEMENTS
call count cpu elapsed disk query current rows
Parse 4 0.08 0.06 0 129 0 0
Execute 4 0.00 0.00 0 0 0 0
Fetch 137 140.50 5827.54 1040645 5864173 0 1978
total 145 140.58 5827.61 1040645 5864302 0 1978
Misses in library cache during parse: 3
Elapsed times include waiting on following events:
Event waited on Times Max. Wait Total Waited
---------------------------------------- Waited ---------- ------------
SQL*Net message to client 138 0.00 0.00
SQL*Net message from client 137 1863.66 3030.45
db file sequential read 1040647 0.49 5773.47
latch free 64 0.88 1.65
latch: object queue header operation 1 0.00 0.00
OVERALL TOTALS FOR ALL RECURSIVE STATEMENTS
call count cpu elapsed disk query current rows
Parse 43 0.08 0.09 0 0 0 0
Execute 193 0.10 0.12 2 273 0 0
Fetch 365 0.08 0.15 0 691 0 656
total 601 0.26 0.37 2 964 0 656
Misses in library cache during parse: 33
Misses in library cache during execute: 27
We have found that some of the custom columns being used by the application doesn't have indexes and we see better results after adding a column.
But, i know that we cannot go ahead and add index on each custom column which isn't a good approach.
Appreciate your inputs and thoughts.
Cheers.

Answer 1

Can you explain a bit about what you mean by "application was upgraded to its latest version"? If you're talking about a packaged application, your vendor probably has a pile of documentation that explains how to configure your database in a way that works well for their particular application. There may well be a variety of configuration settings that the vendor recommends or a variety of changes they recommend to the default 10g statistics gathering process.
There may also be application-related reasons for performance to have decreased (i.e. application screens or SQL queries may be doing more work in newer versions than they were doing in older versions). The combination of a new application version and a new database version may also increase hardware requirements-- if you're running applications on the same servers you were using in 2000 when 8.1.7.4 was the new kid on the block, those servers may be overtaxed today.
Other than that, I'd start comparing query plan changes between test & prod. If you can figure out what in the database and/or object statistics is causing the CBO to choose slower plans, you may be able to tweak your statistics gathering process (or potentially even an initialization parameter) to give Oracle the information it needs.
Justin