Performance issue with new release

Question

Dear all,
11.2 rac with ASM on solaris sparc standard edition.
Client was using 10.2.0.3 on solaris 10 before and it is working fine.
after migrating the data to the new 11g instance ,we've a performance
degradation in the application. application forms 10.1.2.3 .Since we
are using stanard edition, there are not much options available for
tuning. the same application when tested with 11gr2 on windows (non-rac)
is more faster than the current prod and the newly installed ones.
I scheduled statspack and info from statspack
Top 5 Timed Events                                                    Avg %Total
~~~~~~~~~~~~~~~~~~                                                   wait   Call
Event                                            Waits    Time (s)   (ms)   Time
CPU time                                                     2,468          87.0
Streams AQ: qmn coordinator waiting for s           18         103   5742    3.6
gc cr block busy                                   295          89    303    3.2
control file sequential read                   235,472          70      0    2.5
reliable message                                 8,701          24      3     .9
Host CPU  (CPUs: 32  Cores: 4  Sockets: 1)
~~~~~~~~              Load Average
                      Begin     End      User  System    Idle     WIO     WCPU
                       1.20    1.37      2.43    1.25   96.32    0.00    0.91
Instance CPU
~~~~~~~~~~~~                                       % Time (seconds)
                     Host: Total time (s):                441,564.6
                  Host: Busy CPU time (s):                 16,239.3
                   % of time Host is Busy:       3.7
             Instance: Total CPU time (s):                  2,682.5
          % of Busy CPU used for Instance:      16.5
        Instance: Total Database time (s):                  3,249.6
  %DB time waiting for CPU (Resource Mgr):       0.0
Memory Statistics                       Begin          End
~~~~~~~~~~~~~~~~~                ------------ ------------
                  Host Mem (MB):     16,256.0     16,256.0
                   SGA use (MB):      4,078.9      4,078.9
                   PGA use (MB):        418.6        415.2
    % Host Mem used for SGA+PGA:         27.7         27.6
          -------------------------------------------------------------Objects are indexes are the same across the environment.stats are updated, but the process is time consuming in the new environment.
Please let me know how to proceed ?
Kai

Answer 1

Kai,
Please take a look in the sqlnet.ora file on the client computer, you should find it in network/admin directory. In that file, do you see either of these two lines?
sqlnet.authentication_services = (NONE)
sqlnet.authentication_services = (NTS)The above might explain why the configuration is fast on a Windows server, but slow when a Solaris server is accessed.
To Mark's point, execution plans can change between different versions - in fact Oracle Database 11.2.0.2 will perform something called adaptive cursor sharing, where an execution plan (you cannot check this using EXPLAIN PLAN from SQL*Plus) will potentially change as different bind variable values are supplied. However, if the application parses once and then holds the cursor open, the adaptive cursor sharing likely will not happen on future executions within that application - an unfortunate initial parse with unusual bind variable values might trigger a hard parse on 11.2.0.2 (thus possibly changing the execution plan), even when that SQL statement had been previously hard parsed. Randolf Geist recently posted an article related to this topic:
http://oracle-randolf.blogspot.com/2011/01/adaptive-cursor-sharing.html
Even with the Standard Edition of Oracle Database you still have a lot of options for troubleshooting this problem. In a 10046 trace file for a session look at the amount of time spent on the SQL*Net events. Do not look at just the summary provided by TKPROF, but look at the raw trace file. Is each SQL*Net related wait for a significant amount of time? If you do see long times for many of the SQL*Net waits, then you should focus your attention to the client, the server's network configuration, and the network hardware.
On the Standard Edition you can also look at the session level time model statistics and wait events to get a better idea of what is happening in the session. For example, if you have access to a spare Windows client computer you can try a script like the one in this blog article to help you review the time model statistics and dig into the session level time model statistics and wait events for a specific session:
http://hoopercharles.wordpress.com/2010/02/09/working-with-oracle%e2%80%99s-time-model-data-3/
Charles Hooper
Co-author of "Expert Oracle Practices: Oracle Database Administration from the Oak Table"
http://hoopercharles.wordpress.com/
IT Manager/Oracle DBA
K&M Machine-Fabricating, Inc.