Oracle 11gR2 Performance Issue

Question

We have Oracle 11gR2 (11.2.0.2) database on HP-UX for GIS application.
We recently upgraded this database from 10g to 11gR2, Everything is working good, no issue expect one month end process. This process hung because of Library Cache Wait (Concurrency). Whenever we start this process the Library Cache Wait is increasing and the process is not completing.
Instance Efficiency Percentages (Target 100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            Buffer Nowait %:  100.00       Redo NoWait %:  100.00
            Buffer  Hit   %:   99.49    In-memory Sort %:  100.00
            Library Hit   %:   22.90        Soft Parse %:   84.22
         Execute to Parse %:   41.06         Latch Hit %:   98.80
Parse CPU to Parse Elapsd %:   26.78     % Non-Parse CPU:   79.40
Shared Pool Statistics        Begin    End
             Memory Usage %:   76.07   67.50
    % SQL with executions>1:   97.69   89.31
  % Memory for SQL w/exec>1:   97.20   98.96
Top 5 Timed Foreground Events
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                                                           Avg
                                                          wait   % DB
Event                                 Waits     Time(s)   (ms)   time Wait Class
library cache lock                3,459,453      12,400      4   55.2 Concurrenc
DB CPU                                            7,840          34.9
cursor: mutex S                     811,819         933      1    4.2 Concurrenc
cursor: pin S wait on X              21,339         813     38    3.6 Concurrenc
db file sequential read              42,997         158      4     .7 User I/O
^LHost CPU (CPUs:    4 Cores:    4 Sockets:    1)
~~~~~~~~         Load Average
               Begin       End     %User   %System      %WIO     %Idle
                0.06      1.76      53.0       1.3       1.0      45.8
Instance CPU
~~~~~~~~~~~~
              % of total CPU for Instance:      54.3
              % of busy  CPU for Instance:     100.1
  %DB time waiting for CPU - Resource Mgr:       0.0
Memory Statistics
~~~~~~~~~~~~~~~~~                       Begin          End
                  Host Mem (MB):     32,666.4     32,666.4
                   SGA use (MB):      3,504.0      3,504.0
                   PGA use (MB):        278.2        324.2
    % Host Mem used for SGA+PGA:        11.58        11.72

Answer 1

I used catupgrd.sql script to upgrade the database, installed 11g on same server in different home and used catupgrd.
Parameter Value_
SQL> show parameter opt
NAME_COL_PLUS_SHOW_PARAM                           TYPE                             VALUE_COL_PLUS_SHOW_PARAM
filesystemio_options                               string                           DIRECTIO
object_cache_optimal_size                          integer                          102400
optimizer_capture_sql_plan_baselines               boolean                          FALSE
optimizer_dynamic_sampling                         integer                          2
optimizer_features_enable                          string                           11.2.0.2
optimizer_index_caching                            integer                          0
optimizer_index_cost_adj                           integer                          100
optimizer_mode                                     string                           ALL_ROWS
optimizer_secure_view_merging                      boolean                          TRUE
optimizer_use_invisible_indexes                    boolean                          FALSE
optimizer_use_pending_statistics                   boolean                          FALSE
optimizer_use_sql_plan_baselines                   boolean                          TRUE
plsql_optimize_level                               integer                          2
SQL>
SQL>
SQL> show parameter cursor
NAME_COL_PLUS_SHOW_PARAM                           TYPE                             VALUE_COL_PLUS_SHOW_PARAM
cursor_bind_capture_destination                    string                           memory+disk
cursor_sharing                                     string                           SIMILAR
cursor_space_for_time                              boolean                          FALSE
open_cursors                                       integer                          2000
session_cached_cursors                             integer                          200
SQL>
SQL> select * from ( select SQL_ID,VERSION_COUNT,EXECUTIONS from v$sqlarea order by VERSION_COUNT desc) where rownum<=20;
SQL_ID        VERSION_COUNT EXECUTIONS
0ynhd4qz3avdh          6453       9198
b1kg0nth4ax9f          6410      15348
7qscuaqtt32zb          4449      17243
duntdcykba52p          2815       1325
g06n424cwup4m          2457       2492
2zgfq8kqqzcm7          1911       1205
b67j985tf4gsw          1868       1740
bmn1m6x3hq8cm          1467        155
fdcvny2d9d6kw           783        405
fphtsy5rd9gkq           401      21702
771v4ntn6mqhg           293      12174
g3dqmahsk5t8f           195       8200
fhrh7858qhrm3           135       2454
8xx3nxsndzyar           112        222
f280fzy0a1bsb            70        263
5rn144gqjkv53            64        125
0v3dvmc22qnam            59         53
f37373tqsbqhz            55         38
f1gu4r0wbhfn3            44         81
dcc7g533zymtm            41         83
SQL ordered by Version Count - From AWR Report_
SQL ordered by Version Count     DB/Inst: GISPROD/gisprod  Snaps: 23062-23063
-> Only Statements with Version Count greater than 20 are displayed
Version
  Count   Executions     SQL Id
   4,449        8,543 7qscuaqtt32zb
Module: gsrvr.exe
SELECT V__276.*, SHAPE.fid,SHAPE.numofpts,SHAPE.entity,SHAPE.points,SHAPE.row
id FROM (SELECT b.OBJECTID,b.ENABLED,b.SHAPE FROM ARCFM.Electric_Network_Juncti
ons b WHERE b.OBJECTID NOT IN (SELECT /*+ HASH_AJ */ SDE_DELETES_ROW_ID FROM ARC
FM.D276 WHERE DELETED_AT IN (SELECT l.lineage_id FROM SDE.state_lineages l WHERE
   2,940          N/A 0ynhd4qz3avdh
SELECT OBJECTID, ENABLED, CREATIONUSER, DATECREATED, DATEMODIFIED, LASTUSE
R, SUBTYPECODE, PHASINGCODE, FEEDERID, FEEDERID2, OPERVOLT, NEUTTYPE, NEU
TSIZE, LOCATIONID, ASSEMBLYCODE, LENGTH, LENGTHSRC, COMMENTS, FDRMGRNONTRA
CEABLE, LABELTEXT, ELECTRICTRACEWEIGHT, FEEDERINFO, RATEDVOLT, CONDUITFLG,
   2,006          N/A b1kg0nth4ax9f
SELECT V__159.*, SHAPE.fid,SHAPE.numofpts,SHAPE.entity,SHAPE.points,SHAPE.row
id FROM (SELECT b.OBJECTID,b.ANCILLARYROLE,b.ENABLED,b.CREATIONUSER,b.DATECREATE
D,b.DATEMODIFIED,b.LASTUSER,b.SUBTYPECODE,b.FACILITYID,b.LOCATIONID,b.PHASINGCOD
E,b.OPERATINGVOLTAGE,b.FEEDERID,b.FEEDERID2,b.ASSEMBLYCODE,b.COMMENTS,b.SYMBOLRO
   1,765           65 duntdcykba52p
Module: gsrvr.exe
SELECT SWITCHTYPE FROM (SELECT b.OBJECTID,b.CREATIONUSER,b.DATECREATED,b.DATEM
ODIFIED,b.LASTUSER,b.FACILITYID,b.PHASINGCODE,b.MANUFACTURER,b.SUBTYPECODE,b.SWI
TCHID,b.SWITCHTYPE,b.SWITCHRATING,b.LEGACYSWITCHID,b.WORKREQUESTID,b.DESIGNID,b.
WORKLOCATIONID,b.WORKFLOWSTATUS,b.WORKFUNCTION FROM ARCFM.SwitchUnit b WHERE b.
   1,467           19 bmn1m6x3hq8cm
Module: gsrvr.exe
select privilege from user_sys_privs where privilege in (:"SYS_B_0",:"SYS_B_1",:
"SYS_B_2",:"SYS_B_3") UNION select privilege from role_sys_privs where privilege
in (:"SYS_B_4",:"SYS_B_5",:"SYS_B_6",:"SYS_B_7")
   1,467           19 bmn1m6x3hq8cm
Module: gsrvr.exe
select privilege from user_sys_privs where privilege in (:"SYS_B_0",:"SYS_B_1",:
"SYS_B_2",:"SYS_B_3") UNION select privilege from role_sys_privs where privilege
in (:"SYS_B_4",:"SYS_B_5",:"SYS_B_6",:"SYS_B_7")
     991          N/A 2zgfq8kqqzcm7
SELECT OBJECTID, ENABLED, CREATIONUSER, DATECREATED, DATEMODIFIED, LASTUSE
R, SUBTYPECODE, PHASINGCODE, FEEDERID, FEEDERID2, OPERVOLT, NEUTTYPE, NEU
TSIZE, LOCATIONID, ASSEMBLYCODE, LENGTH, LENGTHSRC, COMMENTS, FDRMGRNONTRA
CEABLE, LABELTEXT, ELECTRICTRACEWEIGHT, FEEDERINFO, RATEDVOLT, CONSTR, PHA
Active Session History
ASH Report For GISPROD/gisprod
DB Name         DB Id    Instance     Inst Num Release     RAC Host
GISPROD       1774954232 gisprod             1 11.2.0.2.0  NO  hublhp9
CPUs           SGA Size       Buffer Cache        Shared Pool    ASH Buffer Size
   4      4,078M (100%)     1,168M (28.6%)     1,933M (47.4%)              M (%)
          Analysis Begin Time:   06-Feb-12 08:00:00
            Analysis End Time:   06-Feb-12 08:01:00
                 Elapsed Time:         1.0 (mins)
            Begin Data Source:   DBA_HIST_ACTIVE_SESS_HISTORY
                                 in AWR snapshot 23062
              End Data Source:   DBA_HIST_ACTIVE_SESS_HISTORY
                                 in AWR snapshot 23063
                 Sample Count:           2
      Average Active Sessions:        0.33
  Avg. Active Session per CPU:        0.08
                Report Target:   None specified
Top User Events              DB/Inst: GISPROD/gisprod  (Feb 06 08:00 to 08:01)
                                                               Avg Active
Event                               Event Class        % Event   Sessions
CPU + Wait for CPU                  CPU                  50.00       0.17
Top Background Events        DB/Inst: GISPROD/gisprod  (Feb 06 08:00 to 08:01)
                                                               Avg Active
Event                               Event Class     % Activity   Sessions
CPU + Wait for CPU                  CPU                  50.00       0.17
Top Event P1/P2/P3 Values    DB/Inst: GISPROD/gisprod  (Feb 06 08:00 to 08:01)
                  No data exists for this section of the report.
Top Service/Module           DB/Inst: GISPROD/gisprod  (Feb 06 08:00 to 08:01)
Service        Module                   % Activity Action               % Action
SYS$BACKGROUND MMON_SLAVE                    50.00 Auto-Flush Slave A      50.00
SYS$USERS      ArcMap.exe                    50.00 UNNAMED                 50.00
Top Client IDs               DB/Inst: GISPROD/gisprod  (Feb 06 08:00 to 08:01)
                  No data exists for this section of the report.
Top SQL Command Types        DB/Inst: GISPROD/gisprod  (Feb 06 08:00 to 08:01)
-> 'Distinct SQLIDs' is the count of the distinct number of SQLIDs
      with the given SQL Command Type found over all the ASH samples
      in the analysis period
                                           Distinct            Avg Active
SQL Command Type                             SQLIDs % Activity   Sessions
INSERT                                            1      50.00       0.17
SELECT                                            1      50.00       0.17
Top Phases of Execution      DB/Inst: GISPROD/gisprod  (Feb 06 08:00 to 08:01)
                                          Avg Active
Phase of Execution             % Activity   Sessions
SQL Execution                       50.00       0.17
Top SQL with Top Events     DB/Inst: GISPROD/gisprod  (Feb 06 08:00 to 08:01)
                                                        Sampled #
                 SQL ID             Planhash        of Executions     % Activity
Event                          % Event Top Row Source                    % RwSrc
          2prbzh4qfms7u           2500066303                    1          50.00
CPU + Wait for CPU               50.00 FIXED TABLE - FULL                  50.00
insert into wrh$_latch_children (snap_id, dbid, instance_number, latch_hash, c
hild#, gets, misses, sleeps, immediate_gets, immediate_misses, spin_gets, sle
ep1, sleep2, sleep3, sleep4, wait_time) select :snap_id, :dbid, :instance
_number, hash, child#, gets, misses, sleeps, immediate_gets, immediate_misses
          a4w3b09vg8705             52468664                    1          50.00
CPU + Wait for CPU               50.00 SELECT STATEMENT                    50.00
SELECT PROCESS.MM_PX_CURRENT_STATE.*, PROCESS.MM_PX_CURRENT_STATE."ROWID" FROM
PROCESS.MM_PX_CURRENT_STATE
Top SQL with Top Row Sources DB/Inst: GISPROD/gisprod  (Feb 06 08:00 to 08:01)
                                                        Sampled #
                 SQL ID             PlanHash        of Executions     % Activity
Row Source                               % RwSrc Top Event               % Event
          2prbzh4qfms7u           2500066303                    1          50.00
FIXED TABLE - FULL                         50.00 CPU + Wait for CPU        50.00
insert into wrh$_latch_children (snap_id, dbid, instance_number, latch_hash, c
hild#, gets, misses, sleeps, immediate_gets, immediate_misses, spin_gets, sle
ep1, sleep2, sleep3, sleep4, wait_time) select :snap_id, :dbid, :instance
_number, hash, child#, gets, misses, sleeps, immediate_gets, immediate_misses
          a4w3b09vg8705             52468664                    1          50.00
SELECT STATEMENT                           50.00 CPU + Wait for CPU        50.00
SELECT PROCESS.MM_PX_CURRENT_STATE.*, PROCESS.MM_PX_CURRENT_STATE."ROWID" FROM
PROCESS.MM_PX_CURRENT_STATE
Top SQL using literals       DB/Inst: GISPROD/gisprod  (Feb 06 08:00 to 08:01)
                  No data exists for this section of the report.
Top Parsing Module/Action    DB/Inst: GISPROD/gisprod  (Feb 06 08:00 to 08:01)
                  No data exists for this section of the report.
Top PL/SQL Procedures        DB/Inst: GISPROD/gisprod  (Feb 06 08:00 to 08:01)
                  No data exists for this section of the report.
Top Java Workload            DB/Inst: GISPROD/gisprod  (Feb 06 08:00 to 08:01)
                  No data exists for this section of the report.
Top Call Types               DB/Inst: GISPROD/gisprod  (Feb 06 08:00 to 08:01)
Call Type                                     Count % Activity Avg Active
FETCH                                             1      50.00       0.17
Top Sessions                DB/Inst: GISPROD/gisprod  (Feb 06 08:00 to 08:01)
-> '# Samples Active' shows the number of ASH samples in which the session
      was found waiting for that particular event. The percentage shown
      in this column is calculated with respect to wall clock time
      and not total database activity.
-> 'XIDs' shows the number of distinct transaction IDs sampled in ASH
      when the session was waiting for that particular event
-> For sessions running Parallel Queries, this section will NOT aggregate
      the PQ slave activity into the session issuing the PQ. Refer to
      the 'Top Sessions running PQs' section for such statistics.
   Sid, Serial# % Activity Event                             % Event
User                 Program                          # Samples Active     XIDs
       23, 8635      50.00 CPU + Wait for CPU                  50.00
JOHNSONV             ArcMap.exe                             1/6 [ 17%]        0
      150,  963      50.00 CPU + Wait for CPU                  50.00
SYS                  oracle@hublhp9 (M000)                  1/6 [ 17%]        0
Top Blocking Sessions        DB/Inst: GISPROD/gisprod  (Feb 06 08:00 to 08:01)
                  No data exists for this section of the report.
Top Sessions running PQs    DB/Inst: GISPROD/gisprod  (Feb 06 08:00 to 08:01)
                  No data exists for this section of the report.
Top DB Objects               DB/Inst: GISPROD/gisprod  (Feb 06 08:00 to 08:01)
                  No data exists for this section of the report.
Top DB Files                 DB/Inst: GISPROD/gisprod  (Feb 06 08:00 to 08:01)
                  No data exists for this section of the report.
Top Latches                  DB/Inst: GISPROD/gisprod  (Feb 06 08:00 to 08:01)
                  No data exists for this section of the report.
Activity Over Time          DB/Inst: GISPROD/gisprod  (Feb 06 08:00 to 08:01)
-> Analysis period is divided into smaller time slots
-> Top 3 events are reported in each of those slots
-> 'Slot Count' shows the number of ASH samples in that slot
-> 'Event Count' shows the number of ASH samples waiting for
   that event in that slot
-> '% Event' is 'Event Count' over all ASH samples in the analysis period
                         Slot                                   Event
Slot Time (Duration)    Count Event                             Count % Event
08:00:00   (60 secs)        2 CPU + Wait for CPU                    2  100.00
End of Report