Instance is down due to error 4031

Question

in RAC (10.2.0.4, AIX, 64bit), instance 2 is down, check altert log,
ORA-04031: unable to allocate 8416 bytes of shared memory ("shared pool","unknown object","sga heap(1,0)","KSXR large reply queue")
LCK0: terminating instance due to error 4031
( dba broungt it up.)
( check related trace )
=================================
Begin 4031 Diagnostic Information
=================================
The following information assists Oracle in diagnosing
causes of ORA-4031 errors. This trace may be disabled
by setting the init.ora _4031_dump_bitvec = 0
=====================================
Allocation Request Summary Informaton
=====================================
Current information setting: 04014fff
SGA Heap Dump Interval=3600 seconds
Dump Interval=300 seconds
Last Dump Time=09/19/2010 15:18:42
Dump Count=1
Allocation request for: KSXR large reply queue
Heap: 700000010036770, size: 8416
HEAP DUMP heap name="sga heap(1,0)" desc=700000010036770
extent sz=0xfe0 alt=216 het=32767 rec=9 flg=-126 opc=0
parent=0 owner=0 nex=0 xsz=0x1000000
latch set 1 of 2
durations enabled for this heap
reserved granules for root 0 (granule size 16777216)
check memory,
sql> show parameter shared_
NAME TYPE VALUE
hi_shared_memory_address integer 0
max_shared_servers integer
shared_memory_address integer 0
shared_pool_reserved_size big integer 15938355
shared_pool_size big integer 304M
shared_server_sessions integer
shared_servers integer 0
sql>show parameter sga_target
NAME TYPE VALUE
sga_target big integer 7G
sql>show parameter cursor_sharing
NAME TYPE VALUE
cursor_sharing string EXACT
the reason seems from surface is shared_pool is not big enough, but we can see that sga is pretty big,
so it maybe shared pool fragmentation problem.
by the way, the instance down's resaon, is due to
terminating instance due to error 4031
question is here:
1. what is the reasoan for instance shutdown?
2. how to prevernt it happening again? do we need to run flush shared pool periodically, or cursor_sharing to simailar or force?
thanks

Answer 1

-- Display shared pool advice
set lines 100
set pages 999
column c1 heading 'Pool |Size(M)'
column c2 heading 'Size|Factor'
column c3 heading 'Est|LC(M) '
column c4 heading 'Est LC|Mem. Obj.'
column c5 heading 'Est|Time|Saved|(sec)'
column c6 heading 'Est|Parse|Saved|Factor'
column c7 heading 'Est|Object Hits' format 999,999,999
SELECT
2 shared_pool_size_for_estimate c1,
3 shared_pool_size_factor c2,
4 estd_lc_size c3,
5 estd_lc_memory_objects c4,
6 estd_lc_time_saved c5,
7 estd_lc_time_saved_factor c6,
8 estd_lc_memory_object_hits c7
9 FROM
10 v$shared_pool_advice;
Est Est
Time Parse
Pool Size Est Est LC Saved Saved Est
Size(M) Factor LC(M) Mem. Obj. (sec) Factor Object Hits
1088 .3736 326 22232 65966 .8958 4,995,890
1392 .478 628 38639 68647 .9322 5,044,957
1696 .5824 931 56089 70007 .9506 5,068,865
2000 .6868 1234 73521 70497 .9573 5,084,797
2304 .7912 1537 93244 70941 .9633 5,106,783
2608 .8956 1826 114421 71955 .9771 5,140,729
2912 1 2129 135390 73643 1 5,177,134
3216 1.1044 2431 157697 75513 1.0254 5,202,416
3520 1.2088 2735 179011 77204 1.0484 5,221,651
3824 1.3132 3038 203097 78823 1.0703 5,234,965
4128 1.4176 3341 221760 80193 1.0889 5,244,432
4432 1.522 3610 244317 80910 1.0987 5,252,430
4736 1.6264 3761 255231 81101 1.1013 5,256,343
5040 1.7308 3896 269167 81125 1.1016 5,258,292
5344 1.8352 3896 269167 81127 1.1016 5,259,026
5648 1.9396 3896 269167 81128 1.1016 5,259,677
5952 2.044 3896 269167 81129 1.1017 5,260,326
SELECT
f.pool
, f.name
, s.sgasize
, f.bytes
, ROUND(f.bytes/s.sgasize*100, 2) "% Free"
FROM
(SELECT SUM(bytes) sgasize, pool FROM v$sgastat GROUP BY pool) s
, v$sgastat f
WHERE
f.name = 'free memory'
AND f.pool = s.pool
/ 2 3 4 5 6 7 8 9 10 11 12 13
Pool Name Allocated Free % Free
shared pool free memory 3,053,469,832 257,316,432 8.43
large pool free memory 218,103,808 217,027,808 99.51
java pool free memory 218,103,808 218,103,808 100
rem library.sql - lists the library cache
prompt
prompt =========================
prompt LIBRARY CACHE MISS RATIO
prompt =========================
prompt (If > 1 then increase the shared_pool_size in init.ora)
prompt
column "LIBRARY CACHE MISS RATIO" format 99.9999
column "executions" format 999,999,999
column "Cache misses while executing" format 999,999,999
select sum(pins) "executions", sum(reloads) "Cache misses while executing",
(((sum(reloads)/sum(pins)))) "LIBRARY CACHE MISS RATIO"
from v$librarycache;
prompt
prompt =========================
prompt Library Cache Section
prompt =========================
prompt hit ratio should be > 70, and pin ratio > 70 ...
prompt
column "reloads" format 999,999,999
select namespace, trunc(gethitratio * 100) "Hit ratio",
trunc(pinhitratio * 100) "pin hit ratio", reloads "reloads"
from v$librarycache;
NAMESPACE Hit ratio pin hit ratio reloads
SQL AREA 33 44 108,245
TABLE/PROCEDURE 80 99 4,340
BODY 93 99 11
TRIGGER 72 96 0
INDEX 43 60 22
CLUSTER 95 97 3
OBJECT 100 100 0
PIPE 100 100 0
JAVA SOURCE 100 100 0
JAVA RESOURCE 100 100 0
JAVA DATA 100 100 0
select NAMESPACE, GETHITRATIO from v$librarycache;
NAMESPACE GETHITRATIO
SQL AREA .35270614
TABLE/PROCEDURE .806300666
BODY .939540508
TRIGGER .727272727
INDEX .432343234
CLUSTER .956873315
OBJECT 1
PIPE 1
JAVA SOURCE 1
JAVA RESOURCE 1
JAVA DATA 1
---I saw the hit ratio is really low...
THANKS