Oracle instance crashing when enabling use_indirect_data_buffers=true

Question

I have a Windows 2003 EE server (32bit) with 16GB of ram hosting a 10.2.0.2 Oracle server which is used to support a commercial software package (arcsight). I'm trying to get the Oracle backend to leverage the available system memory. I've read 50-60 different articles and posts regarding AWE and Oracle. I have successfully tuned the userva parameter in order to get the server to boot stable with the /3gb boot parameter. I've gotten to the point that the oracle instance will start up, but within about 30-60 seconds the instance will crash. Below is the information I believe that is relevant:
*.......From computer Registry.........*
AWE_MEMORY_WINDOW = 1288486912
ORA_WORKINGSETMIN = 2
*...........From init.ora.............*
*.__dg_broker_service_names=';'
arcsight.__java_pool_size=0
arcsight.__large_pool_size=0
arcsight.__shared_pool_size=314572800
arcsight.__streams_pool_size=0
*.audit_file_dest='E:\oracle10g\OraHome10g\admin\arcsight\adump'
*.audit_sys_operations=true
*.audit_trail='db'
*.background_dump_dest='E:\oracle10g\OraHome10g\admin\arcsight\bdump'
*.compatible='10.2.0.1.0'
*.control_files='E:\oracle10g\OraHome10g\oradata\arcsight\control01.ctl','f:\arcsight\control02.ctl','g:\arcsight\control03.ctl'
*.core_dump_dest='E:\oracle10g\OraHome10g\admin\arcsight\cdump'
*.cursor_sharing='FORCE'
**.db_block_size=16384*
**.db_block_buffers=235929*
*.db_domain=''
*.db_file_multiblock_read_count=16
*.db_files=2000
*.db_name='arcsight'
*.db_writer_processes=4
*.dispatchers=''
*.job_queue_processes=10
*.log_archive_dest_1='LOCATION=H:'
*.log_buffer=1048576
*.open_cursors=2000
*.parallel_max_servers=0
*.pga_aggregate_target=314572800
*.processes=300
*.recyclebin='OFF'
*.remote_login_passwordfile='EXCLUSIVE'
*.sga_target=0
*.undo_management='AUTO'
*.undo_retention=43200
*.undo_tablespace='ARC_UNDO'
*.user_dump_dest='E:\oracle10g\OraHome10g\admin\arcsight\udump'
*.java_pool_size=0
*.large_pool_size=0
*.shared_pool_size=314572800
*.streams_pool_size=0
**.use_indirect_data_buffers=true*
*......From oradim.log.......*
Sun Feb 22 18:37:33 2009
E:\oracle10g\OraHome10g\bin\oradim.exe -shutdown -sid arcsight -usrpwd * -shutmode immediate -log oradim.log
Sun Feb 22 18:37:34 2009
ORA-01012: not logged on
Sun Feb 22 18:37:45 2009
E:\oracle10g\OraHome10g\bin\oradim.exe -startup -sid arcsight -usrpwd * -log oradim.log -nocheck 0
Sun Feb 22 18:37:51 2009
ORA-03113: end-of-file on communication channel
*.......From alert_arcsight.log.........*
Dump file e:\oracle10g\orahome10g\admin\arcsight\bdump\alert_arcsight.log
Sun Feb 22 23:20:51 2009
ORACLE V10.2.0.2.0 - Production vsnsta=0
vsnsql=14 vsnxtr=3
Windows Server 2003 Version V5.2 Service Pack 2
CPU : 8 - type 586, 4 Physical Cores
Process Affinity : 0x00000000
Memory (Avail/Total): Ph:14554M/16215M, Ph+PgF:14862M/15967M, VA:1926M/2047M
Sun Feb 22 23:20:51 2009
Starting ORACLE instance (normal)
Sun Feb 22 23:20:52 2009
Window memory size 1288503296
Sun Feb 22 23:20:52 2009
Minimum working set window size : 4096
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Picked latch-free SCN scheme 2
Autotune of undo retention is turned on.
IMODE=BR
ILAT =36
LICENSE_MAX_USERS = 0
SYS auditing is enabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.2.0.
System parameters with non-default values:
processes = 300
use_indirect_data_buffers= TRUE
__shared_pool_size = 318767104
shared_pool_size = 318767104
__large_pool_size = 0
large_pool_size = 0
__java_pool_size = 0
java_pool_size = 0
__streams_pool_size = 0
streams_pool_size = 0
sga_target = 0
control_files = E:\ORACLE10G\ORAHOME10G\ORADATA\ARCSIGHT\CONTROL01.CTL, F:\ARCSIGHT\CONTROL02.CTL, G:\ARCSIGHT\CONTROL03.CTL
db_block_buffers = 235932
db_block_size = 16384
db_writer_processes = 4
compatible = 10.2.0.1.0
log_archive_dest_1 = LOCATION=H:
log_buffer = 2097152
db_files = 2000
db_file_multiblock_read_count= 16
undo_management = AUTO
undo_tablespace = ARC_UNDO
undo_retention = 43200
recyclebin = OFF
remote_login_passwordfile= EXCLUSIVE
audit_sys_operations = TRUE
db_domain =
__dg_broker_service_names= ;
dispatchers =
job_queue_processes = 10
cursor_sharing = FORCE
parallel_max_servers = 0
audit_file_dest = E:\ORACLE10G\ORAHOME10G\ADMIN\ARCSIGHT\ADUMP
background_dump_dest = E:\ORACLE10G\ORAHOME10G\ADMIN\ARCSIGHT\BDUMP
user_dump_dest = E:\ORACLE10G\ORAHOME10G\ADMIN\ARCSIGHT\UDUMP
core_dump_dest = E:\ORACLE10G\ORAHOME10G\ADMIN\ARCSIGHT\CDUMP
audit_trail = DB
db_name = arcsight
open_cursors = 2000
pga_aggregate_target = 314572800
PMON started with pid=2, OS id=6676
PSP0 started with pid=6, OS id=7544
MMAN started with pid=10, OS id=7560
DBW0 started with pid=14, OS id=6500
DBW1 started with pid=18, OS id=6800
DBW2 started with pid=22, OS id=6276
DBW3 started with pid=26, OS id=520
LGWR started with pid=30, OS id=6756
CKPT started with pid=34, OS id=6380
SMON started with pid=38, OS id=7472
RECO started with pid=42, OS id=7696
CJQ0 started with pid=46, OS id=7912
MMON started with pid=50, OS id=7576
MMNL started with pid=54, OS id=6852
Sun Feb 22 23:20:53 2009
alter database mount exclusive
Sun Feb 22 23:20:57 2009
Setting recovery target incarnation to 1
Sun Feb 22 23:20:57 2009
Successful mount of redo thread 1, with mount id 1799551061
Sun Feb 22 23:20:57 2009
Database mounted in Exclusive Mode
Completed: alter database mount exclusive
Sun Feb 22 23:20:57 2009
alter database open
Sun Feb 22 23:20:58 2009
Beginning crash recovery of 1 threads
parallel recovery setup failed: using serial mode
Sun Feb 22 23:20:58 2009
Started redo scan
Sun Feb 22 23:20:58 2009
Completed redo scan
0 redo blocks read, 0 data blocks need recovery
Sun Feb 22 23:20:58 2009
Started redo application at
Thread 1: logseq 1137, block 3, scn 1707289029
Sun Feb 22 23:20:58 2009
Recovery of Online Redo Log: Thread 1 Group 5 Seq 1137 Reading mem 0
Mem# 0: I:\ARCSIGHT\REDO\REDO5.LOG
Mem# 1: I:\ARCSIGHT\REDO\REDO05B.LOG
Sun Feb 22 23:20:58 2009
Completed redo application
Sun Feb 22 23:20:58 2009
Completed crash recovery at
Thread 1: logseq 1137, block 3, scn 1707309030
0 data blocks read, 0 data blocks written, 0 redo blocks read
Sun Feb 22 23:20:59 2009
LGWR: STARTING ARCH PROCESSES
ARC0 started with pid=62, OS id=6972
Sun Feb 22 23:20:59 2009
ARC0: Archival started
ARC1 started with pid=66, OS id=6640
Sun Feb 22 23:20:59 2009
ARC1: Archival started
LGWR: STARTING ARCH PROCESSES COMPLETE
Thread 1 advanced to log sequence 1138
Thread 1 opened at log sequence 1138
Current log# 4 seq# 1138 mem# 0: G:\ARCSIGHT\REDO\REDO4.LOG
Current log# 4 seq# 1138 mem# 1: G:\ARCSIGHT\REDO\REDO04B.LOG
Successful open of redo thread 1
Sun Feb 22 23:21:00 2009
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Sun Feb 22 23:21:00 2009
ARC0: Becoming the 'no FAL' ARCH
ARC0: Becoming the 'no SRL' ARCH
Sun Feb 22 23:21:00 2009
ARC1: Becoming the heartbeat ARCH
Sun Feb 22 23:21:00 2009
SMON: enabling cache recovery
Sun Feb 22 23:21:02 2009
Errors in file e:\oracle10g\orahome10g\admin\arcsight\bdump\arcsight_pmon_6676.trc:
ORA-27103: internal error
OSD-00028: additional error information
Sun Feb 22 23:21:02 2009
PMON: terminating instance due to error 27103
Sun Feb 22 23:21:02 2009
Errors in file e:\oracle10g\orahome10g\admin\arcsight\bdump\arcsight_reco_7696.trc:
ORA-27103: internal error
Sun Feb 22 23:21:02 2009
Errors in file e:\oracle10g\orahome10g\admin\arcsight\bdump\arcsight_smon_7472.trc:
ORA-27103: internal error
Sun Feb 22 23:21:02 2009
Errors in file e:\oracle10g\orahome10g\admin\arcsight\bdump\arcsight_ckpt_6380.trc:
ORA-27103: internal error
Sun Feb 22 23:21:02 2009
Errors in file e:\oracle10g\orahome10g\admin\arcsight\bdump\arcsight_lgwr_6756.trc:
ORA-27103: internal error
Sun Feb 22 23:21:03 2009
Errors in file e:\oracle10g\orahome10g\admin\arcsight\bdump\arcsight_dbw3_520.trc:
ORA-27103: internal error
Sun Feb 22 23:21:03 2009
Errors in file e:\oracle10g\orahome10g\admin\arcsight\bdump\arcsight_dbw2_6276.trc:
ORA-27103: internal error
Sun Feb 22 23:21:03 2009
Errors in file e:\oracle10g\orahome10g\admin\arcsight\bdump\arcsight_dbw1_6800.trc:
ORA-27103: internal error
Sun Feb 22 23:21:03 2009
Errors in file e:\oracle10g\orahome10g\admin\arcsight\bdump\arcsight_dbw0_6500.trc:
ORA-27103: internal error
Sun Feb 22 23:21:03 2009
Errors in file e:\oracle10g\orahome10g\admin\arcsight\bdump\arcsight_mman_7560.trc:
ORA-27103: internal error
Sun Feb 22 23:21:04 2009
Errors in file e:\oracle10g\orahome10g\admin\arcsight\bdump\arcsight_psp0_7544.trc:
ORA-27103: internal error
Instance terminated by PMON, pid = 6676
I appreciate any input on what to look at to further isolate this issue. I'd run into many other issues along the way (setting AWE_WINDOW_MEMORY to a proper size, setting db_block_buffers to a proper value, etc) that various forum searches helped resolve but I've not been able to find anything related to the errors I'm getting now. If I set use_indirect_data_buffers=false and tune back the db_block_buffers, the instance starts without any problems. Its just when I try and enable the use of AWE that I'm having a problem.
Nick

Answer 1

Just wanted to close out this tread in case anyone else runs into a similar problem. Turns out we ran into a bug documented in the below linked article (we're using AMD processors). Essentially needed to disable NUMA.
http://blog.csdn.net/orapeasant/archive/2007/06/05/1639532.aspx
excerpt ....
But please be aware of Bug 4494543 - affecting 10g and fixed in Oracle 11.0 ......
ORA-7445: CORE DUMP [ACCESS_VIOLATION] WITH USE_INDIRECT_DATA_BUFFERS=TRUE
Rediscovery Information:
1) Using 32-Bit Oracle on a 32-Bit Windows 2003 server running on an AMD Opteron 64-Bit chip.
2) You have set use_indirect_data_buffers=true in init.ora
Workaround: Basically disable NUMA feature on 32-Bit platform :-
1) Set ENABLENUMA = FALSE in Windows registry for the Oracle Home.
2) Set enableNUMA_optimizations = FALSE (init.ora)
Thanks for the help. We'll see if access to the extra memory will be useful or not .....
Nick