Large Core Dumps after DB Backup Control File

Question

Hi All,
Server Specs (it is the test server having no functional modules):
Windows 2003 64bit
Oracle 10.2.0.4 (Patched from 10.2.0.2)
Netweaver 2004s
SQL> RECOVER DATABASE USING BACKUP CONTROLFILE UNTIL CANCEL
ORA-00279: change 21375038 generated at 09/21/2007 20:12:47 needed for thread 1
ORA-00289: suggestion : /export/pspp/oracle/dbs/xxx_example_xxx.dbf
ORA-00280: change 21375038 for thread 1 is in sequence #4
Specify log: {=suggested | filename | AUTO | CANCEL}
/OraRedo/RedoLogFiles/xxx_example_xxx.dbf
Log applied.
Media recovery complete.
I was facing Control file inconsistence issues against 2 partitions. Issue was resolved by using above commands and Netweaver started successfully. Im now facing another problem, im getting huge tracefiles in SAPTRACE\BACKGROUND folder since its recovery. I deleted those tracefiles which were of approx. 86GB.
Current Tracefiles sizes are:
<drive>:\oracle\TST\saptrace\background\tst_m000_5188.trc (7MB) and it is getting double by each generation.
Here are tracefiles of today (old ones deleted by me coz it was consuming a hell of space, e.g: 86GB as mentioned):
tst_m000_5188.trc (7 MB) Latest
tst_j000_5316.trc (4MB)
Here is Initial info from alert_tst.txt file contained in same folder:
Dump file f:\oracle\tst\saptrace\background\alert_tst.log
Thu Sep 02 13:55:34 2010
ORACLE V10.2.0.4.0 - 64bit Production vsnsta=0
vsnsql=14 vsnxtr=3
Windows NT Version V5.2 Service Pack 1
CPU                 : 2 - type 8664, 1 Physical Cores
Process Affinity    : 0x0000000000000000
Memory (Avail/Total): Ph:2886M/4019M, Ph+PgF:19295M/20131M
Thu Sep 02 13:55:34 2010
Starting ORACLE instance (normal)
Thu Sep 02 13:55:35 2010
Specified value of sga_max_size is too small, bumping to 666894336
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Shared memory segment for instance monitoring created
Picked latch-free SCN scheme 3
Autotune of undo retention is turned on.
IMODE=BR
ILAT =10
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.4.0.
System parameters with non-default values:
  processes                = 80
  sessions                 = 96
  event                    = 10191 trace name context forever, level 1
  sga_max_size             = 666894336
  shared_pool_size         = 318767104
  shared_pool_reserved_size= 31598837
  filesystemio_options     = setall
  control_files            = E:\ORACLE\TST\ORIGLOGA\CNTRL\CNTLRTST.DBF, E:\ORACLE\TST\ORIGLOGB\CNTRL\CNTRLTST.DBF, F:\ORACLE\TST\SAPDATA1\CNTRL\CNTRLTST.DBF
  control_file_record_keep_time= 30
  db_block_size            = 8192
  db_cache_size            = 318767104
  compatible               = 10.2.0
  log_archive_dest         = E:\oracle\TST\oraarch\TSTarch
  log_buffer               = 2061312
  log_checkpoint_interval  = 0
  db_files                 = 254
  log_checkpoints_to_alert = TRUE
  dml_locks                = 4000
  undo_management          = AUTO
  undo_tablespace          = PSAPUNDO
  undo_retention           = 43200
  recyclebin               = off
  remote_os_authent        = TRUE
  remote_login_passwordfile= EXCLUSIVE
  job_queue_processes      = 1
  background_dump_dest     = F:\ORACLE\TST\SAPTRACE\BACKGROUND
  user_dump_dest           = F:\ORACLE\TST\SAPTRACE\USERTRACE
  core_dump_dest           = F:\ORACLE\TST\SAPTRACE\BACKGROUND
  optimizer_features_enable= 10.2.0.1
  sort_area_size           = 2097152
  sort_area_retained_size  = 0
  db_name                  = TST
  open_cursors             = 800
  _optim_peek_user_binds   = FALSE
  pga_aggregate_target     = 421317836
  workarea_size_policy     = AUTO
  statistics_level         = typical
MMAN started with pid=4, OS id=2760
PMON started with pid=2, OS id=2748
DBW0 started with pid=5, OS id=712
LGWR started with pid=6, OS id=604
CKPT started with pid=7, OS id=1088
SMON started with pid=8, OS id=1164
RECO started with pid=9, OS id=2680
CJQ0 started with pid=10, OS id=2972
MMON started with pid=11, OS id=1816
MMNL started with pid=12, OS id=2904
PSP0 started with pid=3, OS id=2692
Please Review,
Thanks in Advance.
Best Regards.

Answer 1

All dispatchers and shared servers shutdown
Mon Jan 10 15:19:45 2011
ALTER DATABASE CLOSE NORMAL
Mon Jan 10 15:19:45 2011
SMON: disabling tx recovery
SMON: disabling cache recovery
Mon Jan 10 15:19:46 2011
Shutting down archive processes
Archiving is disabled
Mon Jan 10 15:19:51 2011
ARCH shutting down
ARC1: Archival stopped
Mon Jan 10 15:19:56 2011
ARCH shutting down
ARC0: Archival stopped
Mon Jan 10 15:19:57 2011
Thread 1 closed at log sequence 11
Successful close of redo thread 1
Mon Jan 10 15:19:59 2011
Completed: ALTER DATABASE CLOSE NORMAL
Mon Jan 10 15:19:59 2011
ALTER DATABASE DISMOUNT
Completed: ALTER DATABASE DISMOUNT
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
Shutting down Data Guard Broker processes
Mon Jan 10 15:20:00 2011
Completed: Data Guard Broker shutdown
Mon Jan 10 15:20:02 2011
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
Mon Jan 10 15:21:43 2011
Starting ORACLE instance (normal)
Mon Jan 10 15:21:43 2011
Specified value of sga_max_size is too small, bumping to 645922816
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Picked latch-free SCN scheme 3
Autotune of undo retention is turned on.
IMODE=BR
ILAT =10
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.4.0.
System parameters with non-default values:
  processes                = 80
  sessions                 = 96
  event                    = 10191 trace name context forever, level 1
  sga_max_size             = 645922816
  shared_pool_size         = 318767104
  __java_pool_size         = 4194304
  java_pool_size           = 4194304
  shared_pool_reserved_size= 31598837
  filesystemio_options     = setall
  control_files            = E:\ORACLE\TST\ORIGLOGA\CNTRL\CNTLRTST.DBF, E:\ORACLE\TST\ORIGLOGB\CNTRL\CNTRLTST.DBF, F:\ORACLE\TST\SAPDATA1\CNTRL\CNTRLTST.DBF
  control_file_record_keep_time= 30
  db_block_size            = 8192
  db_cache_size            = 318767104
  compatible               = 10.2.0
  log_archive_dest         = E:\oracle\TST\oraarch\TSTarch
  log_buffer               = 2061312
  log_checkpoint_interval  = 0
  db_files                 = 254
  log_checkpoints_to_alert = TRUE
  dml_locks                = 4000
  undo_management          = AUTO
  undo_tablespace          = PSAPUNDO
  undo_retention           = 43200
  recyclebin               = off
  remote_os_authent        = TRUE
  remote_login_passwordfile= EXCLUSIVE
  dispatchers              = (PROTOCOL=TCP) (SERVICE=TSTXDB)
  job_queue_processes      = 1
  background_dump_dest     = F:\ORACLE\TST\SAPTRACE\BACKGROUND
  user_dump_dest           = F:\ORACLE\TST\SAPTRACE\USERTRACE
  core_dump_dest           = F:\ORACLE\TST\SAPTRACE\BACKGROUND
  optimizer_features_enable= 10.2.0.1
  sort_area_size           = 2097152
  sort_area_retained_size  = 0
  db_name                  = TST
  open_cursors             = 800
  optimpeek_user_binds   = FALSE
  pga_aggregate_target     = 421317836
  workarea_size_policy     = AUTO
  statistics_level         = typical
  dg_broker_start          = TRUE
PMON started with pid=2, OS id=5108
PSP0 started with pid=3, OS id=3884
MMAN started with pid=4, OS id=3212
DBW0 started with pid=5, OS id=4508
LGWR started with pid=6, OS id=4124
CKPT started with pid=7, OS id=3684
SMON started with pid=8, OS id=3284
RECO started with pid=9, OS id=4360
CJQ0 started with pid=10, OS id=2128
MMON started with pid=11, OS id=3976
Mon Jan 10 15:21:43 2011
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
MMNL started with pid=12, OS id=3444
Mon Jan 10 15:21:43 2011
starting up 1 shared server(s) ...
DMON started with pid=15, OS id=4136
Mon Jan 10 15:21:43 2011
ALTER DATABASE   MOUNT
Mon Jan 10 15:21:48 2011
Setting recovery target incarnation to 4
Mon Jan 10 15:21:48 2011
Successful mount of redo thread 1, with mount id 1881221383
Mon Jan 10 15:21:48 2011
Database mounted in Exclusive Mode
Completed: ALTER DATABASE   MOUNT
Mon Jan 10 15:21:48 2011
ALTER DATABASE OPEN
Mon Jan 10 15:21:51 2011
Starting Data Guard Broker (DMON)
INSV started with pid=16, OS id=4264
Mon Jan 10 15:21:55 2011
LGWR: STARTING ARCH PROCESSES
ARC0 started with pid=18, OS id=4768
Mon Jan 10 15:21:55 2011
ARC0: Archival started
Mon Jan 10 15:21:55 2011
ARC1: Archival started
ARC1 started with pid=19, OS id=5288
LGWR: STARTING ARCH PROCESSES COMPLETE
Thread 1 opened at log sequence 11
  Current log# 3 seq# 11 mem# 0: E:\ORACLE\TST\ORIGLOGA\LOG_G13M1.DBF
  Current log# 3 seq# 11 mem# 1: E:\ORACLE\TST\MIRRLOGA\LOG_G13M2.DBF
Successful open of redo thread 1
Mon Jan 10 15:21:57 2011
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Mon Jan 10 15:21:57 2011
SMON: enabling cache recovery
Mon Jan 10 15:21:57 2011
ARC0: Becoming the 'no FAL' ARCH
ARC0: Becoming the 'no SRL' ARCH
Mon Jan 10 15:21:57 2011
ARC1: Becoming the heartbeat ARCH
Mon Jan 10 15:21:58 2011
Incremental checkpoint up to RBA [0xb.16b77.0], current log tail at RBA [0xb.16b77.0]
Mon Jan 10 15:21:59 2011
Successfully onlined Undo Tablespace 1.
Mon Jan 10 15:21:59 2011
SMON: enabling tx recovery
Mon Jan 10 15:21:59 2011
Database Characterset is UTF8
Opening with internal Resource Manager plan
where NUMA PG = 1, CPUs = 2
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
QMNC started with pid=20, OS id=4156
Mon Jan 10 15:22:05 2011
Completed: ALTER DATABASE OPEN
Mon Jan 10 15:52:02 2011
Incremental checkpoint up to RBA [0xb.180ee.0], current log tail at RBA [0xb.1830a.0]
Mon Jan 10 15:52:16 2011
Shutting down instance: further logons disabled
Mon Jan 10 15:52:16 2011
Stopping background process CJQ0
Mon Jan 10 15:52:16 2011
Stopping background process QMNC
Mon Jan 10 15:52:17 2011
Background process QMNC not dead after 150 seconds
Mon Jan 10 15:52:17 2011
Stopping background process MMNL
Mon Jan 10 15:52:18 2011
Background process MMNL not dead after 10 seconds
Mon Jan 10 15:52:18 2011
Stopping background process MMON
Mon Jan 10 15:52:19 2011
Background process MMON not dead after 30 seconds
Mon Jan 10 15:52:19 2011
Shutting down instance (immediate)
License high water mark = 12
Mon Jan 10 15:52:19 2011
Stopping Job queue slave processes, flags = 7
Mon Jan 10 15:52:19 2011
Job queue slave processes stopped
Waiting for dispatcher 'D000' to shutdown
All dispatchers and shared servers shutdown
Mon Jan 10 15:52:24 2011
ALTER DATABASE CLOSE NORMAL
Mon Jan 10 15:52:24 2011
SMON: disabling tx recovery
SMON: disabling cache recovery
Mon Jan 10 15:52:25 2011
Shutting down archive processes
Archiving is disabled
Mon Jan 10 15:52:30 2011
ARCH shutting down
ARC1: Archival stopped
Mon Jan 10 15:52:35 2011
ARCH shutting down
ARC0: Archival stopped
Mon Jan 10 15:52:36 2011
Thread 1 closed at log sequence 11
Successful close of redo thread 1
Mon Jan 10 15:52:37 2011
Completed: ALTER DATABASE CLOSE NORMAL
Mon Jan 10 15:52:37 2011
ALTER DATABASE DISMOUNT
Completed: ALTER DATABASE DISMOUNT
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
Shutting down Data Guard Broker processes
Mon Jan 10 15:52:38 2011
Completed: Data Guard Broker shutdown
Mon Jan 10 15:52:40 2011
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
Mon Jan 10 15:53:28 2011
Starting ORACLE instance (normal)
Mon Jan 10 15:53:28 2011
Specified value of sga_max_size is too small, bumping to 645922816
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Picked latch-free SCN scheme 3
Autotune of undo retention is turned on.
IMODE=BR
ILAT =10
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.4.0.
System parameters with non-default values:
  processes                = 80
  sessions                 = 96
  event                    = 10191 trace name context forever, level 1
  sga_max_size             = 645922816
  __shared_pool_size       = 318767104
  shared_pool_size         = 318767104
  __java_pool_size         = 4194304
  java_pool_size           = 4194304
  shared_pool_reserved_size= 31598837
  filesystemio_options     = setall
  control_files            = E:\ORACLE\TST\ORIGLOGA\CNTRL\CNTLRTST.DBF, E:\ORACLE\TST\ORIGLOGB\CNTRL\CNTRLTST.DBF, F:\ORACLE\TST\SAPDATA1\CNTRL\CNTRLTST.DBF
  control_file_record_keep_time= 30
  db_block_size            = 8192
  __db_cache_size          = 318767104
  db_cache_size            = 318767104
  compatible               = 10.2.0
  log_archive_dest         = E:\oracle\TST\oraarch\TSTarch
  log_buffer               = 2061312
  log_checkpoint_interval  = 0
  db_files                 = 254
  log_checkpoints_to_alert = TRUE
  dml_locks                = 4000
  undo_management          = AUTO
  undo_tablespace          = PSAPUNDO
  undo_retention           = 43200
  recyclebin               = off
  remote_os_authent        = TRUE
  remote_login_passwordfile= EXCLUSIVE
  dispatchers              = (PROTOCOL=TCP) (SERVICE=TSTXDB)
  job_queue_processes      = 1
  background_dump_dest     = F:\ORACLE\TST\SAPTRACE\BACKGROUND
  user_dump_dest           = F:\ORACLE\TST\SAPTRACE\USERTRACE
  core_dump_dest           = F:\ORACLE\TST\SAPTRACE\BACKGROUND
  optimizer_features_enable= 10.2.0.1
  sort_area_size           = 2097152
  sort_area_retained_size  = 0
  db_name                  = TST
  open_cursors             = 800
  optimpeek_user_binds   = FALSE
  pga_aggregate_target     = 421317836
  workarea_size_policy     = AUTO
  statistics_level         = typical
  dg_broker_start          = TRUE
PMON started with pid=2, OS id=5296
PSP0 started with pid=3, OS id=4804
MMAN started with pid=4, OS id=3480
DBW0 started with pid=5, OS id=4312
LGWR started with pid=6, OS id=4036
CKPT started with pid=7, OS id=3372
SMON started with pid=8, OS id=5816
RECO started with pid=9, OS id=5240
CJQ0 started with pid=10, OS id=4524
MMON started with pid=11, OS id=6140
Mon Jan 10 15:53:28 2011
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
MMNL started with pid=12, OS id=5680
Mon Jan 10 15:53:28 2011
starting up 1 shared server(s) ...
DMON started with pid=15, OS id=5308
Mon Jan 10 15:53:28 2011
ALTER DATABASE   MOUNT
Mon Jan 10 15:53:32 2011
Setting recovery target incarnation to 4
Mon Jan 10 15:53:33 2011
Successful mount of redo thread 1, with mount id 1881229688
Mon Jan 10 15:53:33 2011
Database mounted in Exclusive Mode
Completed: ALTER DATABASE   MOUNT
Mon Jan 10 15:53:33 2011
ALTER DATABASE OPEN
Mon Jan 10 15:53:36 2011
Starting Data Guard Broker (DMON)
INSV started with pid=16, OS id=4536
Mon Jan 10 15:53:40 2011
LGWR: STARTING ARCH PROCESSES
ARC0 started with pid=18, OS id=5552
Mon Jan 10 15:53:40 2011
ARC0: Archival started
ARC1: Archival started
ARC1 started with pid=19, OS id=5312
LGWR: STARTING ARCH PROCESSES COMPLETE
Thread 1 opened at log sequence 11
  Current log# 3 seq# 11 mem# 0: E:\ORACLE\TST\ORIGLOGA\LOG_G13M1.DBF
  Current log# 3 seq# 11 mem# 1: E:\ORACLE\TST\MIRRLOGA\LOG_G13M2.DBF
Successful open of redo thread 1
Mon Jan 10 15:53:41 2011
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Mon Jan 10 15:53:41 2011
SMON: enabling cache recovery
Mon Jan 10 15:53:42 2011
ARC1: Becoming the 'no FAL' ARCH
ARC1: Becoming the 'no SRL' ARCH
Mon Jan 10 15:53:42 2011
ARC0: Becoming the heartbeat ARCH
Mon Jan 10 15:53:43 2011
Incremental checkpoint up to RBA [0xb.18403.0], current log tail at RBA [0xb.18403.0]
Mon Jan 10 15:53:44 2011
Successfully onlined Undo Tablespace 1.
Mon Jan 10 15:53:44 2011
SMON: enabling tx recovery
Mon Jan 10 15:53:44 2011
Database Characterset is UTF8
Opening with internal Resource Manager plan
where NUMA PG = 1, CPUs = 2
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
QMNC started with pid=20, OS id=5776
Mon Jan 10 15:53:50 2011
Completed: ALTER DATABASE OPEN
Mon Jan 10 15:55:46 2011
Starting ORACLE instance (normal)
Mon Jan 10 16:00:56 2011
Errors in file f:\oracle\tst\saptrace\background\tst_j000_3228.trc:
Mon Jan 10 16:01:26 2011
Beginning log switch checkpoint up to RBA [0xc.2.10], SCN: 16865686
Mon Jan 10 16:01:26 2011
Thread 1 advanced to log sequence 12 (LGWR switch)
  Current log# 4 seq# 12 mem# 0: E:\ORACLE\TST\ORIGLOGB\LOG_G14M1.DBF
  Current log# 4 seq# 12 mem# 1: E:\ORACLE\TST\MIRRLOGB\LOG_G14M2.DBF
Mon Jan 10 16:03:54 2011
Errors in file f:\oracle\tst\saptrace\background\tst_m000_5396.trc:
ORA-00600: internal error code, arguments: [kcbz_check_objd_typ], [0], [0], [1], [], [], [], []
Mon Jan 10 16:04:01 2011
Errors in file f:\oracle\tst\saptrace\background\tst_m000_5396.trc:
ORA-00600: internal error code, arguments: [kcbgcur_3], [61846], [9], [4], [0], [], [], []
Mon Jan 10 16:04:07 2011
Errors in file f:\oracle\tst\saptrace\background\tst_m000_5396.trc:
ORA-00600: internal error code, arguments: [kcbz_check_objd_typ], [0], [0], [1], [], [], [], []
Mon Jan 10 16:06:30 2011
Completed checkpoint up to RBA [0xc.2.10], SCN: 16865686
Mon Jan 10 16:23:50 2011
Incremental checkpoint up to RBA [0xc.4e3a.0], current log tail at RBA [0xc.4f1f.0]
Mon Jan 10 16:53:57 2011
Incremental checkpoint up to RBA [0xc.55b9.0], current log tail at RBA [0xc.5726.0]
Mon Jan 10 17:00:01 2011
Errors in file f:\oracle\tst\saptrace\background\tst_j000_2516.trc:
Mon Jan 10 17:00:29 2011
Errors in file f:\oracle\tst\saptrace\background\tst_m000_6048.trc:
ORA-00600: internal error code, arguments: [kcbz_check_objd_typ], [0], [0], [1], [], [], [], []
Mon Jan 10 17:00:33 2011
Errors in file f:\oracle\tst\saptrace\background\tst_m000_6048.trc:
ORA-00600: internal error code, arguments: [kcbgcur_3], [61846], [9], [4], [0], [], [], []
Mon Jan 10 17:00:37 2011
Errors in file f:\oracle\tst\saptrace\background\tst_m000_6048.trc:
ORA-00600: internal error code, arguments: [kcbz_check_objd_typ], [0], [0], [1], [], [], [], []
Mon Jan 10 17:24:02 2011
Incremental checkpoint up to RBA [0xc.6125.0], current log tail at RBA [0xc.62ae.0]