Oracle instance crashed

Oracle-Version: 9i R2 9.2.0.1.0.
Hello !
My oracle instance crashed and I found the following message in the log file :
LGWR: terminating instance due to error 472
Instance terminated by LGWR, pid = 1304
I have not found any other messages and the ORA-00472 message says that the PMON process terminated with error.
Did anybody have the same error ?
I need to know why the instance crashed.
I hope you can help me. Thanks in advance.
Andreas

The 472 is just the error issued when the Oracle background processes are signaled to terminate. It would appear log writer took an error and either notified the other background processes to terminate or it terminated and another backgroun process detected the problem, probably smon, and killed the rest of the system processes.
Look in the bdump directory for the log writer trace file and see what it says. There may also be some entries in the alert log depending on exactly what happened.
If the database fired back up without problem and the problem does not reoccur frequently then if might just be one of those things. However, if you can find the trace file and error messages for the problem cause then you should probably open an iTAR with Oracle support to find out he fix/work around.
HTH -- Mark D Powell --

Similar Messages

  • Oracle instance crashed with error 472

    My Oracle instance crashed after running for 2 days
    with 5 reads and 2 updates in every 4 seconds. The
    alert log are
    Fri Oct 24 02:59:13 2003
    found dead multi-threaded server 's000',pid=(10,1)
    Fri Oct 24 03:00:31 2003
    LGWR: terminating instance due to error 472
    Instance terminated by LGWR, pid = 109
    The client access the data through stored procedures.
    The database is running on Windows NT 4.0 service pack 6a.
    The Oracle is v8.1.5.0.0
    Any help will be greatly appreciated.

    ORA-00472 PMON process terminated with error
    Cause: The Process Monitor process terminated abnormally.
    Action: Check the accompanying messages, and the background process trace file. Correct the problem mentioned in the messages. Then shut down and restart the instance. If the trace file mentions any other background process messages, check the trace file for the mentioned process until the root message is found.
    Follow the recomendations above
    Joel P�rez

  • Oracle instance crashing when enabling use_indirect_data_buffers=true

    I have a Windows 2003 EE server (32bit) with 16GB of ram hosting a 10.2.0.2 Oracle server which is used to support a commercial software package (arcsight). I'm trying to get the Oracle backend to leverage the available system memory. I've read 50-60 different articles and posts regarding AWE and Oracle. I have successfully tuned the userva parameter in order to get the server to boot stable with the /3gb boot parameter. I've gotten to the point that the oracle instance will start up, but within about 30-60 seconds the instance will crash. Below is the information I believe that is relevant:
    *.......From computer Registry.........*
    AWE_MEMORY_WINDOW = 1288486912
    ORA_WORKINGSETMIN = 2
    *...........From init.ora.............*
    *.__dg_broker_service_names=';'
    arcsight.__java_pool_size=0
    arcsight.__large_pool_size=0
    arcsight.__shared_pool_size=314572800
    arcsight.__streams_pool_size=0
    *.audit_file_dest='E:\oracle10g\OraHome10g\admin\arcsight\adump'
    *.audit_sys_operations=true
    *.audit_trail='db'
    *.background_dump_dest='E:\oracle10g\OraHome10g\admin\arcsight\bdump'
    *.compatible='10.2.0.1.0'
    *.control_files='E:\oracle10g\OraHome10g\oradata\arcsight\control01.ctl','f:\arcsight\control02.ctl','g:\arcsight\control03.ctl'
    *.core_dump_dest='E:\oracle10g\OraHome10g\admin\arcsight\cdump'
    *.cursor_sharing='FORCE'
    **.db_block_size=16384*
    **.db_block_buffers=235929*
    *.db_domain=''
    *.db_file_multiblock_read_count=16
    *.db_files=2000
    *.db_name='arcsight'
    *.db_writer_processes=4
    *.dispatchers=''
    *.job_queue_processes=10
    *.log_archive_dest_1='LOCATION=H:'
    *.log_buffer=1048576
    *.open_cursors=2000
    *.parallel_max_servers=0
    *.pga_aggregate_target=314572800
    *.processes=300
    *.recyclebin='OFF'
    *.remote_login_passwordfile='EXCLUSIVE'
    *.sga_target=0
    *.undo_management='AUTO'
    *.undo_retention=43200
    *.undo_tablespace='ARC_UNDO'
    *.user_dump_dest='E:\oracle10g\OraHome10g\admin\arcsight\udump'
    *.java_pool_size=0
    *.large_pool_size=0
    *.shared_pool_size=314572800
    *.streams_pool_size=0
    **.use_indirect_data_buffers=true*
    *......From oradim.log.......*
    Sun Feb 22 18:37:33 2009
    E:\oracle10g\OraHome10g\bin\oradim.exe -shutdown -sid arcsight -usrpwd * -shutmode immediate -log oradim.log
    Sun Feb 22 18:37:34 2009
    ORA-01012: not logged on
    Sun Feb 22 18:37:45 2009
    E:\oracle10g\OraHome10g\bin\oradim.exe -startup -sid arcsight -usrpwd * -log oradim.log -nocheck 0
    Sun Feb 22 18:37:51 2009
    ORA-03113: end-of-file on communication channel
    *.......From alert_arcsight.log.........*
    Dump file e:\oracle10g\orahome10g\admin\arcsight\bdump\alert_arcsight.log
    Sun Feb 22 23:20:51 2009
    ORACLE V10.2.0.2.0 - Production vsnsta=0
    vsnsql=14 vsnxtr=3
    Windows Server 2003 Version V5.2 Service Pack 2
    CPU : 8 - type 586, 4 Physical Cores
    Process Affinity : 0x00000000
    Memory (Avail/Total): Ph:14554M/16215M, Ph+PgF:14862M/15967M, VA:1926M/2047M
    Sun Feb 22 23:20:51 2009
    Starting ORACLE instance (normal)
    Sun Feb 22 23:20:52 2009
    Window memory size 1288503296
    Sun Feb 22 23:20:52 2009
    Minimum working set window size : 4096
    LICENSE_MAX_SESSION = 0
    LICENSE_SESSIONS_WARNING = 0
    Picked latch-free SCN scheme 2
    Autotune of undo retention is turned on.
    IMODE=BR
    ILAT =36
    LICENSE_MAX_USERS = 0
    SYS auditing is enabled
    ksdpec: called for event 13740 prior to event group initialization
    Starting up ORACLE RDBMS Version: 10.2.0.2.0.
    System parameters with non-default values:
    processes = 300
    use_indirect_data_buffers= TRUE
    __shared_pool_size = 318767104
    shared_pool_size = 318767104
    __large_pool_size = 0
    large_pool_size = 0
    __java_pool_size = 0
    java_pool_size = 0
    __streams_pool_size = 0
    streams_pool_size = 0
    sga_target = 0
    control_files = E:\ORACLE10G\ORAHOME10G\ORADATA\ARCSIGHT\CONTROL01.CTL, F:\ARCSIGHT\CONTROL02.CTL, G:\ARCSIGHT\CONTROL03.CTL
    db_block_buffers = 235932
    db_block_size = 16384
    db_writer_processes = 4
    compatible = 10.2.0.1.0
    log_archive_dest_1 = LOCATION=H:
    log_buffer = 2097152
    db_files = 2000
    db_file_multiblock_read_count= 16
    undo_management = AUTO
    undo_tablespace = ARC_UNDO
    undo_retention = 43200
    recyclebin = OFF
    remote_login_passwordfile= EXCLUSIVE
    audit_sys_operations = TRUE
    db_domain =
    __dg_broker_service_names= ;
    dispatchers =
    job_queue_processes = 10
    cursor_sharing = FORCE
    parallel_max_servers = 0
    audit_file_dest = E:\ORACLE10G\ORAHOME10G\ADMIN\ARCSIGHT\ADUMP
    background_dump_dest = E:\ORACLE10G\ORAHOME10G\ADMIN\ARCSIGHT\BDUMP
    user_dump_dest = E:\ORACLE10G\ORAHOME10G\ADMIN\ARCSIGHT\UDUMP
    core_dump_dest = E:\ORACLE10G\ORAHOME10G\ADMIN\ARCSIGHT\CDUMP
    audit_trail = DB
    db_name = arcsight
    open_cursors = 2000
    pga_aggregate_target = 314572800
    PMON started with pid=2, OS id=6676
    PSP0 started with pid=6, OS id=7544
    MMAN started with pid=10, OS id=7560
    DBW0 started with pid=14, OS id=6500
    DBW1 started with pid=18, OS id=6800
    DBW2 started with pid=22, OS id=6276
    DBW3 started with pid=26, OS id=520
    LGWR started with pid=30, OS id=6756
    CKPT started with pid=34, OS id=6380
    SMON started with pid=38, OS id=7472
    RECO started with pid=42, OS id=7696
    CJQ0 started with pid=46, OS id=7912
    MMON started with pid=50, OS id=7576
    MMNL started with pid=54, OS id=6852
    Sun Feb 22 23:20:53 2009
    alter database mount exclusive
    Sun Feb 22 23:20:57 2009
    Setting recovery target incarnation to 1
    Sun Feb 22 23:20:57 2009
    Successful mount of redo thread 1, with mount id 1799551061
    Sun Feb 22 23:20:57 2009
    Database mounted in Exclusive Mode
    Completed: alter database mount exclusive
    Sun Feb 22 23:20:57 2009
    alter database open
    Sun Feb 22 23:20:58 2009
    Beginning crash recovery of 1 threads
    parallel recovery setup failed: using serial mode
    Sun Feb 22 23:20:58 2009
    Started redo scan
    Sun Feb 22 23:20:58 2009
    Completed redo scan
    0 redo blocks read, 0 data blocks need recovery
    Sun Feb 22 23:20:58 2009
    Started redo application at
    Thread 1: logseq 1137, block 3, scn 1707289029
    Sun Feb 22 23:20:58 2009
    Recovery of Online Redo Log: Thread 1 Group 5 Seq 1137 Reading mem 0
    Mem# 0: I:\ARCSIGHT\REDO\REDO5.LOG
    Mem# 1: I:\ARCSIGHT\REDO\REDO05B.LOG
    Sun Feb 22 23:20:58 2009
    Completed redo application
    Sun Feb 22 23:20:58 2009
    Completed crash recovery at
    Thread 1: logseq 1137, block 3, scn 1707309030
    0 data blocks read, 0 data blocks written, 0 redo blocks read
    Sun Feb 22 23:20:59 2009
    LGWR: STARTING ARCH PROCESSES
    ARC0 started with pid=62, OS id=6972
    Sun Feb 22 23:20:59 2009
    ARC0: Archival started
    ARC1 started with pid=66, OS id=6640
    Sun Feb 22 23:20:59 2009
    ARC1: Archival started
    LGWR: STARTING ARCH PROCESSES COMPLETE
    Thread 1 advanced to log sequence 1138
    Thread 1 opened at log sequence 1138
    Current log# 4 seq# 1138 mem# 0: G:\ARCSIGHT\REDO\REDO4.LOG
    Current log# 4 seq# 1138 mem# 1: G:\ARCSIGHT\REDO\REDO04B.LOG
    Successful open of redo thread 1
    Sun Feb 22 23:21:00 2009
    MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
    Sun Feb 22 23:21:00 2009
    ARC0: Becoming the 'no FAL' ARCH
    ARC0: Becoming the 'no SRL' ARCH
    Sun Feb 22 23:21:00 2009
    ARC1: Becoming the heartbeat ARCH
    Sun Feb 22 23:21:00 2009
    SMON: enabling cache recovery
    Sun Feb 22 23:21:02 2009
    Errors in file e:\oracle10g\orahome10g\admin\arcsight\bdump\arcsight_pmon_6676.trc:
    ORA-27103: internal error
    OSD-00028: additional error information
    Sun Feb 22 23:21:02 2009
    PMON: terminating instance due to error 27103
    Sun Feb 22 23:21:02 2009
    Errors in file e:\oracle10g\orahome10g\admin\arcsight\bdump\arcsight_reco_7696.trc:
    ORA-27103: internal error
    Sun Feb 22 23:21:02 2009
    Errors in file e:\oracle10g\orahome10g\admin\arcsight\bdump\arcsight_smon_7472.trc:
    ORA-27103: internal error
    Sun Feb 22 23:21:02 2009
    Errors in file e:\oracle10g\orahome10g\admin\arcsight\bdump\arcsight_ckpt_6380.trc:
    ORA-27103: internal error
    Sun Feb 22 23:21:02 2009
    Errors in file e:\oracle10g\orahome10g\admin\arcsight\bdump\arcsight_lgwr_6756.trc:
    ORA-27103: internal error
    Sun Feb 22 23:21:03 2009
    Errors in file e:\oracle10g\orahome10g\admin\arcsight\bdump\arcsight_dbw3_520.trc:
    ORA-27103: internal error
    Sun Feb 22 23:21:03 2009
    Errors in file e:\oracle10g\orahome10g\admin\arcsight\bdump\arcsight_dbw2_6276.trc:
    ORA-27103: internal error
    Sun Feb 22 23:21:03 2009
    Errors in file e:\oracle10g\orahome10g\admin\arcsight\bdump\arcsight_dbw1_6800.trc:
    ORA-27103: internal error
    Sun Feb 22 23:21:03 2009
    Errors in file e:\oracle10g\orahome10g\admin\arcsight\bdump\arcsight_dbw0_6500.trc:
    ORA-27103: internal error
    Sun Feb 22 23:21:03 2009
    Errors in file e:\oracle10g\orahome10g\admin\arcsight\bdump\arcsight_mman_7560.trc:
    ORA-27103: internal error
    Sun Feb 22 23:21:04 2009
    Errors in file e:\oracle10g\orahome10g\admin\arcsight\bdump\arcsight_psp0_7544.trc:
    ORA-27103: internal error
    Instance terminated by PMON, pid = 6676
    I appreciate any input on what to look at to further isolate this issue. I'd run into many other issues along the way (setting AWE_WINDOW_MEMORY to a proper size, setting db_block_buffers to a proper value, etc) that various forum searches helped resolve but I've not been able to find anything related to the errors I'm getting now. If I set use_indirect_data_buffers=false and tune back the db_block_buffers, the instance starts without any problems. Its just when I try and enable the use of AWE that I'm having a problem.
    Nick

    Just wanted to close out this tread in case anyone else runs into a similar problem. Turns out we ran into a bug documented in the below linked article (we're using AMD processors). Essentially needed to disable NUMA.
    http://blog.csdn.net/orapeasant/archive/2007/06/05/1639532.aspx
    excerpt ....
    But please be aware of Bug 4494543 - affecting 10g and fixed in Oracle 11.0 ......
    ORA-7445: CORE DUMP [ACCESS_VIOLATION] WITH USE_INDIRECT_DATA_BUFFERS=TRUE
    Rediscovery Information:
    1) Using 32-Bit Oracle on a 32-Bit Windows 2003 server running on an AMD Opteron 64-Bit chip.
    2) You have set use_indirect_data_buffers=true in init.ora
    Workaround: Basically disable NUMA feature on 32-Bit platform :-
    1) Set ENABLENUMA = FALSE in Windows registry for the Oracle Home.
    2) Set enableNUMA_optimizations = FALSE (init.ora)
    Thanks for the help. We'll see if access to the extra memory will be useful or not .....
    Nick

  • ASM or Oracle instance failure recovery

    Hi,
    If ASM or Oracle instance crashes because of a disk failure, and the data is safe on other ASM disks, can you re-install Oracle and ASM, and mount the database using the data files on existing ASM disks.
    reg,
    Amitoj.

    See https://community.oracle.com/message/12273470#12273470
    1&2: ASM or not - Difference performance wise is not that big
    2- Reasons to separe log & data files are for caching purpuse (i.e. EMC Fast Cache) & recovery.

  • Why my instance crashed down

    HI,everybody,My instance crashed down last day.
    Is there anybody can tell me why ?
    oracle version：10.2.0.1.0 (Only Patch 4612267）
    OS version：redhat as 4.5
    Sorry, I am in china,so there is some chinese character in the alter file,but I think everybody can guess the meanings depending on the location.
    alter file：Thu Apr 8 17:02:38 2010
    Hex dump of (file 3, block 85680) in trace file /u01/app/oracle/admin/WENDING/bdump/wending_dbw0_22490.trc
    Corrupt block relative dba: 0x00c14eb0 (file 3, block 85680)
    Bad header found during preparing block for write
    Data in bad block:
    type: 6 format: 2 rdba: 0x00c14eb0
    last change scn: 0x0001.e55fdf1f seq: 0x1 flg: 0x02
    spare1: 0x0 spare2: 0x0 spare3: 0x2000
    consistency value in tail: 0xdf1f0601
    check value in block header: 0xcf0
    block checksum disabled
    Thu Apr 8 17:02:49 2010
    Errors in file /u01/app/oracle/admin/WENDING/bdump/wending_dbw0_22490.trc:
    ORA-00600: 内部错误代码, 参数: [kcbzpbuf_1], [4], [1], [], [], [], [], []
    Thu Apr 8 17:02:51 2010
    Errors in file /u01/app/oracle/admin/WENDING/bdump/wending_dbw0_22490.trc:
    ORA-00600: 内部错误代码, 参数: [kcbzpbuf_1], [4], [1], [], [], [], [], []
    Thu Apr 8 17:02:51 2010
    DBW0: terminating instance due to error 471
    Termination issued to instance processes. Waiting for the processes to exit
    Thu Apr 8 17:03:02 2010
    Instance termination failed to kill one or more processes
    Instance terminated by DBW0, pid = 22490
    trace file：
    BH (0x85fa400c) file#: 3 rdba: 0x00c14eb0 (3/85680) class: 1 ba: 0x85000000
    set: 3 blksize: 8192 bsi: 0 set-flg: 0 pwbcnt: 0
    dbwrid: 0 obj: 147678 objn: 147678 tsn: 2 afn: 3
    hash: [63ff817c,9d1f6d1c] lru-req: [9d465708,9d465708]
    lru-flags: on_auxiliary_list object_write_list
    ckptq: [26fdb504,30ff0114] fileq: [26fdb50c,6bfbf4cc] objq: [93e35c90,93e35c90]
    st: XCURRENT md: NULL tch: 2
    flags: buffer_dirty being_written gotten_in_current_mode being_evicted
    block_written_once redo_since_read
    LRBA: [0x516a.1a2bd.0] HSCN: [0x1.e55fdf1f] HSUB: [1]
    Dump of buffer cache at level 10 for tsn=2, rdba=12668592
    BH (0x85fa400c) file#: 3 rdba: 0x00c14eb0 (3/85680) class: 1 ba: 0x85000000
    set: 3 blksize: 8192 bsi: 0 set-flg: 0 pwbcnt: 0
    dbwrid: 0 obj: 147678 objn: 147678 tsn: 2 afn: 3
    hash: [63ff817c,9d1f6d1c] lru-req: [9d465708,9d465708]
    lru-flags: on_auxiliary_list object_write_list
    ckptq: [26fdb504,30ff0114] fileq: [26fdb50c,6bfbf4cc] objq: [93e35c90,93e35c90]
    st: XCURRENT md: NULL tch: 2
    flags: buffer_dirty being_written gotten_in_current_mode being_evicted
    block_written_once redo_since_read
    LRBA: [0x516a.1a2bd.0] HSCN: [0x1.e55fdf1f] HSUB: [1]
    buffer tsn: 2 rdba: 0x00c14eb0 (3/85680)
    scn: 0x0001.e55fdf1f seq: 0x01 flg: 0x02 tail: 0xdf1f0601
    frmt: 0x02 chkval: 0x0cf0 type: 0x06=trans data
    Hex dump of corrupt header 4 = CORRUPT
    Edited by: seven qi on 2010-4-9 上午1:30

    thank you for so quick response.
    I don't understand what happened when the error appeared.
    The trace file is so difficult to read, I don't know what it want to show.
    Would you like to tell me what happened with simple language?
    Edited by: seven qi on 2010-4-9 上午1:42

  • ORA-01092: ORACLE instance terminated. Disconnection forced

    Hi Friends,
    Here I'm getting this error when I start oracle database.
    ORA-01092: ORACLE instance terminated. Disconnection forced
    OS is windows XP pro DB 9iR2. The extract from alert log is as follows...
    Sun Feb 11 07:53:41 2007
    ORACLE V9.2.0.1.0 - Production vsnsta=0
    vsnsql=12 vsnxtr=3
    Windows 2000 Version 5.1 , CPU type 586
    Sun Feb 11 07:53:41 2007
    Starting ORACLE instance (normal)
    LICENSE_MAX_SESSION = 0
    LICENSE_SESSIONS_WARNING = 0
    SCN scheme 2
    Using log_archive_dest parameter default value
    LICENSE_MAX_USERS = 0
    SYS auditing is disabled
    Starting up ORACLE RDBMS Version: 9.2.0.1.0.
    System parameters with non-default values:
    processes = 150
    timed_statistics = TRUE
    shared_pool_size = 50331648
    large_pool_size = 8388608
    java_pool_size = 33554432
    control_files = E:\oracle\oradata\test\CONTROL01.CTL, E:\oracle\oradata\test\CONTROL02.CTL, E:\oracle\oradata\test\CONTROL03.CTL
    db_block_size = 4096
    db_cache_size = 33554432
    compatible = 9.2.0.0.0
    db_file_multiblock_read_count= 8
    fast_start_mttr_target = 300
    undo_management = AUTO
    undo_tablespace = UNDOTBS1
    undo_retention = 900
    remote_login_passwordfile= EXCLUSIVE
    db_domain =
    instance_name = test
    dispatchers = (PROTOCOL=TCP) (SERVICE=testXDB)
    job_queue_processes = 10
    hash_join_enabled = FALSE
    background_dump_dest = E:\oracle\admin\test\bdump
    user_dump_dest = E:\oracle\admin\test\udump
    core_dump_dest = E:\oracle\admin\test\cdump
    sort_area_size = 524288
    db_name = test
    open_cursors = 300
    star_transformation_enabled= FALSE
    query_rewrite_enabled = FALSE
    pga_aggregate_target = 16777216
    aq_tm_processes = 1
    PMON started with pid=2
    DBW0 started with pid=3
    LGWR started with pid=4
    CKPT started with pid=5
    SMON started with pid=6
    RECO started with pid=7
    CJQ0 started with pid=8
    QMN0 started with pid=9
    Sun Feb 11 07:53:47 2007
    starting up 1 shared server(s) ...
    starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
    Sun Feb 11 07:53:48 2007
    alter database mount exclusive
    Sun Feb 11 07:53:53 2007
    Successful mount of redo thread 1, with mount id 1912378892.
    Sun Feb 11 07:53:53 2007
    Database mounted in Exclusive Mode.
    Completed: alter database mount exclusive
    Sun Feb 11 07:53:53 2007
    alter database open
    Sun Feb 11 07:53:54 2007
    Beginning crash recovery of 1 threads
    Sun Feb 11 07:53:54 2007
    Started first pass scan
    Sun Feb 11 07:53:54 2007
    Completed first pass scan
    2018 redo blocks read, 177 data blocks need recovery
    Sun Feb 11 07:53:54 2007
    Started recovery at
    Thread 1: logseq 6, block 3, scn 0.0
    Recovery of Online Redo Log: Thread 1 Group 2 Seq 6 Reading mem 0
    Mem# 0 errs 0: E:\ORACLE\ORADATA\TEST\REDO02.LOG
    Sun Feb 11 07:53:55 2007
    Ended recovery at
    Thread 1: logseq 6, block 2021, scn 0.310811
    177 data blocks read, 177 data blocks written, 2018 redo blocks read
    Crash recovery completed successfully
    Sun Feb 11 07:53:56 2007
    LGWR: Primary database is in CLUSTER CONSISTENT mode
    Sun Feb 11 07:53:56 2007
    ARCH: Evaluating archive log 3 thread 1 sequence 4
    ARCH: Beginning to archive log 3 thread 1 sequence 4
    Creating archive destination LOG_ARCHIVE_DEST_1: 'E:\ORACLE\ORA92\RDBMS\ARC00004.001'
    ARCH: Completed archiving log 3 thread 1 sequence 4
    Sun Feb 11 07:53:57 2007
    LGWR: Primary database is in CLUSTER CONSISTENT mode
    Thread 1 advanced to log sequence 7
    Thread 1 opened at log sequence 7
    Current log# 3 seq# 7 mem# 0: E:\ORACLE\ORADATA\TEST\REDO03.LOG
    Successful open of redo thread 1.
    Sun Feb 11 07:53:57 2007
    SMON: enabling cache recovery
    Sun Feb 11 07:54:02 2007
    Undo Segment 1 Onlined
    Undo Segment 2 Onlined
    Undo Segment 3 Onlined
    Undo Segment 4 Onlined
    Undo Segment 5 Onlined
    Undo Segment 6 Onlined
    Undo Segment 7 Onlined
    Undo Segment 8 Onlined
    Undo Segment 9 Onlined
    Undo Segment 10 Onlined
    Successfully onlined Undo Tablespace 1.
    Sun Feb 11 07:54:02 2007
    SMON: enabling tx recovery
    Sun Feb 11 07:54:02 2007
    Database Characterset is WE8MSWIN1252
    Corrupt block relative dba: 0x00406ecc (file 1, block 28364)
    Bad check value found during buffer read
    Data in bad block -
    type: 6 format: 2 rdba: 0x00406ecc
    last change scn: 0x0000.0004668b seq: 0xb flg: 0x04
    consistency value in tail: 0x668b060b
    check value in block header: 0x27f5, computed block checksum: 0x80
    spare1: 0x0, spare2: 0x0, spare3: 0x0
    Reread of rdba: 0x00406ecc (file 1, block 28364) found same corrupted data
    Sun Feb 11 07:54:10 2007
    Errors in file e:\oracle\admin\test\bdump\test_smon_564.trc:
    ORA-00604: error occurred at recursive SQL level 1
    ORA-01578: ORACLE data block corrupted (file # 1, block # 28364)
    ORA-01110: data file 1: 'E:\ORACLE\ORADATA\TEST\SYSTEM01.DBF'
    Sun Feb 11 07:54:10 2007
    Errors in file e:\oracle\admin\test\udump\test_ora_2012.trc:
    ORA-01578: ORACLE data block corrupted (file # 1, block # 28364)
    ORA-01110: data file 1: 'E:\ORACLE\ORADATA\TEST\SYSTEM01.DBF'
    Sun Feb 11 07:54:10 2007
    Error 1578 happened during db open, shutting down database
    USER: terminating instance due to error 1578
    Instance terminated by USER, pid = 2012
    ORA-1092 signalled during: alter database open...
    What I can guess is this all is happenning because system datafile OS block got currupted....is it so or some other reason?....and how to resolve this issue?

    Hi,
    Since it is system tablespace, you can't open the DB. If you can't open the db you can't use dbms_repair. Also DBV doesn't change the content of the file so that also not going to help.
    So only open left is to do point in time recovery.
    Regards,
    Satheesh Babu.S

  • Oracle Instance has been terminated

    Hi,
    we are facing some weird problem with one of our Oracle database where oracle instance is getting terminated very frequently. below are the alert logs. Kindly suggest what could be the issue & how to resolve it.
    Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
    This is an IBM X series server & we have get hardware checked from vednor who have not found anything wrong with hardware (HDD).
    Sat Aug 25 08:00:34 2012
    KCF: read, write or open error, block=0xfba4 online=1
    file=2 'J:\ORACLE\DB1\DB1\SYSAUX01.DBF'
    error=27070 txt: 'OSD-04016: Error queuing an asynchronous I/O request.
    O/S-Error: (OS 1117) The request could not be performed because of an I/O device error.'
    Errors in file J:\ORACLE\DB1\diag\rdbms\DB1\DB1\trace\DB1_dbw0_7012.trc:
    ORA-63999: data file suffered media failure
    ORA-01114: IO error writing block to file 2 (block # 64420)
    ORA-01110: data file 2: 'J:\ORACLE\DB1\DB1\SYSAUX01.DBF'
    ORA-27070: async read/write failed
    OSD-04016: Error queuing an asynchronous I/O request.
    O/S-Error: (OS 1117) The request could not be performed because of an I/O device error.
    Sat Aug 25 08:00:35 2012
    Errors in file J:\ORACLE\DB1\diag\rdbms\DB1\DB1\trace\DB1_m000_5168.trc:
    ORA-00322: log 3 of thread 1 is not current copy
    ORA-00312: online log 3 thread 1: 'H:\ORACLE\DB1\ORIGLOG3\LOG_G3_M1.DBF'
    Sat Aug 25 08:00:35 2012
    System state dump requested by (instance=1, osid=7012 (DBW0)), summary=[abnormal instance termination].
    System State dumped to trace file J:\ORACLE\DB1\diag\rdbms\DB1\DB1\trace\DB1_diag_6996.trc
    DBW0 (ospid: 7012): terminating the instance due to error 63999
    Dumping diagnostic data in directory=[cdmp_20120825080035], requested by (instance=1, osid=7012 (DBW0)), summary=[abnormal instance termination].
    Sat Aug 25 08:00:49 2012
    Exception [type: IN_PAGE_ERROR, ] [] [PC:0x76E59C12, 0000000076E59C12]
    ERROR: Unable to normalize symbol name for the following short stack (at offset 213):
    dbgexProcessError()+200<-dbgeExecuteForError()+65<-dbgDB1stErrorKGE()+2235<-dbkDB1stKGE_kgsf()+77<-kgeade()+563<-kgerelv()+151<-kgerev()+45<-kgerec5()+60<-sss_xcpt_EvalFilterEx()+1869<-sss_xcpt_EvalFilter()+174<-.1.4_5+59<-0000000076E485A8<-0000000076E59D0D<-0000000076E491AF<-0000000076E81278<-0000000076E59C12<-0000000076E48F14<-0000000076E81278<-0000000076E59C12<-0000000076E48F14<-0000000076E81278<-0000000076E59C12<-0000000076E48F14<-0000000076E81278<-0000000076E59C12<-0000000076E48F14<-0000000076E81278<-0000000076E59C12<-0000000076E48F14<-0000000076E81278<-0000000076E59C12<-0000000076E48F14<-0000000076E81278<-0000000076E59C12<-0000000076E48F14<-0000000076E81278<-0000000076E59C12<-0000000076E48F14<-0000000076E81278<-0000000076E59C12<-0000000076E48F14<-0000000076E81278<-0000000076E59C12<-0000000076E48F14<-0000000076E81278<-0000000076E59C12<-0000000076E48F14<-0000000076E81278<-0000000076E59C12<-0000000076E48F14
    Errors in file J:\ORACLE\DB1\diag\rdbms\DB1\DB1\trace\DB1_d000_7044.trc (incident=153752):
    ORA-07445: exception encountered: core dump [PC:0x76E59C12] [IN_PAGE_ERROR] [] [PC:0x76E59C12] [] []
    Incident details in: J:\ORACLE\DB1\diag\rdbms\DB1\DB1\incident\incdir_153752\DB1_d000_7044_i153752.trc
    Use ADRCI or Support Workbench to package the incident.
    See Note 411.1 at My Oracle Support for error and packaging details.
    Sat Aug 25 08:00:52 2012
    Instance terminated by DBW0, pid = 7012
    Dhiraj
    Edited by: 876613 on Aug 25, 2012 1:15 AM

    Sorry i don't have much knowledge about oracle so posting here trace log file details.
    Dhiraj
    Trace file J:\ORACLE\db1\diag\rdbms\db1\db1\trace\db1_dbw0_7012.trc
    Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
    With the Partitioning, OLAP, Data Mining and Real Application Testing options
    Windows NT Version V6.1 Service Pack 1
    CPU : 16 - type 8664, 8 Physical Cores
    Process Affinity : 0x0x0000000000000000
    Memory (Avail/Total): Ph:4773M/8148M, Ph+PgF:23035M/26535M
    Instance name: db1
    Redo thread mounted by this instance: 1
    Oracle process number: 10
    Windows thread id: 7012, image: ORACLE.EXE (DBW0)
    *** 2012-08-25 08:00:34.653
    *** SESSION ID:(161.1) 2012-08-25 08:00:34.653
    *** CLIENT ID:() 2012-08-25 08:00:34.653
    *** SERVICE NAME:(SYS$BACKGROUND) 2012-08-25 08:00:34.653
    *** MODULE NAME:() 2012-08-25 08:00:34.653
    *** ACTION NAME:() 2012-08-25 08:00:34.653
    KCF: read, write or open error, block=0xfba4 online=1
    file=2 'J:\ORACLE\db1\db1\SYSAUX01.DBF'
    error=27070 txt: 'OSD-04016: Error queuing an asynchronous I/O request.
    O/S-Error: (OS 1117) The request could not be performed because of an I/O device error.'
    DDE rules only execution for: ORA 1110
    ----- START Event Driven Actions Dump ----
    ---- END Event Driven Actions Dump ----
    ----- START DDE Actions Dump -----
    Executing SYNC actions
    ----- START DDE Action: 'DB_STRUCTURE_INTEGRITY_CHECK' (Async) -----
    Successfully dispatched
    ----- END DDE Action: 'DB_STRUCTURE_INTEGRITY_CHECK' (SUCCESS, 0 csec) -----
    Executing ASYNC actions
    ----- END DDE Actions Dump (total 0 csec) -----
    error 63999 detected in background process
    ORA-63999: data file suffered media failure
    ORA-01114: IO error writing block to file 2 (block # 64420)
    ORA-01110: data file 2: 'J:\ORACLE\db1\db1\SYSAUX01.DBF'
    ORA-27070: async read/write failed
    OSD-04016: Error queuing an asynchronous I/O request.
    O/S-Error: (OS 1117) The request could not be performed because of an I/O device error.
    kjzduptcctx: Notifying DIAG for crash event
    ----- Abridged Call Stack Trace -----
    ksedsts()+585<-kjzdssdmp()+329<-kjzduptcctx()+288<-kjzdicrshnfy()+87<-ksuitm()+1557<-ksbrdp()+4373<-opirip()+853<-opidrv()+907<-sou2o()+98<-opimai_real()+299<-opimai()+191<-BackgroundThreadStart()+646<-0000000076A1652D<-0000000076E5C521
    ----- End of Abridged Call Stack Trace -----
    *** 2012-08-25 08:00:35.683
    DBW0 (ospid: 7012): terminating the instance due to error 63999
    Dump file j:\oracle\db1\diag\rdbms\db1\db1\trace\db1_dbw0_7012.trc
    Sat Aug 25 08:00:42 2012
    ORACLE V11.2.0.2.0 - 64bit Production vsnsta=0
    vsnsql=16 vsnxtr=3
    *** 2012-08-25 08:00:42.079 ssthreadkillsafe: about to forcefully terminate tid=5904, dumping call stack:
    ------------------- Call Stack Trace ---------------------
    *** 2012-08-25 08:00:42.204
    calling location entry point arg #1 arg #2 arg #3 arg #4
    0000000076E8165A 0000000000000000 0000000000000000 0000000000000011 0000000010847e38 0000000000000011
    000007FEFD3A1203 0000000076E81650 0000000026c0d268 0000000000000000 0000000026c0d1d0 0000000000000000
    ksuitm()+840 000007FEFD3A1150 000000000ae6b180 0000000000000001 0000000000000000 0000000000000000
    ksliwat()+16250 ksuitm() 0000000000000000 0000000000000444 0000000000000000 0000000000000000
    kslwaitctx()+184 ksliwat() 000007ff7b46e698 000007ff7b46e698 000001f4000001f4 0000000000023660
    kslwait()+160 kslwaitctx() 0000000000000001 000007ff747e4240 0000000000000048 0000000145e4d6f9
    ksarcv()+222 kslwait() 000000a2002a280e 00000000000000e2 00000000000000cd 0000000000024dcc
    ksbabs()+539 ksarcv() 00000000000001d0 0000000100000fd8 0000000000000001 000000001d3967e0
    ksbrdp()+1298 ksbabs() 0000000000000000 0000000000000000 0000000000000000 0000000000000000
    opirip()+853 ksbrdp() 000000000000001e 00000001474b1574 0000000026c0f930 0000000000000001
    opidrv()+907 opirip() 0000000000000032 0000000000000004 0000000026c0fd20 4f5c3a4a00000000
    sou2o()+98 opidrv()+150 0000000000000032 0000000000000004 0000000026c0fd20 0000000000000000
    opimai_real()+299 sou2o() 01cd81d90b41b30c 0000000000000000 000e0018000807dc 00050023002e002d
    opimai()+191 opimai_real() 0000000000000000 0000000000000000 0000000000000000 0000000000000000
    BackgroundThreadStart()+646 opimai() 0000000026c0fe68 000000000000000d 0000000000000000 0000000000000000
    0000000076A1652D BackgroundThreadStart() 000000001227de30 0000000000000000 0000000000000000 0000000000000000
    0000000076E5C521 0000000076A16520 0000000000000000 0000000000000000 0000000000000000 0000000000000000
    ---------------- End of Call Stack Trace -----------------
    Call stack acquisition performance stats:
    setup time (lock acquis., memory alloc.): 0 ms
    frame get time (time the target proc was suspended): 94 ms
    symbol translation time: 0 ms
    total time: 94 ms
    *** 2012-08-25 08:00:48.428 ssthreadkillsafe: about to forcefully terminate tid=6232, dumping call stack:
    ------------------- Call Stack Trace ---------------------
    *** 2012-08-25 08:00:48.506
    calling location entry point arg #1 arg #2 arg #3 arg #4
    0000000076E8165A 0000000000000000 0000000000000000 000000000000000d 0000000010847e38 000000000000000d
    000007FEFD3A1203 0000000076E81650 00000000169de798 000080f153baec1f 000007ff01002000 0000000000000000
    sltrusleep()+58 000007FEFD3A1150 0000000000023558 0000002500000000 0000002500000000 0000000000000000
    kskvkrmmain()+817 sltrusleep() 0000000000000000 0000000000000003 0000000000000008 000000209a2e0ea0
    ksbrdp()+1298 kskvkrmmain() 0000000000000000 0000000000000000 0000000000000000 0000000000000000
    opirip()+853 ksbrdp() 000000000000001e 00000001474b1574 00000000169df930 0000000000000001
    opidrv()+907 opirip() 0000000000000032 0000000000000004 00000000169dfd20 4f5c3a4a00000000
    sou2o()+98 opidrv()+150 0000000000000032 0000000000000004 00000000169dfd20 0000000000000000
    opimai_real()+299 sou2o() 01cd8258c2d597a9 0000000000000000 00060019000807dc 0006007000000000
    opimai()+191 opimai_real() 0000000000000000 0000000000000000 0000000000000000 0000000000000000
    BackgroundThreadStart()+646 opimai() 00000000169dfe68 000000000000000d 0000000000000000 0000000000000000
    0000000076A1652D BackgroundThreadStart() 000000000b79d160 0000000000000000 0000000000000000 0000000000000000
    0000000076E5C521 0000000076A16520 0000000000000000 0000000000000000 0000000000000000 0000000000000000
    ---------------- End of Call Stack Trace -----------------
    Call stack acquisition performance stats:
    setup time (lock acquis., memory alloc.): 0 ms
    frame get time (time the target proc was suspended): 62 ms
    symbol translation time: 0 ms
    total time: 62 ms
    *** 2012-08-25 08:00:52.453
    ksuitm: waiting up to [5] seconds before killing DIAG(6996)

  • Rac Instance Crashes

    Dear all,
    My version is 11.2.0.2.5 one of my rac instance crashes with message ORA-00240: control file enqueue held for more than 120 seconds. Received an instance abort message from instance 1.
    here are the contents of alert log file
    IPC Send timeout detected. Receiver ospid 27423 [[email protected] (LMON)]
    2013-03-22 22:30:05.644000 -07:00
    Errors in file /u01/app/oracle/diag/rdbms/lfgoimdb/LFGoimdb2/trace/LFGoimdb2_lmon_27423.trc:
    2013-03-22 22:31:08.734000 -07:00
    Errors in file /u01/app/oracle/diag/rdbms/lfgoimdb/LFGoimdb2/trace/LFGoimdb2_arc2_27691.trc (incident=15905):
    ORA-00240: control file enqueue held for more than 120 seconds
    Incident details in: /u01/app/oracle/diag/rdbms/lfgoimdb/LFGoimdb2/incident/incdir_15905/LFGoimdb2_arc2_27691_i15905.trc
    2013-03-22 22:31:13.409000 -07:00
    Received an instance abort message from instance 1
    Please check instance 1 alert and LMON trace files for detail.
    LMS0 (ospid: 27427): terminating the instance due to error 481
    System state dump requested by (instance=2, osid=27427 (LMS0)), summary=[abnormal instance termination].
    System State dumped to trace file /u01/app/oracle/diag/rdbms/lfgoimdb/LFGoimdb2/trace/LFGoimdb2_diag_27413.trc
    2013-03-22 22:31:18.376000 -07:00
    Dumping diagnostic data in directory=[cdmp_20130322223113], requested by (instance=2, osid=27427 (LMS0)), summary=[abnormal instance termination].
    ORA-1092 : opitsk aborting process
    Instance terminated by LMS0, pid = 27427

    Thanks for reply,
    My redo logs size is default 50mb.There is currently no load on the system since we are not using this environment for time being.The log switches are averaged to be 8 per day.I think Increasing the size of redo will further cause the problems since the archiver may again hold lock for more time.
    Since there is no dedicated connection between the nodes and storage ,So increasing the hardware and network configuration is only solution to this? Or I am still missing something...
    As far as configuration is considered i cannot add more resources to this environment.How can I solve this issue?

  • Oracle instance often down!!!

    Hi
    I've have an Oracle 10g (10.2.0.4) 64bits database installed on a x64 window server. The server itself has 16gb ram. My current SGA + PGA is about 4-5gb memory, processes=500 but I am still getting random problem like oracle instance killed/down, low swap page memory from OS, listener could not hand off client connection etc. Should it be the existing server RAM(10-11gb RAM free after deduct from the SGA+PGA allocated) is more than enough?

    FireFox88 wrote:
    So far from what I can recall my database server used to once a while getting "Significant virtual memory paging was detected on the host operating system" until i turn the OS page memory to system manage which is to use whatever hard disk space available, only then the warning seldom/rarely occur. listener could not hand off client connection is another error quite often prompt. When ever this error prompt, most of the time I've to bounce the database instance.Speculation - could be caused by PGA, as a result of poorly written PL/SQL code using very large PL/SQL arrays (for bulk collection). I have seen a single PGA (private process memory of a dedicated server process) growing to 50% the RAM size of the server. I have even seen a couple of server crashes due to the kernel being starved of CPU time due to swap space thrashing.
    Before resetting a server in such a situation - get whatever diagnostics available. The problem will be gone (for the time being) after reboot, without the ability to get an actual look at just what the heck is going on. There are some magic sysrequest keys on Linux that is useful - all you need is the ability to echo specific keys to a proc device in order talk to the kernel and request various data and process dumps. Pity you are using Windows - not the easiest of kernels to troubleshoot.

  • Database instance crashes when running expdp

    Hi,
    I have oracle 10.2.0.4 running on AIX.
    when i try to run expdp as below...
    expdp system/pwd directory=DATA_PUMP_DIR2 tables=TOAD_PLAN_TABLE dumpfile=test.dmp
    i get an error as..
    UDE-00008: operation generated ORACLE error 22303
    OCI-22303: type "SYS"."KU$_STATUS1020" not found
    and the database instance crashes.
    I tried increasing the shared_memory_pool, checking the read, write permissions on the directory. but every single time i run the expdp, the instance crashes.
    But when i try to run the exp for the same table, it runs fine.
    Can someone help me resolve this so i can use the expdp feature.
    Thanks.
    Philip.

    I could not find anything proper w.r.t the datapump failure.
    is there a package or something i need to install in the database for the expdp to work.
    or do i have to install any libraries for the datapump to work.
    Thanks.

  • SAP oracle database crash with errore:ORA-00490: PSP process terminated

    Hi ALL,
    Our oracle database crash with errore code in trace : ORA:00490; i started database again and it  working fine; but could not fine reason of down and what is this erroe all about
    can some help me
    Errors in file /oracle/SRD/saptrace/background/srd_pmon_28096.trc:
    ORA-00490: PSP process terminated with error
    Tue Nov 25 09:00:57 2008
    PMON: terminating instance due to error 490
    Instance terminated by PMON, pid = 28096
    Thanks,
    Dinesh

    Hi stefen,
    please find the trace file as below
    /oracle/SRD/saptrace/background/srd_pmon_28096.trc
    Oracle Database 10g Enterprise Edition Release 10.2.0.2.0 - 64bit Production
    With the Partitioning and Data Mining options
    ORACLE_HOME = /oracle/SRD/102_64
    System name:    SunOS
    Node name:      nzlsfn23
    Release:        5.10
    Version:        Generic_137111-01
    Machine:        sun4u
    Instance name: SRD
    Redo thread mounted by this instance: 1
    Oracle process number: 2
    Unix process pid: 28096, image: oracle@nzlsfn23 (PMON)
    2008-11-25 09:00:57.497
    SERVICE NAME:(SYS$BACKGROUND) 2008-11-25 09:00:56.210
    SESSION ID:(24.1) 2008-11-25 09:00:56.194
    Background process PSP0 found dead
    Oracle pid = 6
    OS pid (from detached process) = 28098
    OS pid (from process state) = 28098
    dtp = 38000afd8, proc = 497000860
    Dump of memory from 0x000000038000AFD8 to 0x000000038000B020
    38000AFD0                   00000005 00000000          [........]
    38000AFE0 00000004 97000860 00000000 00000000  [.......`........]
    38000AFF0 00000000 50535030 00020000 00000000  [....PSP0........]
    38000B000 00006DC2 00000000 00000000 48E50DA4  [..m.........H...]
    38000B010 00000001 000E3273 00040081 00000000  [......2s........]
    Dump of memory from 0x0000000497000860 to 0x0000000497001048
    497000860 02010000 00000000 00000000 00000000  [................]
    497000870 00000000 00000000 00000000 00000000  [................]
    497000880 00000004 97042570 00000004 97047810  [......%p......x.]
    497000890 00000004 97028E68 00000004 97045BE8  [.......h......[.]
    4970008A0 00000000 00000000 00000004 97045C70  [..............\p]
    4970008B0 00000004 97045C70 00000004 97047800  [......\p......x.]
    4970008C0 01060000 00000000 00000004 97025ED8  [..............^.]
    4970008D0 00000004 97028E68 00000006 00000000  [.......h........]
    4970008E0 00000000 00000000 00000000 00000000  [................]
    4970008F0 00000000 00000000 00000004 97042490  [..............$.]
    497000900 00000004 970425A0 00000000 00000000  [......%.........]
    497000910 00000000 00000000 00000000 00000000  [................]
            Repeat 3 times
    497000950 00000003 00000000 00000000 00000000  [................]
            Repeat 1 times
    497000970 00000000 00000000 00000000 00000000  [................]
    497000980 00000004 00000000 00000000 00000000  [................]
    497000990 00000003 00000000 00000000 00000000  [................]
            Repeat 1 times
    4970009B0 00000004 00000000 00000000 00000000  [................]
    4970009C0 00000005 00000000 00000000 00000000  [................]
    4970009D0 00000003 00000000 00000000 00000000  [................]
    4970009E0 00000000 00000000 00000000 00000000  [................]
            Repeat 8 times
    497000A70 00000000 00000000 00000004 97000A78  [...............x]
    497000A80 00000004 97000A78 00000000 00000000  [.......x........]
    497000A90 00000000 00000000 00000004 97000A98  [................]
    497000AA0 00000004 97000A98 00000000 00000000  [................]
    497000AB0 00000000 00000000 00000000 00000000  [................]
            Repeat 2 times
    497000AE0 00000000 00000000 00000018 00000030  [...............0]
    497000AF0 00000001 00000B3D 00000004 970037D0  [.......=......7.]
    497000B00 00000004 580096B0 00000001 00000000  [....X...........]
    497000B10 00000000 00000000 00000000 00000000  [................]
            Repeat 2 times
    497000B40 00006DC2 00000000 00000000 00000000  [..m.............]
    497000B50 00000000 00000000 00000000 00000000  [................]
            Repeat 2 times
    497000B80 00000004 97000860 00000000 00000000  [.......`........]
    497000B90 00000000 00000000 00000000 00000000  [................]
            Repeat 7 times
    497000C10 00000004 97000C10 00000004 97000C10  [................]
    497000C20 00000000 00000000 00010000 00000000  [................]
    497000C30 00000000 00000117 0000000A 00000000  [................]
    497000C40 00006DC2 00000000 00000000 48E50DA4  [..m.........H...]
    497000C50 00000001 00000000 00000000 00000000  [................]
    497000C60 00000000 00000000 00000000 00000000  [................]
            Repeat 2 times
    497000C90 00000000 00000000 00000003 FFFFFFFF  [................]
    497000CA0 00000000 00000000 00000000 00000000  [................]
            Repeat 13 times
    497000D80 73726461 646D0000 00000000 00000000  [srdadm..........]
    497000D90 00000000 00000000 00000000 00000000  [................]
    497000DA0 00000000 00000006 6E7A6C73 666E3233  [........nzlsfn23]
    497000DB0 00000000 00000000 00000000 00000000  [................]
            Repeat 2 times
    497000DE0 00000000 00000000 00000000 00000008  [................]
    497000DF0 554E4B4E 4F574E00 00000000 00000000  [UNKNOWN.........]
    497000E00 00000000 00000000 00000000 00000000  [................]
    497000E10 00000000 00000008 32383039 38000000  [........28098...]
    497000E20 00000000 00000000 00000000 00000000  [................]
    497000E30 00000000 00000005 6F726163 6C65406E  [........oracle@n]
    497000E40 7A6C7366 6E323320 28505350 30290000  [zlsfn23 (PSP0)..]
    497000E50 00000000 00000000 00000000 00000000  [................]
    497000E60 00000000 00000000 00000000 00000016  [................]
    497000E70 00000000 00000002 00000000 00000000  [................]
    497000E80 00000000 00000000 00000000 00000000  [................]
            Repeat 8 times
    497000F10 00000000 00000000 00000000 00020000  [................]
    497000F20 00000000 00000000 00000000 00000000  [................]
    497000F30 00000000 00000000 00000003 9E1F6748  [..............gH]
    497000F40 00000004 97001728 00000004 97000758  [.......(.......X]
    497000F50 00000000 00000000 00000003 9E26B5B0  [.............&..]
    497000F60 00000000 00000000 00000000 00000000  [................]
            Repeat 1 times
    497000F80 00000004 97000F80 00000004 97000F80  [................]
    497000F90 00000000 00040000 00000000 00000000  [................]
    497000FA0 00000000 00031A55 00000000 0004D7DD  [.......U........]
    497000FB0 00000000 00071A55 00000000 00000000  [.......U........]
    497000FC0 00000000 00000000 00000000 00000000  [................]
    497000FD0 00000000 00000828 00000000 000000E0  [.......(........]
    497000FE0 00000000 00000828 00000000 00000000  [.......(........]
    497000FF0 00000000 00000000 00000000 00000000  [................]
            Repeat 4 times
    497001040 00000002 00000000                    [........]
    error 490 detected in background process
    ORA-00490: PSP process terminated with error

  • OC4J instance crashing after Ping Failed: default_group~home~default_group~

    Hi All,
    I have noticed from opmn logs that [pm-process] Process Crashes and System has been restarted automatically by itself.
    I have read the tread with title "OC4J instance crashing - can't find any diagnostics " and I have configured the opmn.xml file in order to enable opmn.dbg logs.
    In the meantime I have suspected that the BPEL crashes are the same date with default_group~home~default_group logs rotation.
    [oracle@csporacle1 logs] cat /etc/redhat-release
    Red Hat Enterprise Linux ES release 4 (Nahant Update 5)
    [oracle@csporacle1 logs] cat opmn.log | grep "Restarting"
    10/05/06 17:10:18 [pm-process] Process Crashed: default_group~home~default_group~1 (481247799:27664) - Restarting
    10/07/08 03:38:12 [pm-process] Process Crashed: default_group~home~default_group~1 (481247801:12964) - Restarting
    10/08/05 15:01:08 [pm-process] Process Crashed: default_group~home~default_group~1 (481247802:22304) - Restarting
    10/09/10 04:19:56 [pm-process] Process Crashed: default_group~home~default_group~1 (481247803:5081) - Restarting
    10/10/15 00:04:21 [pm-process] Process Crashed: default_group~home~default_group~1 (481247804:7275) - Restarting
    10/11/20 14:56:29 [pm-process] Process Crashed: default_group~home~default_group~1 (481247806:28469) - Restarting
    10/12/25 11:15:39 [pm-process] Process Crashed: default_group~home~default_group~1 (481247807:17717) - Restarting
    11/02/02 22:09:16 [pm-process] Process Crashed: default_group~home~default_group~1 (481247808:540) - Restarting
    11/02/26 12:14:55 [pm-process] Process Crashed: default_group~home~default_group~1 (481247809:26490) - Restarting
    11/03/12 04:28:19 [pm-process] Process Crashed: default_group~home~default_group~1 (481247812:7348) - Restarting
    11/04/25 05:43:59 [pm-process] Process Crashed: default_group~home~default_group~1 (481247817:9090) - Restarting
    11/05/21 08:27:55 [pm-process] Process Crashed: default_group~home~default_group~1 (481247818:26304) - Restarting
    11/06/18 08:04:03 [pm-process] Process Crashed: default_group~home~default_group~1 (481247826:28958) - Restarting
    11/07/12 19:04:46 [pm-process] Process Crashed: default_group~home~default_group~1 (481247828:10867) - Restarting
    11/08/21 14:33:08 [pm-process] Process Crashed: default_group~home~default_group~1 (481247832:13556) - Restarting
    11/09/18 20:40:16 [pm-process] Process Crashed: default_group~home~default_group~1 (481247835:22024) - Restarting
    11/10/17 13:56:47 [pm-process] Process Crashed: default_group~home~default_group~1 (481247838:20504) - Restarting
    [oracle@csporacle1 logs] ls -lrt
    total 6020344
    -rw-r--r--  1 oracle dba          0 Feb 17  2010 opmn.out
    -rw-r--r--  1 oracle dba   84895911 Feb  2  2011 default_group~home~default_group~1.log.11-02-02_22:09:16.gz
    -rw-r--r--  1 oracle dba   83151884 Feb 26  2011 default_group~home~default_group~1.log.11-02-26_12:14:55.gz
    -rw-r--r--  1 oracle dba   79803990 Mar 12  2011 default_group~home~default_group~1.log.11-03-12_04:28:19.gz
    -rw-r--r--  1 oracle dba   66470738 Mar 29  2011 default_group~home~default_group~1.log.11-03-29_00:17:05.gz
    -rw-r--r--  1 oracle dba   84542865 Apr 25  2011 default_group~home~default_group~1.log.11-04-25_05:43:59.gz
    -rw-r--r--  1 oracle dba   83672948 May 21 08:27 default_group~home~default_group~1.log.11-05-21_08:27:55.gz
    -rw-r--r--  1 oracle dba   84825282 Jun 18 08:04 default_group~home~default_group~1.log.11-06-18_08:04:03.gz
    -rw-r--r--  1 oracle dba   86740423 Jul 12 19:04 default_group~home~default_group~1.log.11-07-12_19:04:46.gz
    -rw-r--r--  1 oracle dba   66921060 Aug  3 23:06 default_group~home~default_group~1.log.11-08-04_02:28:06.gz
    -rw-r--r--  1 oracle dba   71221025 Aug 21 14:33 default_group~home~default_group~1.log.11-08-21_14:33:08.gz
    -rw-r--r--  1 oracle dba 2147483647 Sep 18 20:39 default_group~home~default_group~1.log.11-09-18_20:40:16
    -rw-r--r--  1 oracle dba 2147483647 Oct 17 13:56 default_group~home~default_group~1.log.11-10-17_13:56:49
    drwxr-x---  2 oracle dba       4096 Oct 24 00:11 states
    -rw-r--r--  1 oracle dba      64203 Oct 24 20:09 opmn.log
    -rw-r--r--  1 oracle dba  976001440 Oct 28 02:35 default_group~home~default_group~1.log
    -rw-r--r--  1 oracle dba   95431960 Oct 28 02:36 opmn.dbg
    [oracle@csporacle1 logs] After this I have check the opmn.xml and found that the ORACLE_HOME has been set to other oracle on the same Unit.
    oracle@csporacle1 opmn] cat conf/opmn.xml | head -20
    <?xml version = '1.0' encoding = 'UTF-8'?>
    <opmn xmlns="http://www.oracle.com/ias-instance">
       <log path="$ORACLE_HOME/opmn/logs/opmn.log" comp="internal;ons;pm" rotation-size="1500000"/>
       <debug path="$ORACLE_HOME/opmn/logs/opmn.dbg" comp="internal;pm;ons" rotation-size="1500000"/>
       <notification-server interface="ipv4">
          <port local="6100" remote="6200" request="6003"/>
          <ssl enabled="true" wallet-file="$ORACLE_HOME/opmn/conf/ssl.wlt/default"/>
       </notification-server>
       <process-manager>
          <process-modules>
             <module path="$ORACLE_HOME/opmn/lib/libopmnohs">
                <module-id id="OHS"/>
             </module>
             <module path="$ORACLE_HOME/opmn/lib/libopmnoc4j">
                <module-id id="OC4J"/>
             </module>
             <module path="$ORACLE_HOME/opmn/lib/libopmncustom">
                <module-id id="CUSTOM"/>
             </module>
             <module path="$ORACLE_HOME/opmn/lib/libopmniaspt">
    [oracle@csporacle1 opmn] pwd
    /space/u01/app/oracle/product/10.1.3.1/OracleAS_1/opmn
    [oracle@csporacle1 opmn] cd $ORACLE_HOME
    [oracle@csporacle1 10.1] pwd
    /u01/app/oracle/product/10.1
    [oracle@csporacle1 10.1] Are these error are related with my findings or are there any criteria must i check further.
    Could you please help me to solve this problem.
    Thanks & Best Regards,
    Bahtiyar

    gday mattysmith --
    Unfortunately that's a very good question.
    As OPMN is in the picture here, the first step may be to enable debug logging for OPMN in the chance that it may give you more information about why it thinks the process crashed.
    Generally speaking, when OPMN is restarting a process, it can be because it recognized that a managed process has dissapeared (ie JVM core dumped, etc.) or it can also be because the JVM didn't respond to any of OPMNs requests to let it know it was still alive. In which case OPMN will kill then restart the managed process. This can be caused by situations where the JVM has run out of memory, etc. and becomes unresponsive, leading to it eventually being detected as such and restarted.
    What you can do is to add the OPMN PM submodule to the OPMN debug logger.
    <opmn xmlns="http://www.oracle.com/ias-instance">
    <log path="$ORACLE_HOME\opmn\logs\opmn.log" comp="internal;ons;pm" rotation-s
    ize="1500000"/>
    <debug path="$ORACLE_HOME\opmn\logs\opmn.dbg" comp="internal;pm;ons" rotation-size="1500000"/>
    </opmn>
    That should then produce an opmn.dbg file in the $ORACLE_HOME/opmn/logs directory. It'll be very verbose, so you'll need to associate the restart times from the other log file to locate the relevant section of the file. Hopefully that may shed some light on what is happening, at least at the restart level.
    Usually if there are runtime exceptions such as OOM errors, etc. then those are dumped to stderr/stdout. Check the contents of the other log files in the opmn/logs directory of the form default_group~home~default_group~1.log to see if there is anything obvious in there.
    There has been reports of an issue with PermGen space usage -- that could be something you are running into under the covers. Try resizing the permgen space to be a little larger and see if the problem goes away -- that can be done in opmn.xml in the java > start-options section of the instance in question.
    See how any of that goes!
    -steve-

  • Instance crash?

    Imagine these circumstances:
    1. I make an update to one/multiple rows of a table, but do not commit
    2. My instance crashes.
    How does Oracle manage this? Please correct me if i am wrong
    Below is what i have understood
    1. SMON does the instance Recovery (Rolling Forward only) for below cases
         a)Uncommited transaction will be recovered if only dirty buffer is written to the datafile even not committed [ alter system checkpoint ]
         b)Uncommited transaction will be recovered if only Redo is generated or log switches occur, or the logs are archived [alter system switch
           logfile/alter system archive log current]
    2. Uncommited transaction will be not be recovered if all above of these doesn't happen.Then SMON will do [ Rolling Forward + Rolling Backward ]

    913410 wrote:
    Imagine these circumstances:
    1. I make an update to one/multiple rows of a table, but do not commit
    2. My instance crashes.
    How does Oracle manage this? Please correct me if i am wrong
    Below is what i have understood
    1. SMON does the instance Recovery (Rolling Forward only) for below cases
         a)Uncommited transaction will be recovered if only dirty buffer is written to the datafile even not committed [ alter system checkpoint ]
         b)Uncommited transaction will be recovered if only Redo is generated or log switches occur, or the logs are archived [alter system switch
           logfile/alter system archive log current]
    2. Uncommited transaction will be not be recovered if all above of these doesn't happen.Then SMON will do [ Rolling Forward + Rolling Backward ]
    This is quite well explained here,
    http://docs.oracle.com/cd/E11882_01/server.112/e25789/startup.htm#CNCPT1301
    Aman....

  • Instance crash on RAC will impact goldengate extract process?

    Dear All,
    DB Version: 11.2.0.3
    GG version: 11.2.0.1
    Could you please let me know, how gg works when a instance crashes in a RAC cluster. Will extract proess get abend or extract will continue to capture
    Thank you very much in advance
    Regards
    Mvk

    If GG is configured properly for the RAC, the GG manager should start extract on the living node and there should not be any data loss.
    Please refer to Oracle documentation on how to configure GG in RAC.

  • Difference between Instance Crash and Database Crash in Standalone and RAC?

    I am new to Oracle Environment. I would appreiciate if i get clarified on the "_+Difference between Instance Crash and Database Crash in Standalone and
    RAC_".+
    Oracle Version: 11.1.0.7.0
    O.S Version: HP-UX B.11.31 U

    Sivaprasad S wrote:
    I am new to Oracle Environment. I would appreiciate if i get clarified on the "_+Difference between Instance Crash and Database Crash in Standalone and
    RAC_".+That depends on your definitions of database crash versus instance crash.
    Literally, database crash means that the database itself is damaged in some way. A disk is no more. Data files are corrupted. Etc. The database itself thus causes the crash - and a database crash will also cause all database instances (RAC or single) for that database to crash too.
    An instance crash will therefore mean that the crash occurs in the instance due to an Oracle bug or similar. In other words a s/w error and not a database error. The database is still "fine", but the instance not. Instance crashes. However, any other instance on that same (RAC) database will still be fine as the database itself is unaffected by that s/w error in the crashed instance.
    If you mean something different, please enlighten us. Not all crashes are equal. {noformat};-){noformat}

Maybe you are looking for