ORA-00476 RECO process terminated with erro

When running the code below the 10g/Linux instance errors with
Error: ORA 476
Text: RECO process terminated with error
Cause: The distributed transaction, two-phase commit, recovery process died.
Action: Restart the instance
There's almost 21 million rows in tbl stage action. tbl v_map's empty before running this.
SELECT DISTINCT DECODE(NVL(m_id,'0'),'0', ip_address, m_id), trg, dma, state
bulk collect
INTO actnlog
FROM stage_action
minus
SELECT v_name, segment, dma, state
FROM v_map;
** FORALL insert into v_map
FORALL i IN 1 .. actnlog.COUNT
INSERT INTO
(SELECT v_name, segment, dma, state FROM v_map)
VALUES actnlog(i);
COMMIT;
Can someone tell me what's wrong with this code? Why would it cause the error? Is it the COMMIT that executes 21 mln times?
Thanks,

I know...I didn't think it was the code that's causing this error, until it happened twice in a row in test environment and then in dev. Unfortunately, I don't administer the servers and don't have DBA access, but DBA told me that this has caused the servers to go down in all three cases.
Thanks,

Similar Messages

  • ORA-00476: RECO process terminated with error

    My dev instance shutdown automatically due to error 476. I looked up on the error details and found this:
    Sun Jan 7 10:17:39 2007
    Errors in file /usr/lib/oracle/xe/app/oracle/admin/XE/bdump/xe_pmon_26220.trc:
    ORA-00476: RECO process terminated with error
    Sun Jan 7 10:17:39 2007
    PMON: terminating instance due to error 476
    Instance terminated by PMON, pid = 26220
    Details for this error from xe_pmon_26220.trc file:
    /usr/lib/oracle/xe/app/oracle/admin/XE/bdump/xe_pmon_26220.trc
    Oracle Database 10g Express Edition Release 10.2.0.1.0 - Production
    ORACLE_HOME = /usr/lib/oracle/xe/app/oracle/product/10.2.0/server
    System name:     Linux
    Node name:     boromir
    Release:     2.6.9-34.0.1.ELsmp
    Version:     #1 SMP Wed May 17 17:05:24 EDT 2006
    Machine:     i686
    Instance name: XE
    Redo thread mounted by this instance: 1
    Oracle process number: 2
    Unix process pid: 26220, (PMON)
    *** 2007-01-07 10:17:34.543
    *** SERVICE NAME:(SYS$BACKGROUND) 2007-01-07 10:17:18.261
    *** SESSION ID:(225.1) 2007-01-07 10:17:18.261
    Background process RECO found dead
    Oracle pid = 9
    OS pid (from detached process) = 26234
    OS pid (from process state) = 26234
    error 476 detected in background process
    ORA-00476: RECO process terminated with error
    After restarting database I found information about succesfull data recovery
    Mon Jan 8 10:37:59 2007
    Starting ORACLE instance (normal)
    Started redo scan
    Completed redo scan
    2126 redo blocks read, 42 data blocks need recovery
    Started redo application at
    Thread 1: logseq 152, block 56161
    Recovery of Online Redo Log: Thread 1 Group 1 Seq 152 Reading mem 0
    Mem# 0 errs 0: /usr/lib/oracle/xe/app/oracle/flash_recovery_area/XE/onlinelog/o1_mf_1_2f61ppsy_.log
    Completed redo application
    Completed crash recovery at
    Thread 1: logseq 152, block 58287, scn 6441576
    42 data blocks read, 42 data blocks written, 2126 redo blocks read
    Thread 1 checkpoint: logseq 152, block 2, scn 6369309
    cache-low rba: logseq 152, block 56161
    on-disk rba: logseq 152, block 58287, scn 6421576
    start recovery at logseq 152, block 56161, scn 0
    ----- Redo read statistics for thread 1 -----
    Read rate (ASYNC): 1063Kb in 0.20s => 5.19 Mb/sec
    Total physical reads: 4096Kb
    Longest record: 2Kb, moves: 0/1139 (0%)
    Change moves: 0/24 (0%), moved: 0Mb
    Longest LWN: 6Kb, moves: 0/1047 (0%), moved: 0Mb
    Last redo scn: 0x0000.0061fc47 (6421575)
    *** 2007-01-08 10:38:05.342
    KCRA: start recovery claims for 42 data blocks
    *** 2007-01-08 10:38:05.446
    KCRA: blocks processed = 42/42, claimed = 42, eliminated = 0
    *** 2007-01-08 10:38:05.448
    Recovery of Online Redo Log: Thread 1 Group 1 Seq 152 Reading mem 0
    What are the possible reasons of error ??
    Konrad

    it seems that it might be related to killing the RECO
    process on the OS level. The crash happen on Sunday (Sun Jan 7 10:17:39 2007) and no one have phisical access to OS
    Are you using database links? Might be related to
    that, too. No I dont use database links. One database with two schemas.
    Does it happen frequently?It happen only one time.
    Regards,
    Konrad.

  • ORA-00472: PMON  process terminated with error

    Hi,
    I got this error in alert log.But the clients didn't complain they had a downtime.But in alert log I see the following.Why the db got restarted and why the clients could not notice it.Can somebody please reply why this has happened.
    Mon Jul 18 12:28:02 2011
    ARC1: terminating instance due to error 472
    Mon Jul 18 12:28:03 2011
    Errors in file /backups/p01/app/oracle/admin/plsprod/bdump/plsprod_lns1_3772752.trc:
    ORA-00472: PMON process terminated with error
    Instance terminated by ARC1, pid = 2306362
    Mon Jul 18 12:28:52 2011
    Starting ORACLE instance (normal)
    LICENSE_MAX_SESSION = 0
    LICENSE_SESSIONS_WARNING = 0
    Picked latch-free SCN scheme 3
    Autotune of undo retention is turned on.
    IMODE=BR
    ILAT =242
    LICENSE_MAX_USERS = 0
    SYS auditing is disabled
    ksdpec: called for event 13740 prior to event group initialization
    Starting up ORACLE RDBMS Version: 10.2.0.3.0.
    System parameters with non-default values:
    processes = 2000
    shared_pool_size = 536870912
    large_pool_size = 54525952
    java_pool_size = 75497472
    streams_pool_size = 54525952
    control_files = /data/plsprod/p02/control01.ctl, /data/plsprod/p03/control02.ctl, /data/plsprod/p04/control03.ctl
    db_block_size = 8192
    db_cache_size = 104857600
    compatible = 10.2.0.3
    log_archive_dest_1 = location=/data/plsprod/archredo MANDATORY REOPEN
    log_archive_dest_2 = SERVICE=talon_plsprod.travimp.com LGWR ASYNC NOAFFIRM REOPEN=60 NODELAY
    log_archive_dest_state_1 = enable
    log_archive_dest_state_2 = enable
    standby_archive_dest = /data/plsprod/archredo
    log_archive_format = redo_plsprod_%r_%t_%s.arc
    log_buffer = 6120448
    fast_start_mttr_target = 30
    undo_management = AUTO
    undo_tablespace = plsprod_undo
    undo_retention = 1500
    remote_login_passwordfile= EXCLUSIVE
    db_domain = travimp.com
    instance_name = plsprod
    service_names = plsprod.travimp.com
    local_listener = (ADDRESS=(PROTOCOL=TCP)(HOST=catalina)(PORT=1521))
    background_dump_dest = /backups/p01/app/oracle/admin/plsprod/bdump
    user_dump_dest = /backups/p01/app/oracle/admin/plsprod/udump
    core_dump_dest = /backups/p01/app/oracle/admin/plsprod/cdump
    session_max_open_files = 60
    sort_area_size = 1048576
    sort_area_retained_size = 1048576
    db_name = plsprod
    open_cursors = 100
    os_authent_prefix =
    pga_aggregate_target = 26214400
    aq_tm_processes = 0
    PMON started with pid=2, OS id=5623926
    PSP0 started with pid=3, OS id=5566632
    MMAN started with pid=4, OS id=5328992
    DBW0 started with pid=5, OS id=4362640
    DBW1 started with pid=6, OS id=4808940
    LGWR started with pid=7, OS id=4358468
    CKPT started with pid=8, OS id=4784160
    SMON started with pid=9, OS id=4354538
    RECO started with pid=10, OS id=4313420
    MMON started with pid=11, OS id=4771908
    MMNL started with pid=12, OS id=4743272
    Mon Jul 18 12:28:53 2011
    ALTER DATABASE MOUNT
    Mon Jul 18 12:28:57 2011
    Setting recovery target incarnation to 1
    Mon Jul 18 12:28:57 2011
    Successful mount of redo thread 1, with mount id 2894903685
    Mon Jul 18 12:28:57 2011
    Database mounted in Exclusive Mode
    Completed: ALTER DATABASE MOUNT
    Mon Jul 18 12:28:58 2011
    ALTER DATABASE OPEN
    Mon Jul 18 12:28:58 2011
    Beginning crash recovery of 1 threads
    parallel recovery started with 15 processes
    Mon Jul 18 12:28:58 2011
    Started redo scan
    Mon Jul 18 12:28:58 2011
    Completed redo scan
    1696 redo blocks read, 56 data blocks need recovery
    Mon Jul 18 12:28:58 2011
    Started redo application at
    Thread 1: logseq 17759, block 26354
    Mon Jul 18 12:28:58 2011
    Recovery of Online Redo Log: Thread 1 Group 1 Seq 17759 Reading mem 0
    Mem# 0: /data/plsprod/p01/redo_t1_g1_f1.log
    Mem# 1: /data/plsprod/p02/redo_t1_g1_f2.log
    Mon Jul 18 12:28:58 2011
    Completed redo application
    Mon Jul 18 12:28:58 2011
    Completed crash recovery at
    Thread 1: logseq 17759, block 28050, scn 35195499598
    56 data blocks read, 56 data blocks written, 1696 redo blocks read
    Mon Jul 18 12:28:58 2011
    LGWR: STARTING ARCH PROCESSES
    ARC0 started with pid=29, OS id=4338022
    Mon Jul 18 12:28:58 2011
    ARC0: Archival started
    ARC1: Archival started
    LGWR: STARTING ARCH PROCESSES COMPLETE
    ARC1 started with pid=30, OS id=1081522
    LNS1 started with pid=31, OS id=5537854
    Mon Jul 18 12:29:01 2011
    Thread 1 advanced to log sequence 17760
    Thread 1 opened at log sequence 17760
    Current log# 2 seq# 17760 mem# 0: /data/plsprod/p01/redo_t1_g2_f1.log
    Current log# 2 seq# 17760 mem# 1: /data/plsprod/p02/redo_t1_g2_f2.log
    Successful open of redo thread 1
    Mon Jul 18 12:29:01 2011
    ARC0: STARTING ARCH PROCESSES
    Mon Jul 18 12:29:01 2011
    ARC1: Becoming the 'no FAL' ARCH
    ARC1: Becoming the 'no SRL' ARCH
    Mon Jul 18 12:29:01 2011
    SMON: enabling cache recovery
    Mon Jul 18 12:29:02 2011
    ARC2: Archival started
    ARC0: STARTING ARCH PROCESSES COMPLETE
    ARC0: Becoming the heartbeat ARCH
    ARC2 started with pid=32, OS id=5644370
    Mon Jul 18 12:29:02 2011
    ARC0: Standby redo logfile selected for thread 1 sequence 17759 for destination LOG_ARCHIVE_DEST_2
    Mon Jul 18 12:29:02 2011
    LGWR: Setting 'active' archival for destination LOG_ARCHIVE_DEST_2
    Mon Jul 18 12:29:03 2011
    Successfully onlined Undo Tablespace 16.
    Mon Jul 18 12:29:03 2011
    SMON: enabling tx recovery
    Mon Jul 18 12:29:03 2011
    Database Characterset is US7ASCII
    replication_dependency_tracking turned off (no async multimaster replication found)
    WARNING: AQ_TM_PROCESSES is set to 0. System operation might be adversely affected.
    Completed: ALTER DATABASE OPEN
    Mon Jul 18 12:29:58 2011
    Shutting down archive processes
    Mon Jul 18 12:30:03 2011
    ARCH shutting down
    ARC2: Archival stopped

    Hi CKPT,
    Please find the output of the trace file as below.I deleted the repetitive output "Attempting to send buffer 0x111441d30 start block 91942 block count 84
    total size to send 43008 blocksize 512" as it exceeds maximum length of 30000 characters in this post.
    CATALINA_mardiprd>cat /backups/p01/app/oracle/admin/plsprod/bdump/plsprod_lns1_3772752.trc
    /backups/p01/app/oracle/admin/plsprod/bdump/plsprod_lns1_3772752.trc
    Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bit Production
    With the Partitioning, OLAP and Data Mining options
    ORACLE_HOME = /p01/app/oracle10g/product/10.2.0.3
    System name: AIX
    Node name: catalina
    Release: 3
    Version: 5
    Machine: 00044C5FD600
    Instance name: plsprod
    Redo thread mounted by this instance: 1
    Oracle process number: 16
    Unix process pid: 3772752, image: oracle@catalina (LNS1)
    *** SERVICE NAME:() 2011-07-18 09:42:11.573
    *** SESSION ID:(2189.1) 2011-07-18 09:42:11.573
    *** 2011-07-18 09:42:11.573 57516 kcrr.c
    LNS1: initializing for LGWR communication
    LNS1: connecting to KSR channel
    Success
    LNS1: subscribing to KSR channel
    Success
    *** 2011-07-18 09:42:11.574 57569 kcrr.c
    LNS1: initialized successfully ASYNC=1
    Destination is specified with ASYNC=61440
    *** 2011-07-18 09:42:12.577 71560 kcrr.c
    Sending online log thread 1 seq 17731 [logfile 3] to standby
    Opening logfile [logno 3]
    Redo shipping client performing standby login
    *** 2011-07-18 09:42:12.693 65194 kcrr.c
    Logged on to standby successfully
    Client logon and security negotiation successful!
    Destination LOG_ARCHIVE_DEST_2 is in MAXIMUM PERFORMANCE mode
    Archiving to destination talon_plsprod.travimp.com ASYNC blocks=20480
    Allocate ASYNC blocks: Previous blocks=0 New blocks=20480
    Log file opened [logno 3]
    Attempting to send buffer 0x111441d30 start block 2 block count 1
    total size to send 512 blocksize 512
    Attempting to send buffer 0x111441d30 start block 3 block count 35
    total size to send 17920 blocksize 512
    *** 2011-07-18 09:42:27.009
    Attempting to send buffer 0x111441d30 start block 38 block count 2914
    total size to send 1491968 blocksize 512
    *** 2011-07-18 09:42:50.007
    Attempting to send buffer 0x111441d30 start block 2952 block count 2
    total size to send 1024 blocksize 512
    *** 2011-07-18 09:43:01.441
    Attempting to send buffer 0x111441d30 start block 2954 block count 19286
    total size to send 9874432 blocksize 512
    Attempting to send buffer 0x111441d30 start block 22240 block count 19657
    total size to send 10064384 blocksize 512
    Attempting to send buffer 0x111441d30 start block 41897 block count 1687
    total size to send 863744 blocksize 512
    *** 2011-07-18 09:43:13.007
    Attempting to send buffer 0x111441d30 start block 43584 block count 1
    total size to send 512 blocksize 512
    Attempting to send buffer 0x111441d30 start block 43585 block count 13
    total size to send 6656 blocksize 512
    Attempting to send buffer 0x111441d30 start block 43598 block count 19008
    total size to send 9732096 blocksize 512
    *** 2011-07-18 09:43:24.265
    Attempting to send buffer 0x111441d30 start block 62606 block count 19349
    total size to send 9906688 blocksize 512
    *** 2011-07-18 09:43:25.524 72367 kcrr.c
    Detected proper completion of online log
    *** 2011-07-18 09:43:25.524 72204 kcrr.c
    Detected physical end of file
    Attempting to send buffer 0x111441d30 start block 81955 block count 10066
    total size to send 5153792 blocksize 512
    Online entry still exists for log 3 thr 1 seq 17731. Using it
    Closing thread 1 sequence 17731 logno 3 nab 92021 los 35193489640 nxs 35193490559
    *** 2011-07-18 09:43:26.686 71560 kcrr.c
    Sending online log thread 1 seq 17732 [logfile 1] to standby
    Opening logfile [logno 1]
    Destination LOG_ARCHIVE_DEST_2 is in MAXIMUM PERFORMANCE mode
    Archiving to destination talon_plsprod.travimp.com ASYNC blocks=20480
    Log file opened [logno 1]
    Attempting to send buffer 0x111441d30 start block 2 block count 20407
    total size to send 10448384 blocksize 512
    Attempting to send buffer 0x111441d30 start block 20409 block count 20349
    total size to send 10418688 blocksize 512
    Attempting to send buffer 0x111441d30 start block 40758 block count 19751
    total size to send 10112512 blocksize 512
    Attempting to send buffer 0x111441d30 start block 60509 block count 20260
    total size to send 10373120 blocksize 512
    *** 2011-07-18 09:43:33.262 72367 kcrr.c
    Detected proper completion of online log
    *** 2011-07-18 09:43:33.262 72204 kcrr.c
    Detected physical end of file
    Attempting to send buffer 0x111441d30 start block 80769 block count 11251
    total size to send 5760512 blocksize 512
    Online entry still exists for log 1 thr 1 seq 17732. Using it
    Closing thread 1 sequence 17732 logno 1 nab 92020 los 35193490559 nxs 35193490735
    *** 2011-07-18 09:43:33.541 71560 kcrr.c
    Sending online log thread 1 seq 17733 [logfile 2] to standby
    Opening logfile [logno 2]
    Destination LOG_ARCHIVE_DEST_2 is in MAXIMUM PERFORMANCE mode
    *** 2011-07-18 09:43:34.387
    Archiving to destination talon_plsprod.travimp.com ASYNC blocks=20480
    Log file opened [logno 2]
    Attempting to send buffer 0x111441d30 start block 2 block count 19192
    total size to send 9826304 blocksize 512
    Attempting to send buffer 0x111441d30 start block 19194 block count 18876
    total size to send 9664512 blocksize 512
    Attempting to send buffer 0x111441d30 start block 38070 block count 19565
    total size to send 10017280 blocksize 512
    Attempting to send buffer 0x111441d30 start block 57635 block count 19765
    total size to send 10119680 blocksize 512
    *** 2011-07-18 09:43:41.660 72367 kcrr.c
    Detected proper completion of online log
    *** 2011-07-18 09:43:41.660 72204 kcrr.c
    Detected physical end of file
    Attempting to send buffer 0x111441d30 start block 77400 block count 14627
    total size to send 7489024 blocksize 512
    Online entry still exists for log 2 thr 1 seq 17733. Using it
    Closing thread 1 sequence 17733 logno 2 nab 92027 los 35193490735 nxs 35193490800
    *** 2011-07-18 09:43:41.928 71560 kcrr.c
    Sending online log thread 1 seq 17734 [logfile 3] to standby
    Opening logfile [logno 3]
    Destination LOG_ARCHIVE_DEST_2 is in MAXIMUM PERFORMANCE mode
    Archiving to destination talon_plsprod.travimp.com ASYNC blocks=20480
    Log file opened [logno 3]
    Attempting to send buffer 0x111441d30 start block 59038 block count 18758
    total size to send 9604096 blocksize 512
    *** 2011-07-18 09:44:42.141
    *** 2011-07-18 09:44:42.141 72367 kcrr.c
    Detected proper completion of online log
    *** 2011-07-18 09:44:42.142 72204 kcrr.c
    Detected physical end of file
    total size to send 586240 blocksize 512
    Online entry still exists for log 1 thr 1 seq 17744. Using it
    Closing thread 1 sequence 17744 logno 1 nab 92029 los 35193494218 nxs 35193494529
    *** 2011-07-18 09:45:58.974 71560 kcrr.c
    Sending online log thread 1 seq 17745 [logfile 2] to standby
    Opening logfile [logno 2]
    Destination LOG_ARCHIVE_DEST_2 is in MAXIMUM PERFORMANCE mode
    Archiving to destination talon_plsprod.travimp.com ASYNC blocks=20480
    Log file opened [logno 2]
    Attempting to send buffer 0x111441d30 start block 2 block count 9013
    total size to send 4614656 blocksize 512
    Attempting to send buffer 0x111441d30 start block 9015 block count 20176
    total size to send 10330112 blocksize 512
    Attempting to send buffer 0x111441d30 start block 29191 block count 19808
    total size to send 10141696 blocksize 512
    Attempting to send buffer 0x111441d30 start block 48999 block count 18939
    total size to send 9696768 blocksize 512
    *** 2011-07-18 09:46:10.146
    Attempting to send buffer 0x111441d30 start block 67938 block count 19124
    total size to send 9791488 blocksize 512
    Attempting to send buffer 0x111441d30 start block 87062 block count 3690
    total size to send 1889280 blocksize 512
    *** 2011-07-18 09:46:12.644 72367 kcrr.c
    Detected proper completion of online log
    *** 2011-07-18 09:46:12.644 72204 kcrr.c
    Detected physical end of file
    Attempting to send buffer 0x111441d30 start block 90752 block count 1267
    total size to send 648704 blocksize 512
    Online entry still exists for log 2 thr 1 seq 17745. Using it
    Closing thread 1 sequence 17745 logno 2 nab 92019 los 35193494529 nxs 35193494940
    *** 2011-07-18 09:46:13.690 71560 kcrr.c
    Sending online log thread 1 seq 17746 [logfile 3] to standby
    Opening logfile [logno 3]
    Destination LOG_ARCHIVE_DEST_2 is in MAXIMUM PERFORMANCE mode
    Archiving to destination talon_plsprod.travimp.com ASYNC blocks=20480
    Log file opened [logno 3]
    Attempting to send buffer 0x111441d30 start block 2 block count 19533
    total size to send 10000896 blocksize 512
    Attempting to send buffer 0x111441d30 start block 19535 block count 11151
    total size to send 5709312 blocksize 512
    Attempting to send buffer 0x111441d30 start block 30686 block count 20154
    total size to send 10318848 blocksize 512
    *** 2011-07-18 09:46:20.822
    Attempting to send buffer 0x111441d30 start block 50840 block count 19083
    total size to send 9770496 blocksize 512
    Attempting to send buffer 0x111441d30 start block 69923 block count 19167
    total size to send 9813504 blocksize 512
    *** 2011-07-18 09:46:22.573 72367 kcrr.c
    Detected proper completion of online log
    *** 2011-07-18 09:46:22.573 72204 kcrr.c
    Detected physical end of file
    Attempting to send buffer 0x111441d30 start block 89090 block count 2933
    total size to send 1501696 blocksize 512
    Online entry still exists for log 3 thr 1 seq 17746. Using it
    Closing thread 1 sequence 17746 logno 3 nab 92023 los 35193494940 nxs 35193495191
    *** 2011-07-18 09:46:23.637 71560 kcrr.c
    Sending online log thread 1 seq 17747 [logfile 1] to standby
    Opening logfile [logno 1]
    Destination LOG_ARCHIVE_DEST_2 is in MAXIMUM PERFORMANCE mode
    Archiving to destination talon_plsprod.travimp.com ASYNC blocks=20480
    Log file opened [logno 1]
    Attempting to send buffer 0x111441d30 start block 2 block count 8093
    total size to send 4143616 blocksize 512
    Attempting to send buffer 0x111441d30 start block 8095 block count 20288
    total size to send 10387456 blocksize 512
    Attempting to send buffer 0x111441d30 start block 28383 block count 18967
    total size to send 9711104 blocksize 512
    *** 2011-07-18 09:46:31.007
    Attempting to send buffer 0x111441d30 start block 47350 block count 5237
    total size to send 2681344 blocksize 512
    Attempting to send buffer 0x111441d30 start block 52587 block count 20188
    total size to send 10336256 blocksize 512
    Attempting to send buffer 0x111441d30 start block 72775 block count 19167
    total size to send 9813504 blocksize 512
    *** 2011-07-18 09:46:37.430 72367 kcrr.c
    Detected proper completion of online log
    *** 2011-07-18 09:46:37.430 72204 kcrr.c
    Detected physical end of file
    Attempting to send buffer 0x111441d30 start block 91942 block count 84
    total size to send 43008 blocksize 512
    Online entry still exists for log 1 thr 1 seq 17747. Using it
    Closing thread 1 sequence 17747 logno 1 nab 92026 los 35193495191 nxs 35193495503
    *** 2011-07-18 09:46:37.501 71560 kcrr.c
    Sending online log thread 1 seq 17748 [logfile 2] to standby
    Opening logfile [logno 2]
    Destination LOG_ARCHIVE_DEST_2 is in MAXIMUM PERFORMANCE mode
    Archiving to destination talon_plsprod.travimp.com ASYNC blocks=20480
    Log file opened [logno 2]
    Attempting to send buffer 0x111441d30 start block 2 block count 2052
    total size to send 1050624 blocksize 512
    *** 2011-07-18 09:46:43.059
    total size to send 8164352 blocksize 512
    *** 2011-07-18 09:48:18.781
    *** 2011-07-18 09:48:18.781 72367 kcrr.c
    Detected proper completion of online log
    **** 2011-07-18 09:48:18.781 72204 kcrr.c*
    Detected physical end of file
    Attempting to send buffer 0x111441d30  start block 90604  block count 1416
    total size to send 724992  blocksize 512
    Online entry still exists for log 2 thr 1 seq 17754. Using it
    Closing thread 1 sequence 17754 logno 2 nab 92020 los 35193496751 nxs 35193496934
    **** 2011-07-18 09:48:19.891 71560 kcrr.c*
    *Sending online log thread 1 seq 17755 [logfile 3] to standby*
    *Opening logfile [logno 3]*
    Destination LOG_ARCHIVE_DEST_2 is in MAXIMUM PERFORMANCE mode
    Archiving to destination talon_plsprod.travimp.com ASYNC blocks=20480
    *Log file opened [logno 3]*
    Attempting to send buffer 0x111441d30 start block 2 block count 19248
    total size to send 9854976 blocksize 512
    Attempting to send buffer 0x111441d30 start block 19250 block count 20166
    total size to send 10324992 blocksize 512
    Attempting to send buffer 0x111441d30 start block 39416 block count 4955
    total size to send 2536960 blocksize 512
    Attempting to send buffer 0x111441d30 start block 44371 block count 19903
    total size to send 10190336 blocksize 512
    *** 2011-07-18 09:48:30.215
    Attempting to send buffer 0x111441d30 start block 64274 block count 19991
    total size to send 10235392 blocksize 512
    *** 2011-07-18 09:48:32.370 72367 kcrr.c
    Detected proper completion of online log
    *** 2011-07-18 09:48:32.370 72204 kcrr.c
    Detected physical end of file
    Attempting to send buffer 0x111441d30 start block 84265 block count 7754
    total size to send 3970048 blocksize 512
    Online entry still exists for log 3 thr 1 seq 17755. Using it
    Closing thread 1 sequence 17755 logno 3 nab 92019 los 35193496934 nxs 35193498387
    *** 2011-07-18 09:48:32.472 71560 kcrr.c
    Sending online log thread 1 seq 17756 [logfile 1] to standby
    Opening logfile [logno 1]
    Destination LOG_ARCHIVE_DEST_2 is in MAXIMUM PERFORMANCE mode
    Archiving to destination talon_plsprod.travimp.com ASYNC blocks=20480
    Log file opened [logno 1]
    Attempting to send buffer 0x111441d30 start block 2 block count 19214
    total size to send 9837568 blocksize 512
    *** 2011-07-18 09:48:40.267
    Attempting to send buffer 0x111441d30 start block 19216 block count 19981
    total size to send 10230272 blocksize 512
    Attempting to send buffer 0x111441d30 start block 39197 block count 19586
    total size to send 10028032 blocksize 512
    Attempting to send buffer 0x111441d30 start block 58783 block count 6569
    total size to send 3363328 blocksize 512
    Attempting to send buffer 0x111441d30 start block 65352 block count 19894
    total size to send 10185728 blocksize 512
    *** 2011-07-18 09:48:47.133 72367 kcrr.c
    Detected proper completion of online log
    *** 2011-07-18 09:48:47.133 72204 kcrr.c
    Detected physical end of file
    Attempting to send buffer 0x111441d30 start block 85246 block count 6772
    total size to send 3467264 blocksize 512
    Online entry still exists for log 1 thr 1 seq 17756. Using it
    Closing thread 1 sequence 17756 logno 1 nab 92018 los 35193498387 nxs 35193500455
    *** 2011-07-18 09:48:47.344 71560 kcrr.c
    Sending online log thread 1 seq 17757 [logfile 2] to standby
    Opening logfile [logno 2]
    Destination LOG_ARCHIVE_DEST_2 is in MAXIMUM PERFORMANCE mode
    Archiving to destination talon_plsprod.travimp.com ASYNC blocks=20480
    Log file opened [logno 2]
    *** 2011-07-18 09:48:50.302
    Attempting to send buffer 0x111441d30 start block 2 block count 18546
    total size to send 9495552 blocksize 512
    Attempting to send buffer 0x111441d30 start block 18548 block count 19821
    total size to send 10148352 blocksize 512
    Attempting to send buffer 0x111441d30 start block 38369 block count 20457
    total size to send 10473984 blocksize 512
    *** 2011-07-18 09:49:02.194
    Attempting to send buffer 0x111441d30 start block 58826 block count 20477
    total size to send 10484224 blocksize 512
    Attempting to send buffer 0x111441d30 start block 79303 block count 12397
    total size to send 6347264 blocksize 512
    *** 2011-07-18 09:49:08.443 72367 kcrr.c
    Detected proper completion of online log
    *** 2011-07-18 09:49:08.443 72204 kcrr.c
    Detected physical end of file
    Attempting to send buffer 0x111441d30 start block 91700 block count 333
    total size to send 170496 blocksize 512
    Online entry still exists for log 2 thr 1 seq 17757. Using it
    Closing thread 1 sequence 17757 logno 2 nab 92033 los 35193500455 nxs 35193504614
    *** 2011-07-18 09:49:08.549 71560 kcrr.c
    Sending online log thread 1 seq 17758 [logfile 3] to standby
    Opening logfile [logno 3]
    Destination LOG_ARCHIVE_DEST_2 is in MAXIMUM PERFORMANCE mode
    Archiving to destination talon_plsprod.travimp.com ASYNC blocks=20480
    Log file opened [logno 3]
    Attempting to send buffer 0x111441d30 start block 2 block count 19640
    total size to send 10055680 blocksize 512
    *** 2011-07-18 09:49:15.005
    Attempting to send buffer 0x111441d30 start block 19642 block count 12646
    total size to send 6474752 blocksize 512
    Attempting to send buffer 0x111441d30 start block 92000 block count 23
    total size to send 11776 blocksize 512
    *** 2011-07-18 11:30:19.515 72367 kcrr.c
    Detected proper completion of online log
    *** 2011-07-18 11:30:19.515 72204 kcrr.c
    Detected physical end of file
    Attempting to send buffer 0x111441d30 start block 92023 block count 11
    total size to send 5632 blocksize 512
    Online entry still exists for log 3 thr 1 seq 17758. Using it
    Closing thread 1 sequence 17758 logno 3 nab 92034 los 35193504614 nxs 35194291751
    *** 2011-07-18 11:30:21.546 71560 kcrr.c
    Sending online log thread 1 seq 17759 [logfile 1] to standby
    Opening logfile [logno 1]
    Destination LOG_ARCHIVE_DEST_2 is in MAXIMUM PERFORMANCE mode
    Archiving to destination talon_plsprod.travimp.com ASYNC blocks=20480
    Log file opened [logno 1]
    Attempting to send buffer 0x111441d30 start block 2 block count 24
    total size to send 12288 blocksize 512
    *** 2011-07-18 11:30:27.006
    Attempting to send buffer 0x111441d30 start block 26 block count 1
    total size to send 512 blocksize 512
    Attempting to send buffer 0x111441d30 start block 27 block count 34
    total size to send 17408 blocksize 512
    Attempting to send buffer 0x111441d30 start block 61 block count 31
    total size to send 15872 blocksize 512
    *** 2011-07-18 11:30:39.003
    Attempting to send buffer 0x111441d30 start block 92 block count 32
    total size to send 16384 blocksize 512
    Attempting to send buffer 0x111441d30 start block 124 block count 2
    total size to send 1024 blocksize 512
    Attempting to send buffer 0x111441d30 start block 126 block count 1
    total size to send 512 blocksize 512
    *** 2011-07-18 11:30:51.000
    Attempting to send buffer 0x111441d30 start block 127 block count 1
    total size to send 512 blocksize 512
    Attempting to send buffer 0x111441d30 start block 128 block count 30
    total size to send 15360 blocksize 512
    Attempting to send buffer 0x111441d30 start block 158 block count 88
    total size to send 45056 blocksize 512
    Attempting to send buffer 0x111441d30 start block 246 block count 3
    total size to send 1536 blocksize 512
    *** 2011-07-18 11:31:03.002
    Attempting to send buffer 0x111441d30 start block 249 block count 32
    total size to send 16384 blocksize 512
    Attempting to send buffer 0x111441d30 start block 27416 block count 4
    total size to send 2048 blocksize 512
    Attempting to send buffer 0x111441d30 start block 27420 block count 14
    total size to send 7168 blocksize 512
    Attempting to send buffer 0x111441d30 start block 27434 block count 54
    total size to send 27648 blocksize 512
    Attempting to send buffer 0x111441d30 start block 27488 block count 10
    total size to send 5120 blocksize 512
    *** 2011-07-18 12:27:01.000
    Attempting to send buffer 0x111441d30 start block 27498 block count 78
    total size to send 39936 blocksize 512
    Attempting to send buffer 0x111441d30 start block 27576 block count 2
    total size to send 1024 blocksize 512
    Attempting to send buffer 0x111441d30 start block 27578 block count 56
    total size to send 28672 blocksize 512
    Attempting to send buffer 0x111441d30 start block 27634 block count 60
    total size to send 30720 blocksize 512
    *** 2011-07-18 12:27:14.006
    Attempting to send buffer 0x111441d30 start block 27694 block count 7
    total size to send 3584 blocksize 512
    Attempting to send buffer 0x111441d30 start block 27701 block count 1
    total size to send 512 blocksize 512
    Attempting to send buffer 0x111441d30 start block 27702 block count 106
    total size to send 54272 blocksize 512
    *** 2011-07-18 12:27:28.001
    Attempting to send buffer 0x111441d30 start block 27808 block count 2
    total size to send 1024 blocksize 512
    Attempting to send buffer 0x111441d30 start block 27810 block count 37
    total size to send 18944 blocksize 512
    Attempting to send buffer 0x111441d30 start block 27847 block count 2
    total size to send 1024 blocksize 512
    *** 2011-07-18 12:27:42.002
    Attempting to send buffer 0x111441d30 start block 27849 block count 78
    total size to send 39936 blocksize 512
    Attempting to send buffer 0x111441d30 start block 27927 block count 43
    total size to send 22016 blocksize 512
    *** 2011-07-18 12:27:52.001
    Attempting to send buffer 0x111441d30 start block 27970 block count 1
    total size to send 512 blocksize 512
    Attempting to send buffer 0x111441d30 start block 27971 block count 2
    total size to send 1024 blocksize 512
    Attempting to send buffer 0x111441d30 start block 27973 block count 38
    total size to send 19456 blocksize 512
    *** 2011-07-18 12:28:03.007
    ORA-00472: PMON  process terminated with error
    Closing Redo Read Context
    Redo Push Server: Freeing ASYNC PGA buffer
    LNS1: Doing a channel reset for next time around...
    Please help me out.

  • Error 472 in job queue process + ORA-472: PMON  process terminated with err

    Hi,
    I've an Oracle Database 11g Release 11.1.0.6.0 - 64bit Production With the Real Application Clusters option.
    Yesterday instance on node1 crashed.
    This is the extract of alert log:
    Errors in file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_pmon_9297.trc (incident=195198):
    ORA-00600: internal error code, arguments: [17183], [0x14E251C38], [], [], [], [], [], []
    Incident details in: /u01/app/oracle/diag/rdbms/evodb/EVODB1/incident/incdir_195198/EVODB1_pmon_9297_i195198.trc
    Sat Jan 22 16:24:47 2011
    Trace dumping is performing id=[cdmp_20110122162447]
    Sat Jan 22 16:24:50 2011
    Sweep Incident[195198]: completed
    Sat Jan 22 16:25:00 2011
    Errors in file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_pmon_9297.trc (incident=195199):
    ORA-00600: internal error code, arguments: [17183], [0x14E251C38], [], [], [], [], [], []
    Incident details in: /u01/app/oracle/diag/rdbms/evodb/EVODB1/incident/incdir_195199/EVODB1_pmon_9297_i195199.trc
    Sat Jan 22 16:25:10 2011
    Trace dumping is performing id=[cdmp_20110122162510]
    Sat Jan 22 16:25:12 2011
    Errors in file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_pmon_9297.trc (incident=195200):
    ORA-00600: internal error code, arguments: [17183], [0x14E251C38], [], [], [], [], [], []
    Incident details in: /u01/app/oracle/diag/rdbms/evodb/EVODB1/incident/incdir_195200/EVODB1_pmon_9297_i195200.trc
    Sat Jan 22 16:25:23 2011
    DBRM (ospid: 9310): terminating the instance due to error 472
    Sat Jan 22 16:25:23 2011
    System state dump is made for local instance
    System State dumped to trace file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_diag_16320.trc
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (3724_47445977853072)
    Sat Jan 22 16:25:23 2011
    SUCCESS: diskgroup ARCHIVELOG was dismounted
    Sat Jan 22 16:14:22 2011
    DDE: Problem Key 'ORA 600 [17183]' was completely flood controlled (0x6)
    Further messages for this problem key will be suppressed for up to 10 minutes
    Sat Jan 22 16:15:56 2011
    Thread 1 advanced to log sequence 171322
    Current log# 18 seq# 171322 mem# 0: +ONLINELOG/evodb/onlinelog/group_18.264.729348055
    Sat Jan 22 16:15:56 2011
    SUCCESS: diskgroup ARCHIVELOG was mounted
    Sat Jan 22 16:16:15 2011
    SUCCESS: diskgroup ARCHIVELOG was dismounted
    Sat Jan 22 16:20:52 2011
    Thread 1 advanced to log sequence 171323
    Current log# 12 seq# 171323 mem# 0: +ONLINELOG/evodb/onlinelog/group_12.267.729333861
    Sat Jan 22 16:20:53 2011
    SUCCESS: diskgroup ARCHIVELOG was mounted
    Sat Jan 22 16:21:14 2011
    SUCCESS: diskgroup ARCHIVELOG was dismounted
    Sat Jan 22 16:24:23 2011
    DDE: Problem Key 'ORA 600 [17183]' was completely flood controlled (0x2)
    Further messages for this problem key will be suppressed for up to 10 minutes
    Sat Jan 22 16:24:38 2011
    Errors in file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_pmon_9297.trc (incident=195198):
    ORA-00600: internal error code, arguments: [17183], [0x14E251C38], [], [], [], [], [], []
    Incident details in: /u01/app/oracle/diag/rdbms/evodb/EVODB1/incident/incdir_195198/EVODB1_pmon_9297_i195198.trc
    Sat Jan 22 16:24:47 2011
    Trace dumping is performing id=[cdmp_20110122162447]
    Sat Jan 22 16:24:50 2011
    Sweep Incident[195198]: completed
    Sat Jan 22 16:25:00 2011
    Errors in file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_pmon_9297.trc (incident=195199):
    ORA-00600: internal error code, arguments: [17183], [0x14E251C38], [], [], [], [], [], []
    Incident details in: /u01/app/oracle/diag/rdbms/evodb/EVODB1/incident/incdir_195199/EVODB1_pmon_9297_i195199.trc
    Sat Jan 22 16:25:10 2011
    Trace dumping is performing id=[cdmp_20110122162510]
    Sat Jan 22 16:25:12 2011
    Errors in file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_pmon_9297.trc (incident=195200):
    ORA-00600: internal error code, arguments: [17183], [0x14E251C38], [], [], [], [], [], []
    Incident details in: /u01/app/oracle/diag/rdbms/evodb/EVODB1/incident/incdir_195200/EVODB1_pmon_9297_i195200.trc
    Sat Jan 22 16:25:23 2011
    DBRM (ospid: 9310): terminating the instance due to error 472
    Sat Jan 22 16:25:23 2011
    System state dump is made for local instance
    System State dumped to trace file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_diag_16320.trc
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (3724_47445977853072)
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (3650_47673206152336)
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (19709_47962498952336)
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (3715_47239602621584)
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (3718_47120460606608)
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (3646_47899813694608)
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (3671_47700480982160)
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (3637_47223057443984)
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (3695_47109627223184)
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (3679_47262512999568)
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (3733_46976527121552)
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (3667_47474016554128)
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (3669_47585103858832)
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (3658_47992462539920)
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (3654_47736794324112)
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (3686_47295241862288)
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (3706_47055349939344)
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (3630_47331091332240)
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (3713_47194150415504)
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (3726_47289949199504)
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (3677_47842172337296)
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (19131_47557370236048)
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (27091_47096969530512)
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (3688_47422639070352)
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (3652_47944422886544)
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (3648_47426149817488)
    Sat Jan 22 16:25:24 2011
    Errors in file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_j005_9360.trc:
    ORA-00472: PMON  process terminated with error
    ORA-1092 : opidrv aborting process J005 ospid (9360_47420959304848)
    Sat Jan 22 16:25:24 2011
    ORA-1092 : opiodr aborting process unknown ospid (18393_47566684101776)
    Sat Jan 22 16:25:24 2011
    Errors in file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_j003_7054.trc:
    ORA-00472: PMON process terminated with error
    ORA-1092 : opidrv aborting process J003 ospid (7054_47244451724432)
    Sat Jan 22 16:25:24 2011
    ORA-1092 : opiodr aborting process unknown ospid (19719_47840732109968)
    Sat Jan 22 16:25:24 2011
    ORA-1092 : opiodr aborting process unknown ospid (22184_47575322619024)
    Sat Jan 22 16:25:24 2011
    ORA-1092 : opiodr aborting process unknown ospid (19721_47679094451344)
    Sat Jan 22 16:25:24 2011
    ORA-1092 : opiodr aborting process unknown ospid (30170_47106572815504)
    Sat Jan 22 16:25:24 2011
    ORA-472 : opidrv aborting process S000 ospid (9367_46990649617552)
    Sat Jan 22 16:25:24 2011
    Errors in file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_j002_7052.trc:
    ORA-00472: PMON process terminated with error
    ORA-1092 : opidrv aborting process J002 ospid (7052_47411695402128)
    Sat Jan 22 16:25:24 2011
    ORA-1092 : opiodr aborting process unknown ospid (18493_47469336738960)
    Sat Jan 22 16:25:24 2011
    Errors in file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_j006_10011.trc:
    ORA-00472: PMON process terminated with error
    ORA-1092 : opidrv aborting process J006 ospid (10011_47836408102032)
    Sat Jan 22 16:25:24 2011
    ORA-1092 : opiodr aborting process unknown ospid (19714_47958179912848)
    Sat Jan 22 16:25:24 2011
    Errors in file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_j007_10289.trc:
    ORA-00472: PMON process terminated with error
    ORA-1092 : opidrv aborting process J007 ospid (10289_47356480347280)
    Sat Jan 22 16:25:24 2011
    ORA-1092 : opiodr aborting process unknown ospid (9076_47765392362640)
    Sat Jan 22 16:25:24 2011
    ORA-1092 : opitsk aborting process
    Sat Jan 22 16:25:25 2011
    ORA-1092 : opiodr aborting process unknown ospid (22167_47949919177872)
    Sat Jan 22 16:25:25 2011
    ORA-1092 : opiodr aborting process unknown ospid (24320_47408077532304)
    Sat Jan 22 16:25:25 2011
    ORA-1092 : opiodr aborting process unknown ospid (22130_47998542101648)
    Sat Jan 22 16:25:25 2011
    ORA-472 : opidrv aborting process N000 ospid (14774_47466200525968)
    Sat Jan 22 16:25:25 2011
    Errors in file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_q000_14701.trc:
    ORA-00472: PMON process terminated with error
    ORA-1092 : opidrv aborting process Q000 ospid (14701_47693688122512)
    Sat Jan 22 16:25:25 2011
    ORA-1092 : opiodr aborting process unknown ospid (18386_47900707175568)
    Sat Jan 22 16:25:25 2011
    ORA-1092 : opiodr aborting process unknown ospid (27553_46921398619280)
    Sat Jan 22 16:25:26 2011
    Errors in file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_j001_7050.trc:
    ORA-00472: PMON process terminated with error
    ORA-1092 : opidrv aborting process J001 ospid (7050_47290016251024)
    Sat Jan 22 16:25:26 2011
    Errors in file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_o001_5297.trc:
    ORA-00472: PMON process terminated with error
    ORA-1092 : opidrv aborting process O001 ospid (5297_47702958673040)
    Sat Jan 22 16:25:26 2011
    Errors in file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_o000_5277.trc:
    ORA-00472: PMON process terminated with error
    ORA-1092 : opidrv aborting process O000 ospid (5277_47599668255888)
    Sat Jan 22 16:25:26 2011
    ORA-1092 : opiodr aborting process unknown ospid (19745_47560156560528)
    Sat Jan 22 16:25:26 2011
    ORA-1092 : opiodr aborting process unknown ospid (27044_47388543339664)
    Sat Jan 22 16:25:26 2011
    Errors in file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_j004_7061.trc:
    ORA-00472: PMON process terminated with error
    ORA-1092 : opidrv aborting process J004 ospid (7061_47341656689808)
    Sat Jan 22 16:25:27 2011
    ORA-1092 : opiodr aborting process unknown ospid (14431_47662297960112)
    Sat Jan 22 16:25:27 2011
    ORA-1092 : opiodr aborting process unknown ospid (14433_47826608635568)
    Sat Jan 22 16:25:31 2011
    ORA-1092 : opitsk aborting process
    Sat Jan 22 16:25:32 2011
    License high water mark = 269
    Instance terminated by DBRM, pid = 9310
    USER (ospid: 10539): terminating the instance
    Instance terminated by USER, pid = 10539
    Sat Jan 22 16:25:49 2011
    Starting ORACLE instance (normal)
    This is the trace /u01/app/oracle/diag/rdbms/evodb/EVODB1/incident/incdir_195200/EVODB1_pmon_9297_i195200.trc
    (the one regarding the error DBRM (ospid: 9310): terminating the instance due to error 472)
    Oracle Database 11g Release 11.1.0.6.0 - 64bit Production
    With the Real Application Clusters option
    ORACLE_HOME = /u01/app/oracle/product/11.1.0/db1
    System name:    Linux
    Node name:      dcsrv-evodb01
    Release:        2.6.18-164.10.1.el5
    Version:        #1 SMP Wed Dec 30 18:35:28 EST 2009
    Machine:        x86_64
    Instance name: EVODB1
    Redo thread mounted by this instance: 1
    Oracle process number: 2
    Unix process pid: 9297, image: oracle@dcsrv-evodb01 (PMON)
    *** 2011-01-22 16:25:12.984
    *** SESSION ID:(884.1) 2011-01-22 16:25:12.984
    *** CLIENT ID:() 2011-01-22 16:25:12.984
    *** SERVICE NAME:(SYS$BACKGROUND) 2011-01-22 16:25:12.984
    *** MODULE NAME:() 2011-01-22 16:25:12.984
    *** ACTION NAME:() 2011-01-22 16:25:12.984
    Dump continued from file: /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_pmon_9297.trc
    ORA-00600: internal error code, arguments: [17183], [0x14E251C38], [], [], [], [], [], []
    +========= Dump for incident 195200 (ORA 600 [17183]) ========+
    ----- Beginning of Customized Incident Dump(s) -----
    And this is the trace /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_j005_9360.trc
    (the one regarding the error ORA-00472: PMON process terminated with error)
    Trace file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_j005_9360.trc
    Oracle Database 11g Release 11.1.0.6.0 - 64bit Production
    With the Real Application Clusters option
    ORACLE_HOME = /u01/app/oracle/product/11.1.0/db1
    System name:    Linux
    Node name:      dcsrv-evodb01
    Release:        2.6.18-164.10.1.el5
    Version:        #1 SMP Wed Dec 30 18:35:28 EST 2009
    Machine:        x86_64
    Instance name: EVODB1
    Redo thread mounted by this instance: 1
    Oracle process number: 95
    Unix process pid: 9360, image: oracle@dcsrv-evodb01 (J005)
    *** 2011-01-22 16:25:24.057
    *** SESSION ID:(649.45863) 2011-01-22 16:25:24.057
    *** CLIENT ID:() 2011-01-22 16:25:24.057
    *** SERVICE NAME:(SYS$USERS) 2011-01-22 16:25:24.057
    *** MODULE NAME:() 2011-01-22 16:25:24.057
    *** ACTION NAME:() 2011-01-22 16:25:24.057
    error 472 in job queue process
    ORA-00472: PMON  process terminated with error
    My questions are:
    - how can I discover the nature of the ORA-00600: internal error code, arguments: [17183], [0x14E251C38], [], [], [], [], [], []?
    - what does error 472 in job queue proces mean?
    - is it normal the the error 472 in job queue proces crash the instance?
    Thanks in advanca for any kind of help!
    Samuel

    >
    how can I discover the nature of the ORA-00600: internal error code, arguments: [17183], [0x14E251C38], [], [], [], [], [], []?
    >
    ORA-00600 should be reported to oracle. Only they know the meaning.
    Regards,
    - wiZ

  • ORA-00471: DBWR process terminated with error when running mrua.pl

    we are upgrading to Portal 10.1.4 from 10.2.0.2.
    Right in the middle of the upgrade scripts running get ORA-00471: DBWR process terminated with error...
    Any suggestion?
    TIA

    sorry, it is me agin:
    I forgot:
    it must a trace file produced by PMON because it was PMON who killed the instance.
    These background processes function as watchdogs for each other. If one finds that one of the others is missing it will kill the instance.
    Regards,
    Lutz

  • ORA-00205 error leading to ORA-00470 LGWR process terminated with error.

    Hello Gurus,
    One of our database had gone down, I tried to bring it up. When tried to do so it gave follwoing errot ORA-00205 error in identifying control file, check alert log ..... When I checked alter log entried I found a new error ORA-00470 LGWR process terminated with eror.
    All my control files are intact where supposed to be, so are all redo log files. Please could any would help parity at my end.
    In need for restoring my database at the earliest.
    Regards
    Shiva

    Could you please post the full alert.log contents when the database starts up ?
    Are you really sure that all your control files are OK ?
    ORA-00205 error in identifying controlfile, check alert log for more info
    Cause: The system could not find a control file of the specified name and size.
    Action: One of the following:
    Check that the proper control file name is referenced in the CONTROL_FILES initialization parameter in the initialization parameter file and try again.
    When using mirrored control files, that is, more than one control file is referenced in the initialization parameter file, remove the control file name listed in the message from the initialization parameter file and restart the instance. If the message does not recur, remove the problem control file from the initialization parameter file and create another copy of the control file with a new file name in the initialization parameter file.

  • Instance crashing with ORA-00469: CKPT process terminated with error

    Hi,
    Instance crashing with ORA-00469: CKPT process terminated with error on 10.2.0.4. Any possibe causes?
    Thanks,
    Madhav

    I did check alert log and trace files. I just find this in alert log:
    ORA-00469: CKPT process terminated with error
    Trace file:
    *** 2009-07-29 02:22:06.840
    *** SERVICE NAME:(SYS$BACKGROUND) 2009-07-29 02:22:06.839
    *** SESSION ID:(275.1) 2009-07-29 02:22:06.839
    error 469 detected in background process
    ORA-00469: CKPT process terminated with error

  • ORA-00488: RBAL process terminated with error

    hi friends,
    OS: Enterprise Linux - 4 32bit, DB: oracle database 10g (10.2.0.1)
    I'm working around RAC for study purpose using VMWare server 1.0.4..
    everythings goes fine till CLUSTER (10.2.0.1) installation, oracle database software installed without any error. but when i'm trying to create database using DBCA, in ASM instance creation. I got the following error and database installation stops
    ORA-00488: RBAL process terminated with error
    can anyone tell me what doese cause it.?
    how to get around with this error..?
    Thanks in advance.
    Regards

    VMware is an unsupported environment. Even if you get this working you will be unable to obtain support for it. Give serious consideration to using supported configurations.

  • ORA-00488: RBAL process terminated with error causing ASM instance to crash

    Hi Freinds,
    I am manging a three node ASM cluster on 10g(10.2.0.2).Yesterday I observed one of the ASM instance got crashed .On reviewing the Alert log of crashed ASM instance I found
    "*ORA-00488: RBAL process terminated with error*"
    and subsequently
    "*PMON: terminating instance due to error 488*".
    I could restart the crashed instance without an issues but I want to know what kind of event could lead to such scenario.
    I searched net and metalink but didn't found any helpful information.
    Did any of you faced similar issues and do you have understanding why and when it happens.
    Thanks in Advance
    Arindam.

    Hi,
    There is bug reported on such issue, please check further on metalink if you have support.
    Dilip Patel.

  • ORA-00471: DBWR process terminated with error

    Dear all,
    I got this error in alert log and database got down.
    ORA-00471: DBWR process terminated with error
    Wed May 24 19:17:37 2006
    PMON: terminating instance due to error 471
    Instance terminated by PMON, pid = 11642
    Wed May 24 19:28:15 2006
    I am having 10g R1 on LINUX.
    To find cause of error.I read somewhere that I should check sid_dbwr_*.trc file.
    But there is no file bdump related to DBWR.
    According to me there must be atleast one trace file of DBWR in bdump.
    Can you tell me why there is no trace file of DBWR in bdump and how can I ensure that i find trace files of dbwr in bdump.
    Waiting for your reply,
    Regards,
    Amit.
    Message was edited by:
    Amit Singla

    sorry, it is me agin:
    I forgot:
    it must a trace file produced by PMON because it was PMON who killed the instance.
    These background processes function as watchdogs for each other. If one finds that one of the others is missing it will kill the instance.
    Regards,
    Lutz

  • ORA-00470: LGWR process terminated with error

    Hi,
    Often we are getting the following error on our 10g instance.
    ORA-00470: LGWR process terminated with error
    Mon Sep 6 22:47:53 2010
    PMON: terminating instance due to error 470
    Instance terminated by PMON, pid = 16986
    And checked in /var/log/messages showing the following
    kernel: Out of Memory: Killed process 16995 (oracle)
    How can I solve this?
    Thanks,
    KAvitha

    Hi,
    I am using 10g database on Red Hat Enterprise Linux AS release 4 (Nahant Update 4).
    The alert log showing the following information.
    Mon Sep 6 23:09:45 2010
    Completed: ALTER DATABASE OPEN
    Mon Sep 6 23:17:30 2010
    Errors in file /d04/u02/oracle/admin/cidb/bdump/cidb_pmon_19969.trc:
    ORA-00470: LGWR process terminated with error
    Mon Sep 6 23:17:30 2010
    PMON: terminating instance due to error 470
    Termination issued to instance processes. Waiting for the processes to exit
    Mon Sep 6 23:17:44 2010
    Instance termination failed to kill one or more processes
    Instance terminated by PMON, pid = 19969
    And also check cidb_pmon_19969.trc file
    /d04/u02/oracle/admin/cidb/bdump/cidb_pmon_19969.trc
    Oracle Database 10g Enterprise Edition Release 10.2.0.2.0 - Production
    With the Partitioning, OLAP and Data Mining options
    ORACLE_HOME = /d04/u02/oracle/cidb
    System name: Linux
    Node name: sflxdb07.donerus.com
    Release: 2.6.9-34.ELsmp
    Version: #1 SMP Fri Feb 24 16:54:53 EST 2006
    Machine: i686
    Instance name: cidb
    Redo thread mounted by this instance: 1
    Oracle process number: 2
    Unix process pid: 19969, image: [email protected] (PMON)
    *** 2010-09-06 23:17:30.676
    *** SERVICE NAME:(SYS$BACKGROUND) 2010-09-06 23:17:30.530
    *** SESSION ID:(170.1) 2010-09-06 23:17:30.530
    Background process LGWR found dead
    Oracle pid = 6
    OS pid (from detached process) = 19977
    OS pid (from process state) = 19977
    dtp = 0x2000bab8, proc = 0x8b2b13fc
    Dump of memory from 0x2000BAB8 to 0x2000BAE4
    2000BAB0 00000074 8B2B13FC [t.....+.]
    2000BAC0 00000000 00000000 5257474C 00000200 [........LGWR....]
    2000BAD0 00004E09 113B6215 00000001 2AD428DC [.N...b;......(.*]
    2000BAE0 00040001 [....]
    Dump of memory from 0x8B2B13FC to 0x8B2B19B4
    8B2B13F0 00000102 [....]
    8B2B1400 00000000 00000000 00000000 00000000 [................]
    8B2B1410 89555C60 8B82DB2C 8B3C2078 8B82CBEC [`\U.,...x <.....]
    8B2B1420 00000000 8B82CC50 8B82CC50 8B82DB20 [....P...P... ...]
    8B2B1430 00000601 8B3AE848 8B3C2078 00000006 [....H.:.x <.....]
    8B2B1440 00000000 00000000 00000000 8B4D2F48 [............H/M.]
    8B2B1450 8B4D2FE4 00000000 00000000 00000000 [./M.............]
    8B2B1460 00000000 00000000 00000000 00000000 [................]
    Repeat 2 times
    8B2B1490 00000000 00000000 00030000 00000000 [................]
    8B2B14A0 00030000 00000000 00000000 00000000 [................]
    Repeat 1 times
    8B2B17B0 00000000 00000000 00000014 4E4B4E55 [............UNKN]
    8B2B17C0 004E574F 00000000 00000000 00000000 [OWN.............]
    8B2B17D0 00000000 00000000 00000000 00000008 [................]
    8B2B17E0 37393931 00000037 00000000 00000000 [19977...........]
    8B2B17F0 00000000 00000000 00000005 6361726F [............orac]
    8B2B19B0 00000006 [....]
    error 470 detected in background process
    ORA-00470: LGWR process terminated with error
    *** 2010-09-06 23:17:44.081
    Instance termination failed to kill one or more processes
    ksuitm_check: OS PID=20017 is still alive
    *** 2010-09-06 23:17:44.083
    Dumping diagnostic information for [email protected] (J002):
    OS pid = 20017
    loadavg : 63.67 22.12 10.45
    memory info: free memory = 0.00M
    swap info: free = 0.00M alloc = 0.00M total = 0.00M
    F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
    ERROR: process 20017 is not alive
    *** 2010-09-06 23:17:44.814
    Thanks,
    Kavitha

  • ORA-484 LMS* process terminated with error

    Instance is crashing frequently with the ORA-484 LMS* process terminated with error. Oracle RAC 10.2.0.3 on HP_UX 11i on ASM. Please advise

    The problem got solved after installing an interim patch p5375583_10203_HP64

  • ORA-00480: LCK* process terminated with error

    RA-00480: LCK* process terminated with error very frequently getting this said error in our oracle 10g RAC database.
    We have Node1 / Node2,
    Node1 : we have scheduled jobs running & in addition to that end user access the database.
    Node2: 24*7 application process the jobs runnning.
    your suggestion / comments will be highly appreciated.
    Regards,
    Mansoor Ali

    System Information:
    CPU info:
    4 Intel(R) Itanium 2 9000 series processors (1.59 GHz, 24 MB)
    532 MT/s bus, CPU version C2
    8 logical processors (2 per socket)
    Memory: 32736 MB (31.97 GB)
    Firmware info:
    Firmware revision: 02.03
    FP SWA driver revision: 1.18
    IPMI is supported on this system.
    BMC firmware revision: 5.14
    Platform info:
    Model: "ia64 hp server rx6600"
    Machine ID number: 6737643a-4da1-11dc-ab2c-d7eddf21d43f
    Machine serial number: DEH472820A
    OS info:
    Nodename: Node2
    Release: HP-UX B.11.31
    Version: U (unlimited-user license)
    Machine: ia64
    ID Number: 1731683386
    vmunix releaseversion:
    @(#) $Revision: vmunix: B.11.31_LR FLAVOR=perf
    Alert log details
    System State dumped to trace file /oracle/product/admin/tap3plus/bdump/tap3plus2_diag_27593.trc
    Mon Jul 5 20:47:47 2010
    Errors in file /oracle/product/admin/tap3plus/bdump/tap3plus2_lmd0_27603.trc:
    ORA-00480: LCK* process terminated with error
    Mon Jul 5 20:47:47 2010
    Errors in file /oracle/product/admin/tap3plus/bdump/tap3plus2_lms0_27616.trc:
    ORA-00480: LCK* process terminated with error
    Mon Jul 5 20:47:47 2010
    Errors in file /oracle/product/admin/tap3plus/bdump/tap3plus2_lmon_27601.trc:
    ORA-00480: LCK* process terminated with error
    Mon Jul 5 20:47:47 2010
    Errors in file /oracle/product/admin/tap3plus/bdump/tap3plus2_lms1_27634.trc:
    ORA-00480: LCK* process terminated with error
    Mon Jul 5 20:47:49 2010

  • ORA-00490: PSP process terminated with error

    Solaris 10
    Oracle 10G R 2
    We get this error and db goes down. Till warm start.
    There is not much information on PSP0 process on the net.
    Any ideas?

    Not much on Metalink either. PSP is the Process SPawner. It is responsible for spawning and managing the Oracle process (Note: 331000.1).
    Are there any errors or traces generated in the Alert log that occur before the instance is terminated by PSP?

  • Ora-00472 PMON process terminated with error on production system

    Hi,
    I am getting the above error when I shutdown the database after a power outage. Alert log file shows " ora-00130 Invalid listener address when I bring up the database...." Nothing has changed between prior to the power outage and after that.
    RDBMS 9i.
    Help please!!
    A/A

    This is what the listener status shows
    Alias                     FINPROD
    Version                   TNSLSNR for HPUX: Version 9.2.0.6.0 - Production
    Start Date                26-JUL-2010 13:45:15
    Uptime                    0 days 2 hr. 9 min. 22 sec
    Trace Level               off
    Security                  OFF
    SNMP                      OFF
    Listener Parameter File   /d01/finprod/product/9i/network/admin/listener.ora
    Listener Log File         /d01/finprod/product/9i/network/admin/finprod.log
    Listening Endpoints Summary...
      (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROCFINPROD)))
      (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=hp3.nfpc.com)(PORT=1521)))
    Services Summary...
    Service "FINPROD" has 1 instance(s).
      Instance "FINPROD", status UNKNOWN, has 1 handler(s) for this service...
    Service "FINPROD.nfpc.com" has 1 instance(s).
      Instance "FINPROD", status READY, has 1 handler(s) for this service...
    Service "PLSExtProc" has 1 instance(s).
      Instance "PLSExtProc", status UNKNOWN, has 1 handler(s) for this service...
    The command completed successfully

Maybe you are looking for

  • Day 5 of uploading

    And still have over 2000 songs to go. I had a ton of unmatched songs, but is it really acceptable that it should take over a week (or more!) to upload everything I have? I have good bandwidth here, so that's not the problem. It's really not ok.

  • More than 2 windows display issue

    Hello, I have an application that displays different windows... In a specific situation, it seems that I cannot pile up more than 2 windows. Here are some details: I have 3 windows (W1, W2, W3) All the windows are window_style=document, and made of a

  • HT4236 deleted photos when syncing Ipod touch

    My son attempted to sync his Itouch to his macbook pro and lost all pictures.  Apple store stated "..they were gone because the ipod synced before downloading."  I cannot believe that is the answer.  In researching I see there there is some software

  • Just a question about a song!!! HELP! CAN'T FIND IT!

    I was wondering if anyone could help me find a song? It's the one that is playing in the new ipod commerical; the one where they are salsa-ing(I know that's not a word). I can't seem to find it on itunes. I know at some point they had a playlist of a

  • HT201656 I have an iPhone 4S and I have 13.3GB. How many GB is my phone?

    I was thinking maybe 8GB or 16GB?