ORA-00480: LCK* process terminated with error

RA-00480: LCK* process terminated with error very frequently getting this said error in our oracle 10g RAC database.
We have Node1 / Node2,
Node1 : we have scheduled jobs running & in addition to that end user access the database.
Node2: 24*7 application process the jobs runnning.
your suggestion / comments will be highly appreciated.
Regards,
Mansoor Ali

System Information:
CPU info:
4 Intel(R) Itanium 2 9000 series processors (1.59 GHz, 24 MB)
532 MT/s bus, CPU version C2
8 logical processors (2 per socket)
Memory: 32736 MB (31.97 GB)
Firmware info:
Firmware revision: 02.03
FP SWA driver revision: 1.18
IPMI is supported on this system.
BMC firmware revision: 5.14
Platform info:
Model: "ia64 hp server rx6600"
Machine ID number: 6737643a-4da1-11dc-ab2c-d7eddf21d43f
Machine serial number: DEH472820A
OS info:
Nodename: Node2
Release: HP-UX B.11.31
Version: U (unlimited-user license)
Machine: ia64
ID Number: 1731683386
vmunix releaseversion:
@(#) $Revision: vmunix: B.11.31_LR FLAVOR=perf
Alert log details
System State dumped to trace file /oracle/product/admin/tap3plus/bdump/tap3plus2_diag_27593.trc
Mon Jul 5 20:47:47 2010
Errors in file /oracle/product/admin/tap3plus/bdump/tap3plus2_lmd0_27603.trc:
ORA-00480: LCK* process terminated with error
Mon Jul 5 20:47:47 2010
Errors in file /oracle/product/admin/tap3plus/bdump/tap3plus2_lms0_27616.trc:
ORA-00480: LCK* process terminated with error
Mon Jul 5 20:47:47 2010
Errors in file /oracle/product/admin/tap3plus/bdump/tap3plus2_lmon_27601.trc:
ORA-00480: LCK* process terminated with error
Mon Jul 5 20:47:47 2010
Errors in file /oracle/product/admin/tap3plus/bdump/tap3plus2_lms1_27634.trc:
ORA-00480: LCK* process terminated with error
Mon Jul 5 20:47:49 2010

Similar Messages

  • ORA-00472: PMON  process terminated with error

    Hi,
    I got this error in alert log.But the clients didn't complain they had a downtime.But in alert log I see the following.Why the db got restarted and why the clients could not notice it.Can somebody please reply why this has happened.
    Mon Jul 18 12:28:02 2011
    ARC1: terminating instance due to error 472
    Mon Jul 18 12:28:03 2011
    Errors in file /backups/p01/app/oracle/admin/plsprod/bdump/plsprod_lns1_3772752.trc:
    ORA-00472: PMON process terminated with error
    Instance terminated by ARC1, pid = 2306362
    Mon Jul 18 12:28:52 2011
    Starting ORACLE instance (normal)
    LICENSE_MAX_SESSION = 0
    LICENSE_SESSIONS_WARNING = 0
    Picked latch-free SCN scheme 3
    Autotune of undo retention is turned on.
    IMODE=BR
    ILAT =242
    LICENSE_MAX_USERS = 0
    SYS auditing is disabled
    ksdpec: called for event 13740 prior to event group initialization
    Starting up ORACLE RDBMS Version: 10.2.0.3.0.
    System parameters with non-default values:
    processes = 2000
    shared_pool_size = 536870912
    large_pool_size = 54525952
    java_pool_size = 75497472
    streams_pool_size = 54525952
    control_files = /data/plsprod/p02/control01.ctl, /data/plsprod/p03/control02.ctl, /data/plsprod/p04/control03.ctl
    db_block_size = 8192
    db_cache_size = 104857600
    compatible = 10.2.0.3
    log_archive_dest_1 = location=/data/plsprod/archredo MANDATORY REOPEN
    log_archive_dest_2 = SERVICE=talon_plsprod.travimp.com LGWR ASYNC NOAFFIRM REOPEN=60 NODELAY
    log_archive_dest_state_1 = enable
    log_archive_dest_state_2 = enable
    standby_archive_dest = /data/plsprod/archredo
    log_archive_format = redo_plsprod_%r_%t_%s.arc
    log_buffer = 6120448
    fast_start_mttr_target = 30
    undo_management = AUTO
    undo_tablespace = plsprod_undo
    undo_retention = 1500
    remote_login_passwordfile= EXCLUSIVE
    db_domain = travimp.com
    instance_name = plsprod
    service_names = plsprod.travimp.com
    local_listener = (ADDRESS=(PROTOCOL=TCP)(HOST=catalina)(PORT=1521))
    background_dump_dest = /backups/p01/app/oracle/admin/plsprod/bdump
    user_dump_dest = /backups/p01/app/oracle/admin/plsprod/udump
    core_dump_dest = /backups/p01/app/oracle/admin/plsprod/cdump
    session_max_open_files = 60
    sort_area_size = 1048576
    sort_area_retained_size = 1048576
    db_name = plsprod
    open_cursors = 100
    os_authent_prefix =
    pga_aggregate_target = 26214400
    aq_tm_processes = 0
    PMON started with pid=2, OS id=5623926
    PSP0 started with pid=3, OS id=5566632
    MMAN started with pid=4, OS id=5328992
    DBW0 started with pid=5, OS id=4362640
    DBW1 started with pid=6, OS id=4808940
    LGWR started with pid=7, OS id=4358468
    CKPT started with pid=8, OS id=4784160
    SMON started with pid=9, OS id=4354538
    RECO started with pid=10, OS id=4313420
    MMON started with pid=11, OS id=4771908
    MMNL started with pid=12, OS id=4743272
    Mon Jul 18 12:28:53 2011
    ALTER DATABASE MOUNT
    Mon Jul 18 12:28:57 2011
    Setting recovery target incarnation to 1
    Mon Jul 18 12:28:57 2011
    Successful mount of redo thread 1, with mount id 2894903685
    Mon Jul 18 12:28:57 2011
    Database mounted in Exclusive Mode
    Completed: ALTER DATABASE MOUNT
    Mon Jul 18 12:28:58 2011
    ALTER DATABASE OPEN
    Mon Jul 18 12:28:58 2011
    Beginning crash recovery of 1 threads
    parallel recovery started with 15 processes
    Mon Jul 18 12:28:58 2011
    Started redo scan
    Mon Jul 18 12:28:58 2011
    Completed redo scan
    1696 redo blocks read, 56 data blocks need recovery
    Mon Jul 18 12:28:58 2011
    Started redo application at
    Thread 1: logseq 17759, block 26354
    Mon Jul 18 12:28:58 2011
    Recovery of Online Redo Log: Thread 1 Group 1 Seq 17759 Reading mem 0
    Mem# 0: /data/plsprod/p01/redo_t1_g1_f1.log
    Mem# 1: /data/plsprod/p02/redo_t1_g1_f2.log
    Mon Jul 18 12:28:58 2011
    Completed redo application
    Mon Jul 18 12:28:58 2011
    Completed crash recovery at
    Thread 1: logseq 17759, block 28050, scn 35195499598
    56 data blocks read, 56 data blocks written, 1696 redo blocks read
    Mon Jul 18 12:28:58 2011
    LGWR: STARTING ARCH PROCESSES
    ARC0 started with pid=29, OS id=4338022
    Mon Jul 18 12:28:58 2011
    ARC0: Archival started
    ARC1: Archival started
    LGWR: STARTING ARCH PROCESSES COMPLETE
    ARC1 started with pid=30, OS id=1081522
    LNS1 started with pid=31, OS id=5537854
    Mon Jul 18 12:29:01 2011
    Thread 1 advanced to log sequence 17760
    Thread 1 opened at log sequence 17760
    Current log# 2 seq# 17760 mem# 0: /data/plsprod/p01/redo_t1_g2_f1.log
    Current log# 2 seq# 17760 mem# 1: /data/plsprod/p02/redo_t1_g2_f2.log
    Successful open of redo thread 1
    Mon Jul 18 12:29:01 2011
    ARC0: STARTING ARCH PROCESSES
    Mon Jul 18 12:29:01 2011
    ARC1: Becoming the 'no FAL' ARCH
    ARC1: Becoming the 'no SRL' ARCH
    Mon Jul 18 12:29:01 2011
    SMON: enabling cache recovery
    Mon Jul 18 12:29:02 2011
    ARC2: Archival started
    ARC0: STARTING ARCH PROCESSES COMPLETE
    ARC0: Becoming the heartbeat ARCH
    ARC2 started with pid=32, OS id=5644370
    Mon Jul 18 12:29:02 2011
    ARC0: Standby redo logfile selected for thread 1 sequence 17759 for destination LOG_ARCHIVE_DEST_2
    Mon Jul 18 12:29:02 2011
    LGWR: Setting 'active' archival for destination LOG_ARCHIVE_DEST_2
    Mon Jul 18 12:29:03 2011
    Successfully onlined Undo Tablespace 16.
    Mon Jul 18 12:29:03 2011
    SMON: enabling tx recovery
    Mon Jul 18 12:29:03 2011
    Database Characterset is US7ASCII
    replication_dependency_tracking turned off (no async multimaster replication found)
    WARNING: AQ_TM_PROCESSES is set to 0. System operation might be adversely affected.
    Completed: ALTER DATABASE OPEN
    Mon Jul 18 12:29:58 2011
    Shutting down archive processes
    Mon Jul 18 12:30:03 2011
    ARCH shutting down
    ARC2: Archival stopped

    Hi CKPT,
    Please find the output of the trace file as below.I deleted the repetitive output "Attempting to send buffer 0x111441d30 start block 91942 block count 84
    total size to send 43008 blocksize 512" as it exceeds maximum length of 30000 characters in this post.
    CATALINA_mardiprd>cat /backups/p01/app/oracle/admin/plsprod/bdump/plsprod_lns1_3772752.trc
    /backups/p01/app/oracle/admin/plsprod/bdump/plsprod_lns1_3772752.trc
    Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bit Production
    With the Partitioning, OLAP and Data Mining options
    ORACLE_HOME = /p01/app/oracle10g/product/10.2.0.3
    System name: AIX
    Node name: catalina
    Release: 3
    Version: 5
    Machine: 00044C5FD600
    Instance name: plsprod
    Redo thread mounted by this instance: 1
    Oracle process number: 16
    Unix process pid: 3772752, image: oracle@catalina (LNS1)
    *** SERVICE NAME:() 2011-07-18 09:42:11.573
    *** SESSION ID:(2189.1) 2011-07-18 09:42:11.573
    *** 2011-07-18 09:42:11.573 57516 kcrr.c
    LNS1: initializing for LGWR communication
    LNS1: connecting to KSR channel
    Success
    LNS1: subscribing to KSR channel
    Success
    *** 2011-07-18 09:42:11.574 57569 kcrr.c
    LNS1: initialized successfully ASYNC=1
    Destination is specified with ASYNC=61440
    *** 2011-07-18 09:42:12.577 71560 kcrr.c
    Sending online log thread 1 seq 17731 [logfile 3] to standby
    Opening logfile [logno 3]
    Redo shipping client performing standby login
    *** 2011-07-18 09:42:12.693 65194 kcrr.c
    Logged on to standby successfully
    Client logon and security negotiation successful!
    Destination LOG_ARCHIVE_DEST_2 is in MAXIMUM PERFORMANCE mode
    Archiving to destination talon_plsprod.travimp.com ASYNC blocks=20480
    Allocate ASYNC blocks: Previous blocks=0 New blocks=20480
    Log file opened [logno 3]
    Attempting to send buffer 0x111441d30 start block 2 block count 1
    total size to send 512 blocksize 512
    Attempting to send buffer 0x111441d30 start block 3 block count 35
    total size to send 17920 blocksize 512
    *** 2011-07-18 09:42:27.009
    Attempting to send buffer 0x111441d30 start block 38 block count 2914
    total size to send 1491968 blocksize 512
    *** 2011-07-18 09:42:50.007
    Attempting to send buffer 0x111441d30 start block 2952 block count 2
    total size to send 1024 blocksize 512
    *** 2011-07-18 09:43:01.441
    Attempting to send buffer 0x111441d30 start block 2954 block count 19286
    total size to send 9874432 blocksize 512
    Attempting to send buffer 0x111441d30 start block 22240 block count 19657
    total size to send 10064384 blocksize 512
    Attempting to send buffer 0x111441d30 start block 41897 block count 1687
    total size to send 863744 blocksize 512
    *** 2011-07-18 09:43:13.007
    Attempting to send buffer 0x111441d30 start block 43584 block count 1
    total size to send 512 blocksize 512
    Attempting to send buffer 0x111441d30 start block 43585 block count 13
    total size to send 6656 blocksize 512
    Attempting to send buffer 0x111441d30 start block 43598 block count 19008
    total size to send 9732096 blocksize 512
    *** 2011-07-18 09:43:24.265
    Attempting to send buffer 0x111441d30 start block 62606 block count 19349
    total size to send 9906688 blocksize 512
    *** 2011-07-18 09:43:25.524 72367 kcrr.c
    Detected proper completion of online log
    *** 2011-07-18 09:43:25.524 72204 kcrr.c
    Detected physical end of file
    Attempting to send buffer 0x111441d30 start block 81955 block count 10066
    total size to send 5153792 blocksize 512
    Online entry still exists for log 3 thr 1 seq 17731. Using it
    Closing thread 1 sequence 17731 logno 3 nab 92021 los 35193489640 nxs 35193490559
    *** 2011-07-18 09:43:26.686 71560 kcrr.c
    Sending online log thread 1 seq 17732 [logfile 1] to standby
    Opening logfile [logno 1]
    Destination LOG_ARCHIVE_DEST_2 is in MAXIMUM PERFORMANCE mode
    Archiving to destination talon_plsprod.travimp.com ASYNC blocks=20480
    Log file opened [logno 1]
    Attempting to send buffer 0x111441d30 start block 2 block count 20407
    total size to send 10448384 blocksize 512
    Attempting to send buffer 0x111441d30 start block 20409 block count 20349
    total size to send 10418688 blocksize 512
    Attempting to send buffer 0x111441d30 start block 40758 block count 19751
    total size to send 10112512 blocksize 512
    Attempting to send buffer 0x111441d30 start block 60509 block count 20260
    total size to send 10373120 blocksize 512
    *** 2011-07-18 09:43:33.262 72367 kcrr.c
    Detected proper completion of online log
    *** 2011-07-18 09:43:33.262 72204 kcrr.c
    Detected physical end of file
    Attempting to send buffer 0x111441d30 start block 80769 block count 11251
    total size to send 5760512 blocksize 512
    Online entry still exists for log 1 thr 1 seq 17732. Using it
    Closing thread 1 sequence 17732 logno 1 nab 92020 los 35193490559 nxs 35193490735
    *** 2011-07-18 09:43:33.541 71560 kcrr.c
    Sending online log thread 1 seq 17733 [logfile 2] to standby
    Opening logfile [logno 2]
    Destination LOG_ARCHIVE_DEST_2 is in MAXIMUM PERFORMANCE mode
    *** 2011-07-18 09:43:34.387
    Archiving to destination talon_plsprod.travimp.com ASYNC blocks=20480
    Log file opened [logno 2]
    Attempting to send buffer 0x111441d30 start block 2 block count 19192
    total size to send 9826304 blocksize 512
    Attempting to send buffer 0x111441d30 start block 19194 block count 18876
    total size to send 9664512 blocksize 512
    Attempting to send buffer 0x111441d30 start block 38070 block count 19565
    total size to send 10017280 blocksize 512
    Attempting to send buffer 0x111441d30 start block 57635 block count 19765
    total size to send 10119680 blocksize 512
    *** 2011-07-18 09:43:41.660 72367 kcrr.c
    Detected proper completion of online log
    *** 2011-07-18 09:43:41.660 72204 kcrr.c
    Detected physical end of file
    Attempting to send buffer 0x111441d30 start block 77400 block count 14627
    total size to send 7489024 blocksize 512
    Online entry still exists for log 2 thr 1 seq 17733. Using it
    Closing thread 1 sequence 17733 logno 2 nab 92027 los 35193490735 nxs 35193490800
    *** 2011-07-18 09:43:41.928 71560 kcrr.c
    Sending online log thread 1 seq 17734 [logfile 3] to standby
    Opening logfile [logno 3]
    Destination LOG_ARCHIVE_DEST_2 is in MAXIMUM PERFORMANCE mode
    Archiving to destination talon_plsprod.travimp.com ASYNC blocks=20480
    Log file opened [logno 3]
    Attempting to send buffer 0x111441d30 start block 59038 block count 18758
    total size to send 9604096 blocksize 512
    *** 2011-07-18 09:44:42.141
    *** 2011-07-18 09:44:42.141 72367 kcrr.c
    Detected proper completion of online log
    *** 2011-07-18 09:44:42.142 72204 kcrr.c
    Detected physical end of file
    total size to send 586240 blocksize 512
    Online entry still exists for log 1 thr 1 seq 17744. Using it
    Closing thread 1 sequence 17744 logno 1 nab 92029 los 35193494218 nxs 35193494529
    *** 2011-07-18 09:45:58.974 71560 kcrr.c
    Sending online log thread 1 seq 17745 [logfile 2] to standby
    Opening logfile [logno 2]
    Destination LOG_ARCHIVE_DEST_2 is in MAXIMUM PERFORMANCE mode
    Archiving to destination talon_plsprod.travimp.com ASYNC blocks=20480
    Log file opened [logno 2]
    Attempting to send buffer 0x111441d30 start block 2 block count 9013
    total size to send 4614656 blocksize 512
    Attempting to send buffer 0x111441d30 start block 9015 block count 20176
    total size to send 10330112 blocksize 512
    Attempting to send buffer 0x111441d30 start block 29191 block count 19808
    total size to send 10141696 blocksize 512
    Attempting to send buffer 0x111441d30 start block 48999 block count 18939
    total size to send 9696768 blocksize 512
    *** 2011-07-18 09:46:10.146
    Attempting to send buffer 0x111441d30 start block 67938 block count 19124
    total size to send 9791488 blocksize 512
    Attempting to send buffer 0x111441d30 start block 87062 block count 3690
    total size to send 1889280 blocksize 512
    *** 2011-07-18 09:46:12.644 72367 kcrr.c
    Detected proper completion of online log
    *** 2011-07-18 09:46:12.644 72204 kcrr.c
    Detected physical end of file
    Attempting to send buffer 0x111441d30 start block 90752 block count 1267
    total size to send 648704 blocksize 512
    Online entry still exists for log 2 thr 1 seq 17745. Using it
    Closing thread 1 sequence 17745 logno 2 nab 92019 los 35193494529 nxs 35193494940
    *** 2011-07-18 09:46:13.690 71560 kcrr.c
    Sending online log thread 1 seq 17746 [logfile 3] to standby
    Opening logfile [logno 3]
    Destination LOG_ARCHIVE_DEST_2 is in MAXIMUM PERFORMANCE mode
    Archiving to destination talon_plsprod.travimp.com ASYNC blocks=20480
    Log file opened [logno 3]
    Attempting to send buffer 0x111441d30 start block 2 block count 19533
    total size to send 10000896 blocksize 512
    Attempting to send buffer 0x111441d30 start block 19535 block count 11151
    total size to send 5709312 blocksize 512
    Attempting to send buffer 0x111441d30 start block 30686 block count 20154
    total size to send 10318848 blocksize 512
    *** 2011-07-18 09:46:20.822
    Attempting to send buffer 0x111441d30 start block 50840 block count 19083
    total size to send 9770496 blocksize 512
    Attempting to send buffer 0x111441d30 start block 69923 block count 19167
    total size to send 9813504 blocksize 512
    *** 2011-07-18 09:46:22.573 72367 kcrr.c
    Detected proper completion of online log
    *** 2011-07-18 09:46:22.573 72204 kcrr.c
    Detected physical end of file
    Attempting to send buffer 0x111441d30 start block 89090 block count 2933
    total size to send 1501696 blocksize 512
    Online entry still exists for log 3 thr 1 seq 17746. Using it
    Closing thread 1 sequence 17746 logno 3 nab 92023 los 35193494940 nxs 35193495191
    *** 2011-07-18 09:46:23.637 71560 kcrr.c
    Sending online log thread 1 seq 17747 [logfile 1] to standby
    Opening logfile [logno 1]
    Destination LOG_ARCHIVE_DEST_2 is in MAXIMUM PERFORMANCE mode
    Archiving to destination talon_plsprod.travimp.com ASYNC blocks=20480
    Log file opened [logno 1]
    Attempting to send buffer 0x111441d30 start block 2 block count 8093
    total size to send 4143616 blocksize 512
    Attempting to send buffer 0x111441d30 start block 8095 block count 20288
    total size to send 10387456 blocksize 512
    Attempting to send buffer 0x111441d30 start block 28383 block count 18967
    total size to send 9711104 blocksize 512
    *** 2011-07-18 09:46:31.007
    Attempting to send buffer 0x111441d30 start block 47350 block count 5237
    total size to send 2681344 blocksize 512
    Attempting to send buffer 0x111441d30 start block 52587 block count 20188
    total size to send 10336256 blocksize 512
    Attempting to send buffer 0x111441d30 start block 72775 block count 19167
    total size to send 9813504 blocksize 512
    *** 2011-07-18 09:46:37.430 72367 kcrr.c
    Detected proper completion of online log
    *** 2011-07-18 09:46:37.430 72204 kcrr.c
    Detected physical end of file
    Attempting to send buffer 0x111441d30 start block 91942 block count 84
    total size to send 43008 blocksize 512
    Online entry still exists for log 1 thr 1 seq 17747. Using it
    Closing thread 1 sequence 17747 logno 1 nab 92026 los 35193495191 nxs 35193495503
    *** 2011-07-18 09:46:37.501 71560 kcrr.c
    Sending online log thread 1 seq 17748 [logfile 2] to standby
    Opening logfile [logno 2]
    Destination LOG_ARCHIVE_DEST_2 is in MAXIMUM PERFORMANCE mode
    Archiving to destination talon_plsprod.travimp.com ASYNC blocks=20480
    Log file opened [logno 2]
    Attempting to send buffer 0x111441d30 start block 2 block count 2052
    total size to send 1050624 blocksize 512
    *** 2011-07-18 09:46:43.059
    total size to send 8164352 blocksize 512
    *** 2011-07-18 09:48:18.781
    *** 2011-07-18 09:48:18.781 72367 kcrr.c
    Detected proper completion of online log
    **** 2011-07-18 09:48:18.781 72204 kcrr.c*
    Detected physical end of file
    Attempting to send buffer 0x111441d30  start block 90604  block count 1416
    total size to send 724992  blocksize 512
    Online entry still exists for log 2 thr 1 seq 17754. Using it
    Closing thread 1 sequence 17754 logno 2 nab 92020 los 35193496751 nxs 35193496934
    **** 2011-07-18 09:48:19.891 71560 kcrr.c*
    *Sending online log thread 1 seq 17755 [logfile 3] to standby*
    *Opening logfile [logno 3]*
    Destination LOG_ARCHIVE_DEST_2 is in MAXIMUM PERFORMANCE mode
    Archiving to destination talon_plsprod.travimp.com ASYNC blocks=20480
    *Log file opened [logno 3]*
    Attempting to send buffer 0x111441d30 start block 2 block count 19248
    total size to send 9854976 blocksize 512
    Attempting to send buffer 0x111441d30 start block 19250 block count 20166
    total size to send 10324992 blocksize 512
    Attempting to send buffer 0x111441d30 start block 39416 block count 4955
    total size to send 2536960 blocksize 512
    Attempting to send buffer 0x111441d30 start block 44371 block count 19903
    total size to send 10190336 blocksize 512
    *** 2011-07-18 09:48:30.215
    Attempting to send buffer 0x111441d30 start block 64274 block count 19991
    total size to send 10235392 blocksize 512
    *** 2011-07-18 09:48:32.370 72367 kcrr.c
    Detected proper completion of online log
    *** 2011-07-18 09:48:32.370 72204 kcrr.c
    Detected physical end of file
    Attempting to send buffer 0x111441d30 start block 84265 block count 7754
    total size to send 3970048 blocksize 512
    Online entry still exists for log 3 thr 1 seq 17755. Using it
    Closing thread 1 sequence 17755 logno 3 nab 92019 los 35193496934 nxs 35193498387
    *** 2011-07-18 09:48:32.472 71560 kcrr.c
    Sending online log thread 1 seq 17756 [logfile 1] to standby
    Opening logfile [logno 1]
    Destination LOG_ARCHIVE_DEST_2 is in MAXIMUM PERFORMANCE mode
    Archiving to destination talon_plsprod.travimp.com ASYNC blocks=20480
    Log file opened [logno 1]
    Attempting to send buffer 0x111441d30 start block 2 block count 19214
    total size to send 9837568 blocksize 512
    *** 2011-07-18 09:48:40.267
    Attempting to send buffer 0x111441d30 start block 19216 block count 19981
    total size to send 10230272 blocksize 512
    Attempting to send buffer 0x111441d30 start block 39197 block count 19586
    total size to send 10028032 blocksize 512
    Attempting to send buffer 0x111441d30 start block 58783 block count 6569
    total size to send 3363328 blocksize 512
    Attempting to send buffer 0x111441d30 start block 65352 block count 19894
    total size to send 10185728 blocksize 512
    *** 2011-07-18 09:48:47.133 72367 kcrr.c
    Detected proper completion of online log
    *** 2011-07-18 09:48:47.133 72204 kcrr.c
    Detected physical end of file
    Attempting to send buffer 0x111441d30 start block 85246 block count 6772
    total size to send 3467264 blocksize 512
    Online entry still exists for log 1 thr 1 seq 17756. Using it
    Closing thread 1 sequence 17756 logno 1 nab 92018 los 35193498387 nxs 35193500455
    *** 2011-07-18 09:48:47.344 71560 kcrr.c
    Sending online log thread 1 seq 17757 [logfile 2] to standby
    Opening logfile [logno 2]
    Destination LOG_ARCHIVE_DEST_2 is in MAXIMUM PERFORMANCE mode
    Archiving to destination talon_plsprod.travimp.com ASYNC blocks=20480
    Log file opened [logno 2]
    *** 2011-07-18 09:48:50.302
    Attempting to send buffer 0x111441d30 start block 2 block count 18546
    total size to send 9495552 blocksize 512
    Attempting to send buffer 0x111441d30 start block 18548 block count 19821
    total size to send 10148352 blocksize 512
    Attempting to send buffer 0x111441d30 start block 38369 block count 20457
    total size to send 10473984 blocksize 512
    *** 2011-07-18 09:49:02.194
    Attempting to send buffer 0x111441d30 start block 58826 block count 20477
    total size to send 10484224 blocksize 512
    Attempting to send buffer 0x111441d30 start block 79303 block count 12397
    total size to send 6347264 blocksize 512
    *** 2011-07-18 09:49:08.443 72367 kcrr.c
    Detected proper completion of online log
    *** 2011-07-18 09:49:08.443 72204 kcrr.c
    Detected physical end of file
    Attempting to send buffer 0x111441d30 start block 91700 block count 333
    total size to send 170496 blocksize 512
    Online entry still exists for log 2 thr 1 seq 17757. Using it
    Closing thread 1 sequence 17757 logno 2 nab 92033 los 35193500455 nxs 35193504614
    *** 2011-07-18 09:49:08.549 71560 kcrr.c
    Sending online log thread 1 seq 17758 [logfile 3] to standby
    Opening logfile [logno 3]
    Destination LOG_ARCHIVE_DEST_2 is in MAXIMUM PERFORMANCE mode
    Archiving to destination talon_plsprod.travimp.com ASYNC blocks=20480
    Log file opened [logno 3]
    Attempting to send buffer 0x111441d30 start block 2 block count 19640
    total size to send 10055680 blocksize 512
    *** 2011-07-18 09:49:15.005
    Attempting to send buffer 0x111441d30 start block 19642 block count 12646
    total size to send 6474752 blocksize 512
    Attempting to send buffer 0x111441d30 start block 92000 block count 23
    total size to send 11776 blocksize 512
    *** 2011-07-18 11:30:19.515 72367 kcrr.c
    Detected proper completion of online log
    *** 2011-07-18 11:30:19.515 72204 kcrr.c
    Detected physical end of file
    Attempting to send buffer 0x111441d30 start block 92023 block count 11
    total size to send 5632 blocksize 512
    Online entry still exists for log 3 thr 1 seq 17758. Using it
    Closing thread 1 sequence 17758 logno 3 nab 92034 los 35193504614 nxs 35194291751
    *** 2011-07-18 11:30:21.546 71560 kcrr.c
    Sending online log thread 1 seq 17759 [logfile 1] to standby
    Opening logfile [logno 1]
    Destination LOG_ARCHIVE_DEST_2 is in MAXIMUM PERFORMANCE mode
    Archiving to destination talon_plsprod.travimp.com ASYNC blocks=20480
    Log file opened [logno 1]
    Attempting to send buffer 0x111441d30 start block 2 block count 24
    total size to send 12288 blocksize 512
    *** 2011-07-18 11:30:27.006
    Attempting to send buffer 0x111441d30 start block 26 block count 1
    total size to send 512 blocksize 512
    Attempting to send buffer 0x111441d30 start block 27 block count 34
    total size to send 17408 blocksize 512
    Attempting to send buffer 0x111441d30 start block 61 block count 31
    total size to send 15872 blocksize 512
    *** 2011-07-18 11:30:39.003
    Attempting to send buffer 0x111441d30 start block 92 block count 32
    total size to send 16384 blocksize 512
    Attempting to send buffer 0x111441d30 start block 124 block count 2
    total size to send 1024 blocksize 512
    Attempting to send buffer 0x111441d30 start block 126 block count 1
    total size to send 512 blocksize 512
    *** 2011-07-18 11:30:51.000
    Attempting to send buffer 0x111441d30 start block 127 block count 1
    total size to send 512 blocksize 512
    Attempting to send buffer 0x111441d30 start block 128 block count 30
    total size to send 15360 blocksize 512
    Attempting to send buffer 0x111441d30 start block 158 block count 88
    total size to send 45056 blocksize 512
    Attempting to send buffer 0x111441d30 start block 246 block count 3
    total size to send 1536 blocksize 512
    *** 2011-07-18 11:31:03.002
    Attempting to send buffer 0x111441d30 start block 249 block count 32
    total size to send 16384 blocksize 512
    Attempting to send buffer 0x111441d30 start block 27416 block count 4
    total size to send 2048 blocksize 512
    Attempting to send buffer 0x111441d30 start block 27420 block count 14
    total size to send 7168 blocksize 512
    Attempting to send buffer 0x111441d30 start block 27434 block count 54
    total size to send 27648 blocksize 512
    Attempting to send buffer 0x111441d30 start block 27488 block count 10
    total size to send 5120 blocksize 512
    *** 2011-07-18 12:27:01.000
    Attempting to send buffer 0x111441d30 start block 27498 block count 78
    total size to send 39936 blocksize 512
    Attempting to send buffer 0x111441d30 start block 27576 block count 2
    total size to send 1024 blocksize 512
    Attempting to send buffer 0x111441d30 start block 27578 block count 56
    total size to send 28672 blocksize 512
    Attempting to send buffer 0x111441d30 start block 27634 block count 60
    total size to send 30720 blocksize 512
    *** 2011-07-18 12:27:14.006
    Attempting to send buffer 0x111441d30 start block 27694 block count 7
    total size to send 3584 blocksize 512
    Attempting to send buffer 0x111441d30 start block 27701 block count 1
    total size to send 512 blocksize 512
    Attempting to send buffer 0x111441d30 start block 27702 block count 106
    total size to send 54272 blocksize 512
    *** 2011-07-18 12:27:28.001
    Attempting to send buffer 0x111441d30 start block 27808 block count 2
    total size to send 1024 blocksize 512
    Attempting to send buffer 0x111441d30 start block 27810 block count 37
    total size to send 18944 blocksize 512
    Attempting to send buffer 0x111441d30 start block 27847 block count 2
    total size to send 1024 blocksize 512
    *** 2011-07-18 12:27:42.002
    Attempting to send buffer 0x111441d30 start block 27849 block count 78
    total size to send 39936 blocksize 512
    Attempting to send buffer 0x111441d30 start block 27927 block count 43
    total size to send 22016 blocksize 512
    *** 2011-07-18 12:27:52.001
    Attempting to send buffer 0x111441d30 start block 27970 block count 1
    total size to send 512 blocksize 512
    Attempting to send buffer 0x111441d30 start block 27971 block count 2
    total size to send 1024 blocksize 512
    Attempting to send buffer 0x111441d30 start block 27973 block count 38
    total size to send 19456 blocksize 512
    *** 2011-07-18 12:28:03.007
    ORA-00472: PMON  process terminated with error
    Closing Redo Read Context
    Redo Push Server: Freeing ASYNC PGA buffer
    LNS1: Doing a channel reset for next time around...
    Please help me out.

  • ORA-00471: DBWR process terminated with error when running mrua.pl

    we are upgrading to Portal 10.1.4 from 10.2.0.2.
    Right in the middle of the upgrade scripts running get ORA-00471: DBWR process terminated with error...
    Any suggestion?
    TIA

    sorry, it is me agin:
    I forgot:
    it must a trace file produced by PMON because it was PMON who killed the instance.
    These background processes function as watchdogs for each other. If one finds that one of the others is missing it will kill the instance.
    Regards,
    Lutz

  • Instance crashing with ORA-00469: CKPT process terminated with error

    Hi,
    Instance crashing with ORA-00469: CKPT process terminated with error on 10.2.0.4. Any possibe causes?
    Thanks,
    Madhav

    I did check alert log and trace files. I just find this in alert log:
    ORA-00469: CKPT process terminated with error
    Trace file:
    *** 2009-07-29 02:22:06.840
    *** SERVICE NAME:(SYS$BACKGROUND) 2009-07-29 02:22:06.839
    *** SESSION ID:(275.1) 2009-07-29 02:22:06.839
    error 469 detected in background process
    ORA-00469: CKPT process terminated with error

  • ORA-00488: RBAL process terminated with error

    hi friends,
    OS: Enterprise Linux - 4 32bit, DB: oracle database 10g (10.2.0.1)
    I'm working around RAC for study purpose using VMWare server 1.0.4..
    everythings goes fine till CLUSTER (10.2.0.1) installation, oracle database software installed without any error. but when i'm trying to create database using DBCA, in ASM instance creation. I got the following error and database installation stops
    ORA-00488: RBAL process terminated with error
    can anyone tell me what doese cause it.?
    how to get around with this error..?
    Thanks in advance.
    Regards

    VMware is an unsupported environment. Even if you get this working you will be unable to obtain support for it. Give serious consideration to using supported configurations.

  • ORA-00476: RECO process terminated with error

    My dev instance shutdown automatically due to error 476. I looked up on the error details and found this:
    Sun Jan 7 10:17:39 2007
    Errors in file /usr/lib/oracle/xe/app/oracle/admin/XE/bdump/xe_pmon_26220.trc:
    ORA-00476: RECO process terminated with error
    Sun Jan 7 10:17:39 2007
    PMON: terminating instance due to error 476
    Instance terminated by PMON, pid = 26220
    Details for this error from xe_pmon_26220.trc file:
    /usr/lib/oracle/xe/app/oracle/admin/XE/bdump/xe_pmon_26220.trc
    Oracle Database 10g Express Edition Release 10.2.0.1.0 - Production
    ORACLE_HOME = /usr/lib/oracle/xe/app/oracle/product/10.2.0/server
    System name:     Linux
    Node name:     boromir
    Release:     2.6.9-34.0.1.ELsmp
    Version:     #1 SMP Wed May 17 17:05:24 EDT 2006
    Machine:     i686
    Instance name: XE
    Redo thread mounted by this instance: 1
    Oracle process number: 2
    Unix process pid: 26220, (PMON)
    *** 2007-01-07 10:17:34.543
    *** SERVICE NAME:(SYS$BACKGROUND) 2007-01-07 10:17:18.261
    *** SESSION ID:(225.1) 2007-01-07 10:17:18.261
    Background process RECO found dead
    Oracle pid = 9
    OS pid (from detached process) = 26234
    OS pid (from process state) = 26234
    error 476 detected in background process
    ORA-00476: RECO process terminated with error
    After restarting database I found information about succesfull data recovery
    Mon Jan 8 10:37:59 2007
    Starting ORACLE instance (normal)
    Started redo scan
    Completed redo scan
    2126 redo blocks read, 42 data blocks need recovery
    Started redo application at
    Thread 1: logseq 152, block 56161
    Recovery of Online Redo Log: Thread 1 Group 1 Seq 152 Reading mem 0
    Mem# 0 errs 0: /usr/lib/oracle/xe/app/oracle/flash_recovery_area/XE/onlinelog/o1_mf_1_2f61ppsy_.log
    Completed redo application
    Completed crash recovery at
    Thread 1: logseq 152, block 58287, scn 6441576
    42 data blocks read, 42 data blocks written, 2126 redo blocks read
    Thread 1 checkpoint: logseq 152, block 2, scn 6369309
    cache-low rba: logseq 152, block 56161
    on-disk rba: logseq 152, block 58287, scn 6421576
    start recovery at logseq 152, block 56161, scn 0
    ----- Redo read statistics for thread 1 -----
    Read rate (ASYNC): 1063Kb in 0.20s => 5.19 Mb/sec
    Total physical reads: 4096Kb
    Longest record: 2Kb, moves: 0/1139 (0%)
    Change moves: 0/24 (0%), moved: 0Mb
    Longest LWN: 6Kb, moves: 0/1047 (0%), moved: 0Mb
    Last redo scn: 0x0000.0061fc47 (6421575)
    *** 2007-01-08 10:38:05.342
    KCRA: start recovery claims for 42 data blocks
    *** 2007-01-08 10:38:05.446
    KCRA: blocks processed = 42/42, claimed = 42, eliminated = 0
    *** 2007-01-08 10:38:05.448
    Recovery of Online Redo Log: Thread 1 Group 1 Seq 152 Reading mem 0
    What are the possible reasons of error ??
    Konrad

    it seems that it might be related to killing the RECO
    process on the OS level. The crash happen on Sunday (Sun Jan 7 10:17:39 2007) and no one have phisical access to OS
    Are you using database links? Might be related to
    that, too. No I dont use database links. One database with two schemas.
    Does it happen frequently?It happen only one time.
    Regards,
    Konrad.

  • ORA-00488: RBAL process terminated with error causing ASM instance to crash

    Hi Freinds,
    I am manging a three node ASM cluster on 10g(10.2.0.2).Yesterday I observed one of the ASM instance got crashed .On reviewing the Alert log of crashed ASM instance I found
    "*ORA-00488: RBAL process terminated with error*"
    and subsequently
    "*PMON: terminating instance due to error 488*".
    I could restart the crashed instance without an issues but I want to know what kind of event could lead to such scenario.
    I searched net and metalink but didn't found any helpful information.
    Did any of you faced similar issues and do you have understanding why and when it happens.
    Thanks in Advance
    Arindam.

    Hi,
    There is bug reported on such issue, please check further on metalink if you have support.
    Dilip Patel.

  • ORA-00471: DBWR process terminated with error

    Dear all,
    I got this error in alert log and database got down.
    ORA-00471: DBWR process terminated with error
    Wed May 24 19:17:37 2006
    PMON: terminating instance due to error 471
    Instance terminated by PMON, pid = 11642
    Wed May 24 19:28:15 2006
    I am having 10g R1 on LINUX.
    To find cause of error.I read somewhere that I should check sid_dbwr_*.trc file.
    But there is no file bdump related to DBWR.
    According to me there must be atleast one trace file of DBWR in bdump.
    Can you tell me why there is no trace file of DBWR in bdump and how can I ensure that i find trace files of dbwr in bdump.
    Waiting for your reply,
    Regards,
    Amit.
    Message was edited by:
    Amit Singla

    sorry, it is me agin:
    I forgot:
    it must a trace file produced by PMON because it was PMON who killed the instance.
    These background processes function as watchdogs for each other. If one finds that one of the others is missing it will kill the instance.
    Regards,
    Lutz

  • ORA-00470: LGWR process terminated with error

    Hi,
    Often we are getting the following error on our 10g instance.
    ORA-00470: LGWR process terminated with error
    Mon Sep 6 22:47:53 2010
    PMON: terminating instance due to error 470
    Instance terminated by PMON, pid = 16986
    And checked in /var/log/messages showing the following
    kernel: Out of Memory: Killed process 16995 (oracle)
    How can I solve this?
    Thanks,
    KAvitha

    Hi,
    I am using 10g database on Red Hat Enterprise Linux AS release 4 (Nahant Update 4).
    The alert log showing the following information.
    Mon Sep 6 23:09:45 2010
    Completed: ALTER DATABASE OPEN
    Mon Sep 6 23:17:30 2010
    Errors in file /d04/u02/oracle/admin/cidb/bdump/cidb_pmon_19969.trc:
    ORA-00470: LGWR process terminated with error
    Mon Sep 6 23:17:30 2010
    PMON: terminating instance due to error 470
    Termination issued to instance processes. Waiting for the processes to exit
    Mon Sep 6 23:17:44 2010
    Instance termination failed to kill one or more processes
    Instance terminated by PMON, pid = 19969
    And also check cidb_pmon_19969.trc file
    /d04/u02/oracle/admin/cidb/bdump/cidb_pmon_19969.trc
    Oracle Database 10g Enterprise Edition Release 10.2.0.2.0 - Production
    With the Partitioning, OLAP and Data Mining options
    ORACLE_HOME = /d04/u02/oracle/cidb
    System name: Linux
    Node name: sflxdb07.donerus.com
    Release: 2.6.9-34.ELsmp
    Version: #1 SMP Fri Feb 24 16:54:53 EST 2006
    Machine: i686
    Instance name: cidb
    Redo thread mounted by this instance: 1
    Oracle process number: 2
    Unix process pid: 19969, image: [email protected] (PMON)
    *** 2010-09-06 23:17:30.676
    *** SERVICE NAME:(SYS$BACKGROUND) 2010-09-06 23:17:30.530
    *** SESSION ID:(170.1) 2010-09-06 23:17:30.530
    Background process LGWR found dead
    Oracle pid = 6
    OS pid (from detached process) = 19977
    OS pid (from process state) = 19977
    dtp = 0x2000bab8, proc = 0x8b2b13fc
    Dump of memory from 0x2000BAB8 to 0x2000BAE4
    2000BAB0 00000074 8B2B13FC [t.....+.]
    2000BAC0 00000000 00000000 5257474C 00000200 [........LGWR....]
    2000BAD0 00004E09 113B6215 00000001 2AD428DC [.N...b;......(.*]
    2000BAE0 00040001 [....]
    Dump of memory from 0x8B2B13FC to 0x8B2B19B4
    8B2B13F0 00000102 [....]
    8B2B1400 00000000 00000000 00000000 00000000 [................]
    8B2B1410 89555C60 8B82DB2C 8B3C2078 8B82CBEC [`\U.,...x <.....]
    8B2B1420 00000000 8B82CC50 8B82CC50 8B82DB20 [....P...P... ...]
    8B2B1430 00000601 8B3AE848 8B3C2078 00000006 [....H.:.x <.....]
    8B2B1440 00000000 00000000 00000000 8B4D2F48 [............H/M.]
    8B2B1450 8B4D2FE4 00000000 00000000 00000000 [./M.............]
    8B2B1460 00000000 00000000 00000000 00000000 [................]
    Repeat 2 times
    8B2B1490 00000000 00000000 00030000 00000000 [................]
    8B2B14A0 00030000 00000000 00000000 00000000 [................]
    Repeat 1 times
    8B2B17B0 00000000 00000000 00000014 4E4B4E55 [............UNKN]
    8B2B17C0 004E574F 00000000 00000000 00000000 [OWN.............]
    8B2B17D0 00000000 00000000 00000000 00000008 [................]
    8B2B17E0 37393931 00000037 00000000 00000000 [19977...........]
    8B2B17F0 00000000 00000000 00000005 6361726F [............orac]
    8B2B19B0 00000006 [....]
    error 470 detected in background process
    ORA-00470: LGWR process terminated with error
    *** 2010-09-06 23:17:44.081
    Instance termination failed to kill one or more processes
    ksuitm_check: OS PID=20017 is still alive
    *** 2010-09-06 23:17:44.083
    Dumping diagnostic information for [email protected] (J002):
    OS pid = 20017
    loadavg : 63.67 22.12 10.45
    memory info: free memory = 0.00M
    swap info: free = 0.00M alloc = 0.00M total = 0.00M
    F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
    ERROR: process 20017 is not alive
    *** 2010-09-06 23:17:44.814
    Thanks,
    Kavitha

  • ORA-484 LMS* process terminated with error

    Instance is crashing frequently with the ORA-484 LMS* process terminated with error. Oracle RAC 10.2.0.3 on HP_UX 11i on ASM. Please advise

    The problem got solved after installing an interim patch p5375583_10203_HP64

  • ORA-00205 error leading to ORA-00470 LGWR process terminated with error.

    Hello Gurus,
    One of our database had gone down, I tried to bring it up. When tried to do so it gave follwoing errot ORA-00205 error in identifying control file, check alert log ..... When I checked alter log entried I found a new error ORA-00470 LGWR process terminated with eror.
    All my control files are intact where supposed to be, so are all redo log files. Please could any would help parity at my end.
    In need for restoring my database at the earliest.
    Regards
    Shiva

    Could you please post the full alert.log contents when the database starts up ?
    Are you really sure that all your control files are OK ?
    ORA-00205 error in identifying controlfile, check alert log for more info
    Cause: The system could not find a control file of the specified name and size.
    Action: One of the following:
    Check that the proper control file name is referenced in the CONTROL_FILES initialization parameter in the initialization parameter file and try again.
    When using mirrored control files, that is, more than one control file is referenced in the initialization parameter file, remove the control file name listed in the message from the initialization parameter file and restart the instance. If the message does not recur, remove the problem control file from the initialization parameter file and create another copy of the control file with a new file name in the initialization parameter file.

  • ORA-00490: PSP process terminated with error

    Solaris 10
    Oracle 10G R 2
    We get this error and db goes down. Till warm start.
    There is not much information on PSP0 process on the net.
    Any ideas?

    Not much on Metalink either. PSP is the Process SPawner. It is responsible for spawning and managing the Oracle process (Note: 331000.1).
    Are there any errors or traces generated in the Alert log that occur before the instance is terminated by PSP?

  • Ora-00472 PMON process terminated with error on production system

    Hi,
    I am getting the above error when I shutdown the database after a power outage. Alert log file shows " ora-00130 Invalid listener address when I bring up the database...." Nothing has changed between prior to the power outage and after that.
    RDBMS 9i.
    Help please!!
    A/A

    This is what the listener status shows
    Alias                     FINPROD
    Version                   TNSLSNR for HPUX: Version 9.2.0.6.0 - Production
    Start Date                26-JUL-2010 13:45:15
    Uptime                    0 days 2 hr. 9 min. 22 sec
    Trace Level               off
    Security                  OFF
    SNMP                      OFF
    Listener Parameter File   /d01/finprod/product/9i/network/admin/listener.ora
    Listener Log File         /d01/finprod/product/9i/network/admin/finprod.log
    Listening Endpoints Summary...
      (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROCFINPROD)))
      (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=hp3.nfpc.com)(PORT=1521)))
    Services Summary...
    Service "FINPROD" has 1 instance(s).
      Instance "FINPROD", status UNKNOWN, has 1 handler(s) for this service...
    Service "FINPROD.nfpc.com" has 1 instance(s).
      Instance "FINPROD", status READY, has 1 handler(s) for this service...
    Service "PLSExtProc" has 1 instance(s).
      Instance "PLSExtProc", status UNKNOWN, has 1 handler(s) for this service...
    The command completed successfully

  • Error 472 in job queue process + ORA-472: PMON  process terminated with err

    Hi,
    I've an Oracle Database 11g Release 11.1.0.6.0 - 64bit Production With the Real Application Clusters option.
    Yesterday instance on node1 crashed.
    This is the extract of alert log:
    Errors in file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_pmon_9297.trc (incident=195198):
    ORA-00600: internal error code, arguments: [17183], [0x14E251C38], [], [], [], [], [], []
    Incident details in: /u01/app/oracle/diag/rdbms/evodb/EVODB1/incident/incdir_195198/EVODB1_pmon_9297_i195198.trc
    Sat Jan 22 16:24:47 2011
    Trace dumping is performing id=[cdmp_20110122162447]
    Sat Jan 22 16:24:50 2011
    Sweep Incident[195198]: completed
    Sat Jan 22 16:25:00 2011
    Errors in file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_pmon_9297.trc (incident=195199):
    ORA-00600: internal error code, arguments: [17183], [0x14E251C38], [], [], [], [], [], []
    Incident details in: /u01/app/oracle/diag/rdbms/evodb/EVODB1/incident/incdir_195199/EVODB1_pmon_9297_i195199.trc
    Sat Jan 22 16:25:10 2011
    Trace dumping is performing id=[cdmp_20110122162510]
    Sat Jan 22 16:25:12 2011
    Errors in file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_pmon_9297.trc (incident=195200):
    ORA-00600: internal error code, arguments: [17183], [0x14E251C38], [], [], [], [], [], []
    Incident details in: /u01/app/oracle/diag/rdbms/evodb/EVODB1/incident/incdir_195200/EVODB1_pmon_9297_i195200.trc
    Sat Jan 22 16:25:23 2011
    DBRM (ospid: 9310): terminating the instance due to error 472
    Sat Jan 22 16:25:23 2011
    System state dump is made for local instance
    System State dumped to trace file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_diag_16320.trc
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (3724_47445977853072)
    Sat Jan 22 16:25:23 2011
    SUCCESS: diskgroup ARCHIVELOG was dismounted
    Sat Jan 22 16:14:22 2011
    DDE: Problem Key 'ORA 600 [17183]' was completely flood controlled (0x6)
    Further messages for this problem key will be suppressed for up to 10 minutes
    Sat Jan 22 16:15:56 2011
    Thread 1 advanced to log sequence 171322
    Current log# 18 seq# 171322 mem# 0: +ONLINELOG/evodb/onlinelog/group_18.264.729348055
    Sat Jan 22 16:15:56 2011
    SUCCESS: diskgroup ARCHIVELOG was mounted
    Sat Jan 22 16:16:15 2011
    SUCCESS: diskgroup ARCHIVELOG was dismounted
    Sat Jan 22 16:20:52 2011
    Thread 1 advanced to log sequence 171323
    Current log# 12 seq# 171323 mem# 0: +ONLINELOG/evodb/onlinelog/group_12.267.729333861
    Sat Jan 22 16:20:53 2011
    SUCCESS: diskgroup ARCHIVELOG was mounted
    Sat Jan 22 16:21:14 2011
    SUCCESS: diskgroup ARCHIVELOG was dismounted
    Sat Jan 22 16:24:23 2011
    DDE: Problem Key 'ORA 600 [17183]' was completely flood controlled (0x2)
    Further messages for this problem key will be suppressed for up to 10 minutes
    Sat Jan 22 16:24:38 2011
    Errors in file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_pmon_9297.trc (incident=195198):
    ORA-00600: internal error code, arguments: [17183], [0x14E251C38], [], [], [], [], [], []
    Incident details in: /u01/app/oracle/diag/rdbms/evodb/EVODB1/incident/incdir_195198/EVODB1_pmon_9297_i195198.trc
    Sat Jan 22 16:24:47 2011
    Trace dumping is performing id=[cdmp_20110122162447]
    Sat Jan 22 16:24:50 2011
    Sweep Incident[195198]: completed
    Sat Jan 22 16:25:00 2011
    Errors in file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_pmon_9297.trc (incident=195199):
    ORA-00600: internal error code, arguments: [17183], [0x14E251C38], [], [], [], [], [], []
    Incident details in: /u01/app/oracle/diag/rdbms/evodb/EVODB1/incident/incdir_195199/EVODB1_pmon_9297_i195199.trc
    Sat Jan 22 16:25:10 2011
    Trace dumping is performing id=[cdmp_20110122162510]
    Sat Jan 22 16:25:12 2011
    Errors in file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_pmon_9297.trc (incident=195200):
    ORA-00600: internal error code, arguments: [17183], [0x14E251C38], [], [], [], [], [], []
    Incident details in: /u01/app/oracle/diag/rdbms/evodb/EVODB1/incident/incdir_195200/EVODB1_pmon_9297_i195200.trc
    Sat Jan 22 16:25:23 2011
    DBRM (ospid: 9310): terminating the instance due to error 472
    Sat Jan 22 16:25:23 2011
    System state dump is made for local instance
    System State dumped to trace file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_diag_16320.trc
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (3724_47445977853072)
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (3650_47673206152336)
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (19709_47962498952336)
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (3715_47239602621584)
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (3718_47120460606608)
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (3646_47899813694608)
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (3671_47700480982160)
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (3637_47223057443984)
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (3695_47109627223184)
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (3679_47262512999568)
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (3733_46976527121552)
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (3667_47474016554128)
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (3669_47585103858832)
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (3658_47992462539920)
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (3654_47736794324112)
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (3686_47295241862288)
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (3706_47055349939344)
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (3630_47331091332240)
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (3713_47194150415504)
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (3726_47289949199504)
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (3677_47842172337296)
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (19131_47557370236048)
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (27091_47096969530512)
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (3688_47422639070352)
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (3652_47944422886544)
    Sat Jan 22 16:25:23 2011
    ORA-1092 : opiodr aborting process unknown ospid (3648_47426149817488)
    Sat Jan 22 16:25:24 2011
    Errors in file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_j005_9360.trc:
    ORA-00472: PMON  process terminated with error
    ORA-1092 : opidrv aborting process J005 ospid (9360_47420959304848)
    Sat Jan 22 16:25:24 2011
    ORA-1092 : opiodr aborting process unknown ospid (18393_47566684101776)
    Sat Jan 22 16:25:24 2011
    Errors in file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_j003_7054.trc:
    ORA-00472: PMON process terminated with error
    ORA-1092 : opidrv aborting process J003 ospid (7054_47244451724432)
    Sat Jan 22 16:25:24 2011
    ORA-1092 : opiodr aborting process unknown ospid (19719_47840732109968)
    Sat Jan 22 16:25:24 2011
    ORA-1092 : opiodr aborting process unknown ospid (22184_47575322619024)
    Sat Jan 22 16:25:24 2011
    ORA-1092 : opiodr aborting process unknown ospid (19721_47679094451344)
    Sat Jan 22 16:25:24 2011
    ORA-1092 : opiodr aborting process unknown ospid (30170_47106572815504)
    Sat Jan 22 16:25:24 2011
    ORA-472 : opidrv aborting process S000 ospid (9367_46990649617552)
    Sat Jan 22 16:25:24 2011
    Errors in file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_j002_7052.trc:
    ORA-00472: PMON process terminated with error
    ORA-1092 : opidrv aborting process J002 ospid (7052_47411695402128)
    Sat Jan 22 16:25:24 2011
    ORA-1092 : opiodr aborting process unknown ospid (18493_47469336738960)
    Sat Jan 22 16:25:24 2011
    Errors in file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_j006_10011.trc:
    ORA-00472: PMON process terminated with error
    ORA-1092 : opidrv aborting process J006 ospid (10011_47836408102032)
    Sat Jan 22 16:25:24 2011
    ORA-1092 : opiodr aborting process unknown ospid (19714_47958179912848)
    Sat Jan 22 16:25:24 2011
    Errors in file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_j007_10289.trc:
    ORA-00472: PMON process terminated with error
    ORA-1092 : opidrv aborting process J007 ospid (10289_47356480347280)
    Sat Jan 22 16:25:24 2011
    ORA-1092 : opiodr aborting process unknown ospid (9076_47765392362640)
    Sat Jan 22 16:25:24 2011
    ORA-1092 : opitsk aborting process
    Sat Jan 22 16:25:25 2011
    ORA-1092 : opiodr aborting process unknown ospid (22167_47949919177872)
    Sat Jan 22 16:25:25 2011
    ORA-1092 : opiodr aborting process unknown ospid (24320_47408077532304)
    Sat Jan 22 16:25:25 2011
    ORA-1092 : opiodr aborting process unknown ospid (22130_47998542101648)
    Sat Jan 22 16:25:25 2011
    ORA-472 : opidrv aborting process N000 ospid (14774_47466200525968)
    Sat Jan 22 16:25:25 2011
    Errors in file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_q000_14701.trc:
    ORA-00472: PMON process terminated with error
    ORA-1092 : opidrv aborting process Q000 ospid (14701_47693688122512)
    Sat Jan 22 16:25:25 2011
    ORA-1092 : opiodr aborting process unknown ospid (18386_47900707175568)
    Sat Jan 22 16:25:25 2011
    ORA-1092 : opiodr aborting process unknown ospid (27553_46921398619280)
    Sat Jan 22 16:25:26 2011
    Errors in file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_j001_7050.trc:
    ORA-00472: PMON process terminated with error
    ORA-1092 : opidrv aborting process J001 ospid (7050_47290016251024)
    Sat Jan 22 16:25:26 2011
    Errors in file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_o001_5297.trc:
    ORA-00472: PMON process terminated with error
    ORA-1092 : opidrv aborting process O001 ospid (5297_47702958673040)
    Sat Jan 22 16:25:26 2011
    Errors in file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_o000_5277.trc:
    ORA-00472: PMON process terminated with error
    ORA-1092 : opidrv aborting process O000 ospid (5277_47599668255888)
    Sat Jan 22 16:25:26 2011
    ORA-1092 : opiodr aborting process unknown ospid (19745_47560156560528)
    Sat Jan 22 16:25:26 2011
    ORA-1092 : opiodr aborting process unknown ospid (27044_47388543339664)
    Sat Jan 22 16:25:26 2011
    Errors in file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_j004_7061.trc:
    ORA-00472: PMON process terminated with error
    ORA-1092 : opidrv aborting process J004 ospid (7061_47341656689808)
    Sat Jan 22 16:25:27 2011
    ORA-1092 : opiodr aborting process unknown ospid (14431_47662297960112)
    Sat Jan 22 16:25:27 2011
    ORA-1092 : opiodr aborting process unknown ospid (14433_47826608635568)
    Sat Jan 22 16:25:31 2011
    ORA-1092 : opitsk aborting process
    Sat Jan 22 16:25:32 2011
    License high water mark = 269
    Instance terminated by DBRM, pid = 9310
    USER (ospid: 10539): terminating the instance
    Instance terminated by USER, pid = 10539
    Sat Jan 22 16:25:49 2011
    Starting ORACLE instance (normal)
    This is the trace /u01/app/oracle/diag/rdbms/evodb/EVODB1/incident/incdir_195200/EVODB1_pmon_9297_i195200.trc
    (the one regarding the error DBRM (ospid: 9310): terminating the instance due to error 472)
    Oracle Database 11g Release 11.1.0.6.0 - 64bit Production
    With the Real Application Clusters option
    ORACLE_HOME = /u01/app/oracle/product/11.1.0/db1
    System name:    Linux
    Node name:      dcsrv-evodb01
    Release:        2.6.18-164.10.1.el5
    Version:        #1 SMP Wed Dec 30 18:35:28 EST 2009
    Machine:        x86_64
    Instance name: EVODB1
    Redo thread mounted by this instance: 1
    Oracle process number: 2
    Unix process pid: 9297, image: oracle@dcsrv-evodb01 (PMON)
    *** 2011-01-22 16:25:12.984
    *** SESSION ID:(884.1) 2011-01-22 16:25:12.984
    *** CLIENT ID:() 2011-01-22 16:25:12.984
    *** SERVICE NAME:(SYS$BACKGROUND) 2011-01-22 16:25:12.984
    *** MODULE NAME:() 2011-01-22 16:25:12.984
    *** ACTION NAME:() 2011-01-22 16:25:12.984
    Dump continued from file: /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_pmon_9297.trc
    ORA-00600: internal error code, arguments: [17183], [0x14E251C38], [], [], [], [], [], []
    +========= Dump for incident 195200 (ORA 600 [17183]) ========+
    ----- Beginning of Customized Incident Dump(s) -----
    And this is the trace /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_j005_9360.trc
    (the one regarding the error ORA-00472: PMON process terminated with error)
    Trace file /u01/app/oracle/diag/rdbms/evodb/EVODB1/trace/EVODB1_j005_9360.trc
    Oracle Database 11g Release 11.1.0.6.0 - 64bit Production
    With the Real Application Clusters option
    ORACLE_HOME = /u01/app/oracle/product/11.1.0/db1
    System name:    Linux
    Node name:      dcsrv-evodb01
    Release:        2.6.18-164.10.1.el5
    Version:        #1 SMP Wed Dec 30 18:35:28 EST 2009
    Machine:        x86_64
    Instance name: EVODB1
    Redo thread mounted by this instance: 1
    Oracle process number: 95
    Unix process pid: 9360, image: oracle@dcsrv-evodb01 (J005)
    *** 2011-01-22 16:25:24.057
    *** SESSION ID:(649.45863) 2011-01-22 16:25:24.057
    *** CLIENT ID:() 2011-01-22 16:25:24.057
    *** SERVICE NAME:(SYS$USERS) 2011-01-22 16:25:24.057
    *** MODULE NAME:() 2011-01-22 16:25:24.057
    *** ACTION NAME:() 2011-01-22 16:25:24.057
    error 472 in job queue process
    ORA-00472: PMON  process terminated with error
    My questions are:
    - how can I discover the nature of the ORA-00600: internal error code, arguments: [17183], [0x14E251C38], [], [], [], [], [], []?
    - what does error 472 in job queue proces mean?
    - is it normal the the error 472 in job queue proces crash the instance?
    Thanks in advanca for any kind of help!
    Samuel

    >
    how can I discover the nature of the ORA-00600: internal error code, arguments: [17183], [0x14E251C38], [], [], [], [], [], []?
    >
    ORA-00600 should be reported to oracle. Only they know the meaning.
    Regards,
    - wiZ

  • SMON process terminated with error

    Hi,
    Oracle Version:10.2.0.3
    Operating system:Linux
    I alert log file i found that smon process was terminated with error.Below is the contents of the alert log file.
    Fri Dec 31 05:12:32 2010
    sculkget: failed to lock /home/oracle/oracle/oracle/product/10.2.0/db_1/dbs/lkinstqfundeca exclusive
    sculkget: lock held by PID: 9127
    Oracle Instance Shutdown operation failed. Another process may be attempting to startup or shutdown this Instance.
    Failed to acquire instance startup/shutdown serialization primitive
    Fri Dec 31 05:56:23 2010
    sculkget: failed to lock /home/oracle/oracle/oracle/product/10.2.0/db_1/dbs/lkinstqfundeca exclusive
    sculkget: lock held by PID: 9127
    Oracle Instance Shutdown operation failed. Another process may be attempting to startup or shutdown this Instance.
    Failed to acquire instance startup/shutdown serialization primitive
    Fri Dec 31 06:40:53 2010
    WARNING: inbound connection timed out (ORA-3136)
    Fri Dec 31 06:40:53 2010
    WARNING: inbound connection timed out (ORA-3136)
    Fri Dec 31 06:40:53 2010
    WARNING: inbound connection timed out (ORA-3136)
    Fri Dec 31 06:40:53 2010
    WARNING: inbound connection timed out (ORA-3136)
    Fri Dec 31 06:40:53 2010
    WARNING: inbound connection timed out (ORA-3136)
    Fri Dec 31 06:40:53 2010
    WARNING: inbound connection timed out (ORA-3136)
    Fri Dec 31 06:40:53 2010
    WARNING: inbound connection timed out (ORA-3136)
    Fri Dec 31 06:40:53 2010
    MMNL absent for 7904 secs; Foregrounds taking over
    Fri Dec 31 06:40:53 2010
    Starting background process EMN0
    Fri Dec 31 06:40:53 2010
    The value (30) of MAXTRANS parameter ignored.
    Fri Dec 31 06:40:53 2010
    Thread 1 cannot allocate new log, sequence 62004
    Checkpoint not complete
      Current log# 1 seq# 62003 mem# 0: /u02/oradata/qfundeca/redo01a.log
      Current log# 1 seq# 62003 mem# 1: /u03/oradata/qfundeca/redo01b.log
    EMN0 started with pid=29, OS id=7176
    Fri Dec 31 06:40:53 2010
    Shutting down instance: further logons disabled
    Fri Dec 31 06:40:53 2010
    ksvcreate: Process(m001) creation failed
    Fri Dec 31 06:40:53 2010
    kkjcre1p: unable to spawn jobq slave process, error 1089
    Fri Dec 31 06:40:53 2010
    kkjcre1p: unable to spawn jobq slave process, error 1089
    Fri Dec 31 06:40:54 2010
    Stopping background process QMNC
    Fri Dec 31 06:40:54 2010
    Stopping background process CJQ0
    Fri Dec 31 06:40:54 2010
    Thread 1 cannot allocate new log, sequence 62004
    Private strand flush not complete
      Current log# 1 seq# 62003 mem# 0: /u02/oradata/qfundeca/redo01a.log
      Current log# 1 seq# 62003 mem# 1: /u03/oradata/qfundeca/redo01b.log
    Fri Dec 31 06:40:56 2010
    Stopping background process MMNL
    Fri Dec 31 06:40:56 2010
    Thread 1 advanced to log sequence 62004
      Current log# 2 seq# 62004 mem# 0: /u02/oradata/qfundeca/redo02a.log
      Current log# 2 seq# 62004 mem# 1: /u03/oradata/qfundeca/redo02b.log
    Fri Dec 31 06:40:57 2010
    Starting background process QMNC
    Fri Dec 31 06:40:57 2010
    Errors in file /home/oracle/oracle/admin/qfundeca/udump/qfundeca_ora_3065.trc:
    ORA-00443: background process "QMNC" did not start
    Fri Dec 31 06:41:04 2010
    Stopping background process MMON
    Fri Dec 31 06:41:05 2010
    Errors in file /home/oracle/oracle/admin/qfundeca/udump/qfundeca_ora_9127.trc:
    ORA-00604: error occurred at recursive SQL level 1
    ORA-12663: Services required by client not available on the server
    ORA-36961: Oracle OLAP is not available.
    ORA-06512: at "SYS.OLAPIHISTORYRETENTION", line 1
    ORA-06512: at line 15
    Fri Dec 31 06:41:05 2010
    Shutting down instance (immediate)
    License high water mark = 93
    Fri Dec 31 06:41:05 2010
    Stopping Job queue slave processes
    Fri Dec 31 06:41:05 2010
    Job queue slave processes stopped
    Fri Dec 31 06:41:57 2010
    All dispatchers and shared servers shutdown
    Fri Dec 31 06:41:59 2010
    ALTER DATABASE CLOSE NORMAL
    Fri Dec 31 06:41:59 2010
    SMON: disabling tx recovery
    SMON: disabling cache recovery
    Fri Dec 31 06:42:00 2010
    Shutting down archive processes
    Archiving is disabled
    Fri Dec 31 06:42:05 2010
    ARCH shutting down
    ARC1: Archival stopped
    Fri Dec 31 06:42:11 2010
    ARCH shutting down
    ARC0: Archival stopped
    Fri Dec 31 06:42:12 2010
    Thread 1 closed at log sequence 62004
    Successful close of redo thread 1
    Fri Dec 31 06:42:12 2010
    Completed: ALTER DATABASE CLOSE NORMAL
    Fri Dec 31 06:42:12 2010
    ALTER DATABASE DISMOUNT
    Completed: ALTER DATABASE DISMOUNT
    ARCH: Archival disabled due to shutdown: 1089
    Shutting down archive processes
    Archiving is disabled
    Archive process shutdown avoided: 0 active
    ARCH: Archival disabled due to shutdown: 1089
    Shutting down archive processes
    Archiving is disabled
    Archive process shutdown avoided: 0 active
    Fri Dec 31 06:49:52 2010
    Starting ORACLE instance (normal)
    LICENSE_MAX_SESSION = 0
    LICENSE_SESSIONS_WARNING = 0
    Picked latch-free SCN scheme 2
    Autotune of undo retention is turned on.
    IMODE=BR
    ILAT =27
    LICENSE_MAX_USERS = 0
    SYS auditing is disabled
    ksdpec: called for event 13740 prior to event group initialization
    Starting up ORACLE RDBMS Version: 10.2.0.3.0.
    System parameters with non-default values:
      processes                = 150
      sessions                 = 250
      __shared_pool_size       = 385875968
      __large_pool_size        = 16777216
      __java_pool_size         = 16777216
      __streams_pool_size      = 33554432
      streams_pool_size        = 33554432
      sga_target               = 1610612736
      control_files            = /u02/oradata/qfundeca/control01.ctl, /u02/oradata/qfundeca/control02.ctl, /u03/oradata/qfundeca/control03.ctl
      db_block_size            = 8192
      __db_cache_size          = 1140850688
      compatible               = 10.2.0.3.0
      log_archive_dest_1       = LOCATION=/u02/oradata/arch/qfundeca
      log_archive_format       = %t_%s_%r.dbf
      archive_lag_target       = 2700
      db_file_multiblock_read_count= 16
      db_recovery_file_dest    = /u02/oradata/flash_recovery_area
      db_recovery_file_dest_size= 21474836480
      standby_file_management  = AUTO
      undo_management          = AUTO
      undo_tablespace          = UNDOTBS1
      remote_login_passwordfile= EXCLUSIVE
      db_domain                =
      dispatchers              = (PROTOCOL=TCP) (SERVICE=qfundecaXDB)
      job_queue_processes      = 17
      cursor_sharing           = EXACT
      background_dump_dest     = /home/oracle/oracle/admin/qfundeca/bdump
      user_dump_dest           = /home/oracle/oracle/admin/qfundeca/udump
      core_dump_dest           = /home/oracle/oracle/admin/qfundeca/cdump
      audit_file_dest          = /home/oracle/oracle/admin/qfundeca/adump
      db_name                  = qfundeca
      open_cursors             = 300
      pga_aggregate_target     = 839909376
    PMON started with pid=2, OS id=9591
    PSP0 started with pid=3, OS id=9593
    MMAN started with pid=4, OS id=9595
    DBW0 started with pid=5, OS id=9597
    LGWR started with pid=6, OS id=9599
    CKPT started with pid=7, OS id=9601
    SMON started with pid=8, OS id=9603
    RECO started with pid=9, OS id=9605
    CJQ0 started with pid=10, OS id=9607
    MMON started with pid=11, OS id=9609
    Fri Dec 31 06:49:54 2010
    starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
    MMNL started with pid=12, OS id=9611
    Fri Dec 31 06:49:54 2010
    starting up 1 shared server(s) ...
    Oracle Data Guard is not available in this edition of Oracle.
    Fri Dec 31 06:49:54 2010
    ALTER DATABASE   MOUNT
    Fri Dec 31 06:49:58 2010
    Setting recovery target incarnation to 1
    Fri Dec 31 06:49:58 2010
    Successful mount of redo thread 1, with mount id 256948178
    Fri Dec 31 06:49:58 2010
    Database mounted in Exclusive Mode
    Completed: ALTER DATABASE   MOUNT
    Fri Dec 31 06:49:58 2010
    ALTER DATABASE OPEN
    Fri Dec 31 06:49:58 2010
    LGWR: STARTING ARCH PROCESSES
    ARC0 started with pid=16, OS id=9619
    Fri Dec 31 06:49:58 2010
    ARC0: Archival started
    ARC1: Archival started
    LGWR: STARTING ARCH PROCESSES COMPLETE
    ARC1 started with pid=17, OS id=9621
    Fri Dec 31 06:49:58 2010
    Thread 1 opened at log sequence 62004
      Current log# 2 seq# 62004 mem# 0: /u02/oradata/qfundeca/redo02a.log
      Current log# 2 seq# 62004 mem# 1: /u03/oradata/qfundeca/redo02b.log
    Successful open of redo thread 1
    Fri Dec 31 06:49:58 2010
    ARC0: Becoming the 'no FAL' ARCH
    ARC0: Becoming the 'no SRL' ARCH
    Fri Dec 31 06:49:58 2010
    ARC1: Becoming the heartbeat ARCH
    Fri Dec 31 06:49:58 2010
    SMON: enabling cache recovery
    Fri Dec 31 06:49:58 2010
    Successfully onlined Undo Tablespace 1.
    Fri Dec 31 06:49:58 2010
    SMON: enabling tx recovery
    Fri Dec 31 06:49:58 2010
    Database Characterset is WE8ISO8859P1
    replication_dependency_tracking turned off (no async multimaster replication found)
    Starting background process QMNC
    QMNC started with pid=18, OS id=9623
    Fri Dec 31 06:49:59 2010
    Errors in file /home/oracle/oracle/admin/qfundeca/udump/qfundeca_ora_9617.trc:
    ORA-00604: error occurred at recursive SQL level 1
    ORA-12663: Services required by client not available on the server
    ORA-36961: Oracle OLAP is not available.
    ORA-06512: at "SYS.OLAPIHISTORYRETENTION", line 1
    ORA-06512: at line 15
    Fri Dec 31 06:50:00 2010
    Completed: ALTER DATABASE OPEN
    Fri Dec 31 06:50:00 2010
    db_recovery_file_dest_size of 20480 MB is 0.00% used. This is a
    user-specified limit on the amount of space that will be used by this
    database for recovery-related files, and does not reflect the amount of
    space available in the underlying filesystem or ASM diskgroup.
    Fri Dec 31 06:55:59 2010
    The value (30) of MAXTRANS parameter ignored.
    kupprdp: master process DM00 started with pid=36, OS id=12004
             to execute - SYS.KUPM$MCP.MAIN('QFUNDECA_PROD_BKP', 'QFUNDECAPROD', 'KUPC$C_1_20101231065559', 'KUPC$S_1_20101231065559', 0);
    kupprdp: worker process DW01 started with worker id=1, pid=37, OS id=12009
             to execute - SYS.KUPW$WORKER.MAIN('QFUNDECA_PROD_BKP', 'QFUNDECAPROD');
    Fri Dec 31 07:04:59 2010
    Thread 1 cannot allocate new log, sequence 62005
    Private strand flush not complete
      Current log# 2 seq# 62004 mem# 0: /u02/oradata/qfundeca/redo02a.log
      Current log# 2 seq# 62004 mem# 1: /u03/oradata/qfundeca/redo02b.log
    Thread 1 advanced to log sequence 62005
      Current log# 3 seq# 62005 mem# 0: /u02/oradata/qfundeca/redo03a.log
      Current log# 3 seq# 62005 mem# 1: /u03/oradata/qfundeca/redo03b.log
    Fri Dec 31 07:30:01 2010
    Thread 1 cannot allocate new log, sequence 62006
    Private strand flush not complete
      Current log# 3 seq# 62005 mem# 0: /u02/oradata/qfundeca/redo03a.log
      Current log# 3 seq# 62005 mem# 1: /u03/oradata/qfundeca/redo03b.log
    Thread 1 advanced to log sequence 62006
      Current log# 1 seq# 62006 mem# 0: /u02/oradata/qfundeca/redo01a.log
      Current log# 1 seq# 62006 mem# 1: /u03/oradata/qfundeca/redo01b.log
    Fri Dec 31 07:54:42 2010
    System State dumped to trace file /home/oracle/oracle/admin/qfundeca/udump/qfundeca_ora_24979.trc
    Fri Dec 31 07:55:06 2010
    System State dumped to trace file /home/oracle/oracle/admin/qfundeca/udump/qfundeca_ora_25057.trc
    Fri Dec 31 07:55:37 2010
    System State dumped to trace file /home/oracle/oracle/admin/qfundeca/udump/qfundeca_ora_25209.trc
    Fri Dec 31 07:59:33 2010
    System State dumped to trace file /home/oracle/oracle/admin/qfundeca/udump/qfundeca_ora_26237.trc
    Fri Dec 31 08:00:25 2010
    System State dumped to trace file /home/oracle/oracle/admin/qfundeca/udump/qfundeca_ora_26523.trc
    Fri Dec 31 08:00:40 2010
    System State dumped to trace file /home/oracle/oracle/admin/qfundeca/udump/qfundeca_ora_26909.trc
    Fri Dec 31 08:06:34 2010
    Errors in file /home/oracle/oracle/admin/qfundeca/bdump/qfundeca_pmon_9591.trc:
    *ORA-00474: SMON process terminated with error*
    *Fri Dec 31 08:06:34 2010*
    *PMON: terminating instance due to error 474*
    Instance terminated by PMON, pid = 9591
    Fri Dec 31 08:08:31 2010
    Starting ORACLE instance (normal)
    LICENSE_MAX_SESSION = 0
    LICENSE_SESSIONS_WARNING = 0
    Picked latch-free SCN scheme 2
    Autotune of undo retention is turned on.
    IMODE=BR
    ILAT =27
    LICENSE_MAX_USERS = 0
    SYS auditing is disabled
    ksdpec: called for event 13740 prior to event group initialization
    Starting up ORACLE RDBMS Version: 10.2.0.3.0.
    System parameters with non-default values:
      processes                = 150
      sessions                 = 250
      __shared_pool_size       = 234881024
      __large_pool_size        = 16777216
      __java_pool_size         = 16777216
      __streams_pool_size      = 33554432
      streams_pool_size        = 33554432
      sga_target               = 1610612736
      control_files            = /u02/oradata/qfundeca/control01.ctl, /u02/oradata/qfundeca/control02.ctl, /u03/oradata/qfundeca/control03.ctl
      db_block_size            = 8192
      __db_cache_size          = 1291845632
      compatible               = 10.2.0.3.0
      log_archive_dest_1       = LOCATION=/u02/oradata/arch/qfundeca
      log_archive_format       = %t_%s_%r.dbf
      archive_lag_target       = 2700
      db_file_multiblock_read_count= 16
      db_recovery_file_dest    = /u02/oradata/flash_recovery_area
      db_recovery_file_dest_size= 21474836480
      standby_file_management  = AUTO
      undo_management          = AUTO
      undo_tablespace          = UNDOTBS1
      remote_login_passwordfile= EXCLUSIVE
      db_domain                =
      dispatchers              = (PROTOCOL=TCP) (SERVICE=qfundecaXDB)
      job_queue_processes      = 17
      cursor_sharing           = EXACT
      background_dump_dest     = /home/oracle/oracle/admin/qfundeca/bdump
      user_dump_dest           = /home/oracle/oracle/admin/qfundeca/udump
      core_dump_dest           = /home/oracle/oracle/admin/qfundeca/cdump
      audit_file_dest          = /home/oracle/oracle/admin/qfundeca/adump
      db_name                  = qfundeca
      open_cursors             = 300
      pga_aggregate_target     = 839909376
    PMON started with pid=2, OS id=648
    PSP0 started with pid=3, OS id=650
    MMAN started with pid=4, OS id=652
    DBW0 started with pid=5, OS id=654
    LGWR started with pid=6, OS id=656
    CKPT started with pid=7, OS id=658
    SMON started with pid=8, OS id=660
    RECO started with pid=9, OS id=662
    CJQ0 started with pid=10, OS id=664
    MMON started with pid=11, OS id=666
    Fri Dec 31 08:08:32 2010
    starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
    MMNL started with pid=12, OS id=668
    Fri Dec 31 08:08:32 2010
    starting up 1 shared server(s) ...
    Oracle Data Guard is not available in this edition of Oracle.
    Fri Dec 31 08:08:32 2010
    ALTER DATABASE   MOUNT
    Fri Dec 31 08:08:36 2010
    Setting recovery target incarnation to 1
    Fri Dec 31 08:08:36 2010
    Successful mount of redo thread 1, with mount id 257031232
    Fri Dec 31 08:08:36 2010
    Database mounted in Exclusive Mode
    Completed: ALTER DATABASE   MOUNT
    Fri Dec 31 08:08:36 2010
    ALTER DATABASE OPEN
    Fri Dec 31 08:08:37 2010
    Beginning crash recovery of 1 threads
    Fri Dec 31 08:08:37 2010
    Started redo scan
    Fri Dec 31 08:08:37 2010
    Completed redo scan
    31201 redo blocks read, 754 data blocks need recovery
    Fri Dec 31 08:08:38 2010
    Started redo application at
    Thread 1: logseq 62004, block 66343
    Fri Dec 31 08:08:38 2010
    Recovery of Online Redo Log: Thread 1 Group 2 Seq 62004 Reading mem 0
      Mem# 0: /u02/oradata/qfundeca/redo02a.log
      Mem# 1: /u03/oradata/qfundeca/redo02b.log
    Fri Dec 31 08:08:38 2010
    Recovery of Online Redo Log: Thread 1 Group 3 Seq 62005 Reading mem 0
      Mem# 0: /u02/oradata/qfundeca/redo03a.log
      Mem# 1: /u03/oradata/qfundeca/redo03b.logHow to know what is the reason for this one.On that day the database hangs two time as i mentioned in my previous post.
    Thanks & Regards,
    Poorna Prasad.

    Hi Chinar,
    Here is the contents for the file /home/oracle/oracle/admin/qfundeca/bdump/qfundeca_pmon_9591.trc
    /home/oracle/oracle/admin/qfundeca/bdump/qfundeca_pmon_9591.trc
    Oracle Database 10g Release 10.2.0.3.0 - Production
    ORACLE_HOME = /home/oracle/oracle/oracle/product/10.2.0/db_1
    System name:     Linux
    Node name:     virqts2ora001.localdomain
    Release:     2.6.9-5.ELsmp
    Version:     #1 SMP Wed Jan 5 19:30:39 EST 2005
    Machine:     i686
    Instance name: qfundeca
    Redo thread mounted by this instance: 1
    Oracle process number: 2
    Unix process pid: 9591, image: [email protected] (PMON)
    *** 2010-12-31 08:06:34.723
    *** SERVICE NAME:(SYS$BACKGROUND) 2010-12-31 08:06:34.723
    *** SESSION ID:(250.1) 2010-12-31 08:06:34.723
    Background process SMON found dead
    Oracle pid = 8
    OS pid (from detached process) = 9603
    OS pid (from process state) = 9603
    dtp = 0x2000ed88, proc = 0x7f3528a0
    Dump of memory from 0x2000ED88 to 0x2000EDB4
    2000ED80                   0000007C 7F3528A0          [|....(5.]
    2000ED90 00000000 00000000 4E4F4D53 00000200  [........SMON....]
    2000EDA0 00002583 FFFFFFFF 00000001 5634F590  [.%............4V]
    2000EDB0 00040081                             [....]           
    Dump of memory from 0x7F3528A0 to 0x7F352E58
    7F3528A0 00000102 00000000 00000000 00000000  [................]
    7F3528B0 00000000 7F5FDF30 7F95787C 7F4C61E0  [....0._.|x...aL.]
    7F3528C0 7F956AE4 00000000 7F956B48 7F956B48  [.j......Hk..Hk..]
    7F3528D0 7F957870 00001601 7F4AA968 7F4C61E0  [px......h.J..aL.]
    7F3528E0 00000008 7F4C7430 7F4C75B4 00000000  [....0tL..uL.....]
    7F3528F0 7F5FCE18 7F5FDF48 00000000 00000000  [.._.H._.........]
    7F352900 00000000 00000000 00000000 00000000  [................]
            Repeat 2 times
    7F352930 00000000 00000000 00000000 000C0000  [................]
    7F352940 00000000 00080000 00000000 00050000  [................]
    7F352950 00000000 00040000 00000000 00040000  [................]
    7F352960 00000000 00080000 00000000 000C0000  [................]
    7F352970 00000000 000C0000 00000000 00050000  [................]
    7F352980 00000000 00110000 00000000 000F0000  [................]
    7F352990 00000000 00000000 00000000 00000000  [................]
            Repeat 3 times
    7F3529D0 00000002 7F3529D4 7F3529D4 00000000  [.....)5..)5.....]
    7F3529E0 00000001 00000000 7F3529E8 7F3529E8  [.........)5..)5.]
    7F3529F0 00000000 00000000 00000000 00000000  [................]
            Repeat 1 times
    7F352A10 00000021 00000018 00000095 00000015  [!...............]
    7F352A20 7F366E98 7F351D30 00010000 00000000  [.n6.0.5.........]
    7F352A30 00000000 00000000 00000000 00000000  [................]
            Repeat 1 times
    7F352A50 00000000 00000000 00002583 00000000  [.........%......]
    7F352A60 00000000 00000000 00000000 00000000  [................]
            Repeat 2 times
    7F352A90 00000000 00000000 7F3528A0 00000000  [.........(5.....]
    7F352AA0 00000000 00000000 00000000 00000000  [................]
            Repeat 6 times
    7F352B10 00000000 7F352B14 7F352B14 00000000  [.....+5..+5.....]
    7F352B20 00000000 00000001 00000000 00000000  [................]
    7F352B30 3461800E 0000000C 00002583 FFFFFFFF  [..a4.....%......]
    7F352B40 B74796C0 00000000 00000000 00000000  [..G.............]
    7F352B50 00000000 00000000 00000000 00000000  [................]
    7F352B60 00000000 00000000 00000008 FFFFFFFF  [................]
    7F352B70 00000000 00000000 00000000 00000000  [................]
            Repeat 7 times
    7F352BF0 00000000 00000000 6361726F 0000656C  [........oracle..]
    7F352C00 00000000 00000000 00000000 00000000  [................]
    7F352C10 00000000 00000000 00000006 71726976  [............virq]
    7F352C20 6F327374 30306172 6F6C2E31 646C6163  [ts2ora001.locald]
    7F352C30 69616D6F 0000006E 00000000 00000000  [omain...........]
    7F352C40 00000000 00000000 00000000 00000000  [................]
    7F352C50 00000000 00000000 00000000 00000019  [................]
    7F352C60 4E4B4E55 004E574F 00000000 00000000  [UNKNOWN.........]
    7F352C70 00000000 00000000 00000000 00000000  [................]
    7F352C80 00000008 33303639 00000000 00000000  [....9603........]
    7F352C90 00000000 00000000 00000000 00000004  [................]
    7F352CA0 6361726F 7640656C 74717269 726F3273  [oracle@virqts2or]
    7F352CB0 31303061 636F6C2E 6F646C61 6E69616D  [a001.localdomain]
    7F352CC0 4D532820 00294E4F 00000000 00000000  [ (SMON).........]
    7F352CD0 00000027 00000002 00000001 7864736B  ['...........ksdx]
    7F352CE0 6B747366 312B2928 6B2D3C39 63786473  [fstk()+19<-ksdxc]
    7F352CF0 2B292862 31323331 73732D3C 65737570  [b()+1321<-sspuse]
    7F352D00 2B292872 3C323031 3930302D 41374131  [r()+102<-0091A7A]
    7F352D10 00000032 00000000 00000000 00000000  [2...............]
    7F352D20 00000000 00000000 00000000 00000000  [................]
            Repeat 4 times
    7F352D70 00000000 00000000 00000000 00060200  [................]
    7F352D80 0000691D FFFFFFFF 00000000 00000000  [.i..............]
    7F352D90 7F38693C 7F35334C 7F3527DC 00000000  [<i8.L35..'5.....]
    7F352DA0 7F5D3CEC 00000000 00000000 00000000  [.<].............]
    7F352DB0 00000000 00000000 00000000 00000000  [................]
    7F352DC0 7F352DC0 7F352DC0 001B0000 00130000  [.-5..-5.........]
    7F352DD0 00029615 0007CBB1 001E9615 0000EB2C  [............,...]
    7F352DE0 00000000 00005F28 00000000 0002C440  [....(_......@...]
    7F352DF0 00000000 00000814 00000000 00000088  [................]
    7F352E00 00000000 00000814 00000000 00000000  [................]
    7F352E10 00000000 00000000 00000000 00000000  [................]
            Repeat 3 times
    7F352E50 00000000 00000003                    [........]       
    error 474 detected in background process
    ORA-00474: SMON process terminated with error

Maybe you are looking for