Broadcast On Commit acks

Do you know what BOC acks (Broadcast On Commit acks) means or any document reference for this?
Thx..

Has someone (vendor/consultant/manager) told you that
your DataGuardIt was oracle support who told me to apply this patch. I have a TAR on this issue. I also asked responsible guy about what i asked here through TAR request, not returned yet.
i think i am right to suspect why oracle support wanted me to apply this patch for a dataguard issue.
Thanks for your valuable input Dion_Cho & Hemant K Chitale.

Similar Messages

  • SPA504g and WinSVR DHCP

    I have a strange problem and would appreciate help.
    I have setup DHCP on windows server 2012. I have created scope policy based on Vendor class. The network is based on Linksys srw224g4 and Cisco sf300 switches.
    When I connect Linksys SPA942 or SPA962 phone, they negotiate with dhcp, get the address and then go further on to the phone central (with option 66 and 150). But when I connect Cisco SPA504g it stays in "Initializing Network" state.
    I have tested the comms with Wireshark and what I saw was:
    1. dhcp request broadcast from SPA504G
    2. dhcp NAK broadcast
    3. dhcp ACK to SPA504G
    After few seconds it starts all over again, with phone not taking the lease.
    Any idea why would SPA504G do that ?
    Thx

    Your description is not so good. Who issued NAK mentioned in [2] ? The DHCP server sends [3] and I assume the server didn't issued two replies to one request. Also, we know nothing about content of neither NAK or ACK packet.
    Please attach the Wireshark file here, we will look at it.

  • Database is not starting after RAC installation

    Hi All,
    I have installed 10g R2 2-node RAC on RHEL4 without any problem. But after the installation the database is not starting and i'm not able to connect. It is showing the below error..
    ERROR:
    ORA-01034: ORACLE not available
    ORA-27101: shared memory realm does not exist
    Linux Error: 2: No such file or directory
    Can anybody help what might be the problem?
    Thanks,
    Praveen

    Hi,
    I'm giving the contents of alter log file from the first node...
    Mon Feb 19 16:45:59 2007
    Starting ORACLE instance (normal)
    LICENSE_MAX_SESSION = 0
    LICENSE_SESSIONS_WARNING = 0
    Interface type 1 eth1 10.250.90.0 configured from OCR for use as a cluster interconnect
    Interface type 1 eth0 10.250.90.0 configured from OCR for use as a public interface
    Shared memory segment for instance monitoring created
    Picked latch-free SCN scheme 2
    Autotune of undo retention is turned on.
    IMODE=BR
    ILAT =36
    LICENSE_MAX_USERS = 0
    SYS auditing is disabled
    ksdpec: called for event 13740 prior to event group initialization
    Starting up ORACLE RDBMS Version: 10.2.0.2.0.
    System parameters with non-default values:
    processes = 300
    sessions = 335
    sga_max_size = 1073741824
    sga_target = 1073741824
    db_block_size = 8192
    compatible = 10.2.0.2.0
    log_archive_dest_1 = LOCATION=+BSA_DATA1/bsaprod/
    log_archive_format = %t_%s_%r.dbf
    db_file_multiblock_read_count= 16
    cluster_database_instances= 1
    db_create_file_dest = +ORCL_DATA1
    instance_number = 1
    undo_management = AUTO
    undo_tablespace = UNDOTBS1
    remote_login_passwordfile= EXCLUSIVE
    db_domain =
    dispatchers = (PROTOCOL=TCP) (SERVICE=bsaprodXDB)
    remote_listener = LISTENERS_BSAPROD
    job_queue_processes = 10
    background_dump_dest = /u01/app/oracle/oracle/admin/bsaprod/bdump
    user_dump_dest = /u01/app/oracle/oracle/admin/bsaprod/udump
    core_dump_dest = /u01/app/oracle/oracle/admin/bsaprod/cdump
    audit_file_dest = /u01/app/oracle/oracle/admin/bsaprod/adump
    db_name = bsaprod
    open_cursors = 300
    pga_aggregate_target = 262144000
    Cluster communication is configured to use the following interface(s) for this instance
    10.250.90.107
    Mon Feb 19 16:46:00 2007
    cluster interconnect IPC version:Oracle UDP/IP (generic)
    IPC Vendor 1 proto 2
    PMON started with pid=2, OS id=31282
    DIAG started with pid=3, OS id=31284
    PSP0 started with pid=4, OS id=31286
    LMON started with pid=5, OS id=31300
    LMD0 started with pid=6, OS id=31315
    MMAN started with pid=7, OS id=31317
    DBW0 started with pid=8, OS id=31319
    LGWR started with pid=9, OS id=31321
    CKPT started with pid=10, OS id=31323
    SMON started with pid=11, OS id=31325
    RECO started with pid=12, OS id=31327
    CJQ0 started with pid=13, OS id=31329
    MMON started with pid=14, OS id=31331
    Mon Feb 19 16:46:00 2007
    starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
    MMNL started with pid=15, OS id=31333
    Mon Feb 19 16:46:00 2007
    starting up 1 shared server(s) ...
    Mon Feb 19 16:46:00 2007
    lmon registered with NM - instance id 1 (internal mem no 0)
    Mon Feb 19 16:46:01 2007
    Reconfiguration started (old inc 0, new inc 2)
    List of nodes:
    0
    Global Resource Directory frozen
    * allocate domain 0, invalid = TRUE
    Communication channels reestablished
    Master broadcasted resource hash value bitmaps
    Non-local Process blocks cleaned out
    Resources and enqueues cleaned out
    Resources remastered 0
    Set master node info
    Submitted all remote-enqueue requests
    Dwn-cvts replayed, VALBLKs dubious
    All grantable enqueues granted
    Post SMON to start 1st pass IR
    Submitted all GCS remote-cache requests
    Post SMON to start 1st pass IR
    Reconfiguration complete
    Mon Feb 19 16:46:01 2007
    CREATE DATABASE "bsaprod"
    MAXINSTANCES 32
    MAXLOGHISTORY 1
    MAXLOGFILES 192
    MAXLOGMEMBERS 3
    MAXDATAFILES 1024
    DATAFILE SIZE 300M AUTOEXTEND ON NEXT 10240K MAXSIZE UNLIMITED
    EXTENT MANAGEMENT LOCAL
    SYSAUX DATAFILE SIZE 120M AUTOEXTEND ON NEXT 10240K MAXSIZE UNLIMITED
    SMALLFILE DEFAULT TEMPORARY TABLESPACE TEMP TEMPFILE SIZE 20M AUTOEXTEND ON NEXT 640K MAXSIZE UNLIMITED
    SMALLFILE UNDO TABLESPACE "UNDOTBS1" DATAFILE SIZE 200M AUTOEXTEND ON NEXT 5120K MAXSIZE UNLIMITED
    CHARACTER SET WE8ISO8859P1
    NATIONAL CHARACTER SET AL16UTF16
    LOGFILE GROUP 1 SIZE 51200K,
    GROUP 2 SIZE 51200K
    USER SYS IDENTIFIED BY *USER SYSTEM IDENTIFIED BY
    Mon Feb 19 16:46:01 2007
    Starting background process ASMB
    ASMB started with pid=19, OS id=31365
    Starting background process RBAL
    RBAL started with pid=20, OS id=31369
    Loaded ASM Library - Generic Linux, version 2.0.2 (KABI_V2) library for asmlib interface
    Mon Feb 19 16:46:05 2007
    SUCCESS: diskgroup ORCL_DATA1 was mounted
    SUCCESS: diskgroup ORCL_DATA1 was dismounted
    Mon Feb 19 16:46:06 2007
    SUCCESS: diskgroup ORCL_DATA1 was mounted
    Mon Feb 19 16:46:06 2007
    Database mounted in Exclusive Mode
    Mon Feb 19 16:46:09 2007
    Successful mount of redo thread 1, with mount id 3309011657
    Assigning activation ID 3309011657 (0xc53b82c9)
    Thread 1 opened at log sequence 1
    Current log# 1 seq# 1 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
    Successful open of redo thread 1
    Mon Feb 19 16:46:09 2007
    MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
    Mon Feb 19 16:46:09 2007
    SMON: enabling cache recovery
    Mon Feb 19 16:46:09 2007
    create tablespace SYSTEM datafile SIZE 300M AUTOEXTEND ON NEXT 10240K MAXSIZE UNLIMITED
    EXTENT MANAGEMENT LOCAL online
    Mon Feb 19 16:46:20 2007
    Completed: create tablespace SYSTEM datafile SIZE 300M AUTOEXTEND ON NEXT 10240K MAXSIZE UNLIMITED
    EXTENT MANAGEMENT LOCAL online
    Mon Feb 19 16:46:20 2007
    create rollback segment SYSTEM tablespace SYSTEM
    storage (initial 50K next 50K)
    Completed: create rollback segment SYSTEM tablespace SYSTEM
    storage (initial 50K next 50K)
    Mon Feb 19 16:46:24 2007
    CREATE SMALLFILE UNDO TABLESPACE UNDOTBS1 DATAFILE SIZE 200M AUTOEXTEND ON NEXT 5120K MAXSIZE UNLIMITED
    Mon Feb 19 16:46:31 2007
    Successfully onlined Undo Tablespace 1.
    Completed: CREATE SMALLFILE UNDO TABLESPACE UNDOTBS1 DATAFILE SIZE 200M AUTOEXTEND ON NEXT 5120K MAXSIZE UNLIMITED
    Mon Feb 19 16:46:31 2007
    create tablespace SYSAUX datafile SIZE 120M AUTOEXTEND ON NEXT 10240K MAXSIZE UNLIMITED
    EXTENT MANAGEMENT LOCAL SEGMENT SPACE MANAGEMENT AUTO online
    Completed: create tablespace SYSAUX datafile SIZE 120M AUTOEXTEND ON NEXT 10240K MAXSIZE UNLIMITED
    EXTENT MANAGEMENT LOCAL SEGMENT SPACE MANAGEMENT AUTO online
    Mon Feb 19 16:46:36 2007
    CREATE SMALLFILE TEMPORARY TABLESPACE TEMP TEMPFILE SIZE 20M AUTOEXTEND ON NEXT 640K MAXSIZE UNLIMITED
    Completed: CREATE SMALLFILE TEMPORARY TABLESPACE TEMP TEMPFILE SIZE 20M AUTOEXTEND ON NEXT 640K MAXSIZE UNLIMITED
    Mon Feb 19 16:46:36 2007
    ALTER DATABASE DEFAULT TEMPORARY TABLESPACE TEMP
    Completed: ALTER DATABASE DEFAULT TEMPORARY TABLESPACE TEMP
    Mon Feb 19 16:46:36 2007
    ALTER DATABASE DEFAULT TABLESPACE SYSTEM
    Completed: ALTER DATABASE DEFAULT TABLESPACE SYSTEM
    Mon Feb 19 16:46:38 2007
    SMON: enabling tx recovery
    Mon Feb 19 16:46:39 2007
    Threshold validation cannot be done before catproc is loaded.
    Threshold validation cannot be done before catproc is loaded.
    replication_dependency_tracking turned off (no async multimaster replication found)
    Starting background process QMNC
    QMNC started with pid=22, OS id=32303
    Mon Feb 19 16:46:39 2007
    Completed: CREATE DATABASE "bsaprod"
    MAXINSTANCES 32
    MAXLOGHISTORY 1
    MAXLOGFILES 192
    MAXLOGMEMBERS 3
    MAXDATAFILES 1024
    DATAFILE SIZE 300M AUTOEXTEND ON NEXT 10240K MAXSIZE UNLIMITED
    EXTENT MANAGEMENT LOCAL
    SYSAUX DATAFILE SIZE 120M AUTOEXTEND ON NEXT 10240K MAXSIZE UNLIMITED
    SMALLFILE DEFAULT TEMPORARY TABLESPACE TEMP TEMPFILE SIZE 20M AUTOEXTEND ON NEXT 640K MAXSIZE UNLIMITED
    SMALLFILE UNDO TABLESPACE "UNDOTBS1" DATAFILE SIZE 200M AUTOEXTEND ON NEXT 5120K MAXSIZE UNLIMITED
    CHARACTER SET WE8ISO8859P1
    NATIONAL CHARACTER SET AL16UTF16
    LOGFILE GROUP 1 SIZE 51200K,
    GROUP 2 SIZE 51200K
    USER SYS IDENTIFIED BY *USER SYSTEM IDENTIFIED BY
    Mon Feb 19 16:46:39 2007
    CREATE SMALLFILE UNDO TABLESPACE "UNDOTBS2" DATAFILE SIZE 200M AUTOEXTEND ON NEXT 5120K MAXSIZE UNLIMITED
    Completed: CREATE SMALLFILE UNDO TABLESPACE "UNDOTBS2" DATAFILE SIZE 200M AUTOEXTEND ON NEXT 5120K MAXSIZE UNLIMITED
    Mon Feb 19 16:46:46 2007
    CREATE SMALLFILE TABLESPACE "USERS" LOGGING DATAFILE SIZE 5M AUTOEXTEND ON NEXT 1280K MAXSIZE UNLIMITED EXTENT MANAGEMENT LOCAL SEGMENT SPACE MANAGEMENT AUTO
    Completed: CREATE SMALLFILE TABLESPACE "USERS" LOGGING DATAFILE SIZE 5M AUTOEXTEND ON NEXT 1280K MAXSIZE UNLIMITED EXTENT MANAGEMENT LOCAL SEGMENT SPACE MANAGEMENT AUTO
    Mon Feb 19 16:46:46 2007
    ALTER DATABASE DEFAULT TABLESPACE "USERS"
    Completed: ALTER DATABASE DEFAULT TABLESPACE "USERS"
    Mon Feb 19 16:47:05 2007
    Thread 1 advanced to log sequence 2
    Current log# 2 seq# 2 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
    Mon Feb 19 16:47:34 2007
    Thread 1 cannot allocate new log, sequence 3
    Checkpoint not complete
    Current log# 2 seq# 2 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
    Thread 1 advanced to log sequence 3
    Current log# 1 seq# 3 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
    Mon Feb 19 16:48:10 2007
    Thread 1 cannot allocate new log, sequence 4
    Checkpoint not complete
    Current log# 1 seq# 3 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
    Thread 1 advanced to log sequence 4
    Current log# 2 seq# 4 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
    Mon Feb 19 16:49:05 2007
    Thread 1 cannot allocate new log, sequence 5
    Checkpoint not complete
    Current log# 2 seq# 4 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
    Thread 1 advanced to log sequence 5
    Current log# 1 seq# 5 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
    Mon Feb 19 16:50:19 2007
    Thread 1 cannot allocate new log, sequence 6
    Checkpoint not complete
    Current log# 1 seq# 5 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
    Thread 1 advanced to log sequence 6
    Current log# 2 seq# 6 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
    Mon Feb 19 16:51:00 2007
    Thread 1 cannot allocate new log, sequence 7
    Checkpoint not complete
    Current log# 2 seq# 6 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
    Thread 1 advanced to log sequence 7
    Current log# 1 seq# 7 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
    Mon Feb 19 16:51:54 2007
    Thread 1 cannot allocate new log, sequence 8
    Checkpoint not complete
    Current log# 1 seq# 7 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
    Thread 1 advanced to log sequence 8
    Current log# 2 seq# 8 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
    Mon Feb 19 16:55:32 2007
    Thread 1 cannot allocate new log, sequence 9
    Checkpoint not complete
    Current log# 2 seq# 8 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
    Thread 1 advanced to log sequence 9
    Current log# 1 seq# 9 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
    Mon Feb 19 16:56:35 2007
    Thread 1 cannot allocate new log, sequence 10
    Checkpoint not complete
    Current log# 1 seq# 9 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
    Thread 1 advanced to log sequence 10
    Current log# 2 seq# 10 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
    Mon Feb 19 16:56:52 2007
    Thread 1 cannot allocate new log, sequence 11
    Checkpoint not complete
    Current log# 2 seq# 10 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
    Thread 1 advanced to log sequence 11
    Current log# 1 seq# 11 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
    Mon Feb 19 16:57:04 2007
    Thread 1 cannot allocate new log, sequence 12
    Checkpoint not complete
    Current log# 1 seq# 11 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
    Thread 1 advanced to log sequence 12
    Current log# 2 seq# 12 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
    Mon Feb 19 16:57:14 2007
    Thread 1 cannot allocate new log, sequence 13
    Checkpoint not complete
    Current log# 2 seq# 12 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
    Thread 1 advanced to log sequence 13
    Current log# 1 seq# 13 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
    Mon Feb 19 16:57:31 2007
    Thread 1 cannot allocate new log, sequence 14
    Checkpoint not complete
    Current log# 1 seq# 13 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
    Thread 1 advanced to log sequence 14
    Current log# 2 seq# 14 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
    Mon Feb 19 16:58:13 2007
    Thread 1 cannot allocate new log, sequence 15
    Checkpoint not complete
    Current log# 2 seq# 14 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
    Thread 1 advanced to log sequence 15
    Current log# 1 seq# 15 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
    Mon Feb 19 16:59:42 2007
    Thread 1 cannot allocate new log, sequence 16
    Checkpoint not complete
    Current log# 1 seq# 15 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
    Thread 1 advanced to log sequence 16
    Current log# 2 seq# 16 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
    Mon Feb 19 17:00:33 2007
    Thread 1 cannot allocate new log, sequence 17
    Checkpoint not complete
    Current log# 2 seq# 16 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
    Thread 1 advanced to log sequence 17
    Current log# 1 seq# 17 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
    Mon Feb 19 17:01:49 2007
    Thread 1 cannot allocate new log, sequence 18
    Checkpoint not complete
    Current log# 1 seq# 17 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
    Thread 1 advanced to log sequence 18
    Current log# 2 seq# 18 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
    Mon Feb 19 17:02:23 2007
    Thread 1 cannot allocate new log, sequence 19
    Checkpoint not complete
    Current log# 2 seq# 18 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
    Thread 1 advanced to log sequence 19
    Current log# 1 seq# 19 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
    Mon Feb 19 17:04:06 2007
    Thread 1 cannot allocate new log, sequence 20
    Checkpoint not complete
    Current log# 1 seq# 19 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
    Thread 1 advanced to log sequence 20
    Current log# 2 seq# 20 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
    Mon Feb 19 17:05:22 2007
    Thread 1 cannot allocate new log, sequence 21
    Checkpoint not complete
    Current log# 2 seq# 20 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
    Thread 1 advanced to log sequence 21
    Current log# 1 seq# 21 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
    Mon Feb 19 17:06:22 2007
    Thread 1 cannot allocate new log, sequence 22
    Checkpoint not complete
    Current log# 1 seq# 21 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
    Thread 1 advanced to log sequence 22
    Current log# 2 seq# 22 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
    Mon Feb 19 17:08:42 2007
    Thread 1 cannot allocate new log, sequence 23
    Checkpoint not complete
    Current log# 2 seq# 22 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
    Thread 1 advanced to log sequence 23
    Current log# 1 seq# 23 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
    Mon Feb 19 17:09:43 2007
    Thread 1 cannot allocate new log, sequence 24
    Checkpoint not complete
    Current log# 1 seq# 23 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
    Thread 1 advanced to log sequence 24
    Current log# 2 seq# 24 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
    Mon Feb 19 17:10:42 2007
    Thread 1 cannot allocate new log, sequence 25
    Checkpoint not complete
    Current log# 2 seq# 24 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
    Thread 1 advanced to log sequence 25
    Current log# 1 seq# 25 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
    Mon Feb 19 17:11:31 2007
    Thread 1 cannot allocate new log, sequence 26
    Checkpoint not complete
    Current log# 1 seq# 25 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
    Thread 1 advanced to log sequence 26
    Current log# 2 seq# 26 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
    Mon Feb 19 17:12:01 2007
    Thread 1 cannot allocate new log, sequence 27
    Checkpoint not complete
    Current log# 2 seq# 26 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
    Thread 1 advanced to log sequence 27
    Current log# 1 seq# 27 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
    Mon Feb 19 17:13:12 2007
    Thread 1 cannot allocate new log, sequence 28
    Checkpoint not complete
    Current log# 1 seq# 27 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
    Thread 1 advanced to log sequence 28
    Current log# 2 seq# 28 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
    Mon Feb 19 17:14:05 2007
    Thread 1 cannot allocate new log, sequence 29
    Checkpoint not complete
    Current log# 2 seq# 28 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
    Thread 1 advanced to log sequence 29
    Current log# 1 seq# 29 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
    Mon Feb 19 17:14:57 2007
    Thread 1 cannot allocate new log, sequence 30
    Checkpoint not complete
    Current log# 1 seq# 29 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
    Thread 1 advanced to log sequence 30
    Current log# 2 seq# 30 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
    Mon Feb 19 17:16:29 2007
    Thread 1 cannot allocate new log, sequence 31
    Checkpoint not complete
    Current log# 2 seq# 30 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
    Thread 1 advanced to log sequence 31
    Current log# 1 seq# 31 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
    Mon Feb 19 17:17:54 2007
    Starting background process EMN0
    EMN0 started with pid=25, OS id=15651
    Mon Feb 19 17:17:54 2007
    Shutting down instance: further logons disabled
    Mon Feb 19 17:17:55 2007
    Stopping background process QMNC
    Mon Feb 19 17:17:55 2007
    Stopping background process CJQ0
    Mon Feb 19 17:17:57 2007
    Stopping background process MMNL
    Mon Feb 19 17:17:58 2007
    Stopping background process MMON
    Mon Feb 19 17:17:59 2007
    Shutting down instance (immediate)
    License high water mark = 1
    Mon Feb 19 17:17:59 2007
    Stopping Job queue slave processes
    Mon Feb 19 17:17:59 2007
    Job queue slave processes stopped
    All dispatchers and shared servers shutdown
    Mon Feb 19 17:18:01 2007
    ALTER DATABASE CLOSE NORMAL
    Mon Feb 19 17:18:01 2007
    SMON: disabling tx recovery
    SMON: disabling cache recovery
    Mon Feb 19 17:18:03 2007
    Shutting down archive processes
    Archiving is disabled
    Archive process shutdown avoided: 0 active
    Thread 1 closed at log sequence 31
    Successful close of redo thread 1
    Mon Feb 19 17:18:04 2007
    Completed: ALTER DATABASE CLOSE NORMAL
    Mon Feb 19 17:18:04 2007
    ALTER DATABASE DISMOUNT
    Mon Feb 19 17:18:04 2007
    SUCCESS: diskgroup ORCL_DATA1 was dismounted
    Mon Feb 19 17:18:04 2007
    Completed: ALTER DATABASE DISMOUNT
    ARCH: Archival disabled due to shutdown: 1089
    Shutting down archive processes
    Archiving is disabled
    Archive process shutdown avoided: 0 active
    ARCH: Archival disabled due to shutdown: 1089
    Shutting down archive processes
    Archiving is disabled
    Archive process shutdown avoided: 0 active
    Mon Feb 19 17:18:09 2007
    freeing rdom 0
    Mon Feb 19 17:18:12 2007
    Starting ORACLE instance (normal)
    LICENSE_MAX_SESSION = 0
    LICENSE_SESSIONS_WARNING = 0
    Interface type 1 eth1 10.250.90.0 configured from OCR for use as a cluster interconnect
    Interface type 1 eth0 10.250.90.0 configured from OCR for use as a public interface
    Picked latch-free SCN scheme 2
    Autotune of undo retention is turned on.
    IMODE=BR
    ILAT =36
    LICENSE_MAX_USERS = 0
    SYS auditing is disabled
    ksdpec: called for event 13740 prior to event group initialization
    Starting up ORACLE RDBMS Version: 10.2.0.2.0.
    System parameters with non-default values:
    processes = 300
    sessions = 335
    sga_max_size = 1073741824
    sga_target = 1073741824
    control_files = +ORCL_DATA1/bsaprod/controlfile/current.256.614969167
    db_block_size = 8192
    compatible = 10.2.0.2.0
    log_archive_dest_1 = LOCATION=+BSA_DATA1/bsaprod/
    log_archive_format = %t_%s_%r.dbf
    db_file_multiblock_read_count= 16
    cluster_database_instances= 1
    db_create_file_dest = +ORCL_DATA1
    instance_number = 1
    undo_management = AUTO
    undo_tablespace = UNDOTBS1
    remote_login_passwordfile= EXCLUSIVE
    db_domain =
    dispatchers = (PROTOCOL=TCP) (SERVICE=bsaprodXDB)
    remote_listener = LISTENERS_BSAPROD
    job_queue_processes = 10
    background_dump_dest = /u01/app/oracle/oracle/admin/bsaprod/bdump
    user_dump_dest = /u01/app/oracle/oracle/admin/bsaprod/udump
    core_dump_dest = /u01/app/oracle/oracle/admin/bsaprod/cdump
    audit_file_dest = /u01/app/oracle/oracle/admin/bsaprod/adump
    db_name = bsaprod
    open_cursors = 300
    pga_aggregate_target = 262144000
    Cluster communication is configured to use the following interface(s) for this instance
    10.250.90.107
    Mon Feb 19 17:18:13 2007
    cluster interconnect IPC version:Oracle UDP/IP (generic)
    IPC Vendor 1 proto 2
    PMON started with pid=2, OS id=16124
    DIAG started with pid=3, OS id=16126
    PSP0 started with pid=4, OS id=16128
    LMON started with pid=5, OS id=16130
    LMD0 started with pid=6, OS id=16132
    MMAN started with pid=7, OS id=16134
    DBW0 started with pid=8, OS id=16136
    LGWR started with pid=9, OS id=16138
    CKPT started with pid=10, OS id=16140
    SMON started with pid=11, OS id=16142
    RECO started with pid=12, OS id=16144
    CJQ0 started with pid=13, OS id=16146
    MMON started with pid=14, OS id=16148
    Mon Feb 19 17:18:13 2007
    starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
    MMNL started with pid=15, OS id=16150
    Mon Feb 19 17:18:13 2007
    starting up 1 shared server(s) ...
    Mon Feb 19 17:18:13 2007
    lmon registered with NM - instance id 1 (internal mem no 0)
    Mon Feb 19 17:18:14 2007
    Reconfiguration started (old inc 0, new inc 2)
    List of nodes:
    0
    Global Resource Directory frozen
    * allocate domain 0, invalid = TRUE
    Communication channels reestablished
    Master broadcasted resource hash value bitmaps
    Non-local Process blocks cleaned out
    Resources and enqueues cleaned out
    Resources remastered 0
    Set master node info
    Submitted all remote-enqueue requests
    Dwn-cvts replayed, VALBLKs dubious
    All grantable enqueues granted
    Post SMON to start 1st pass IR
    Submitted all GCS remote-cache requests
    Post SMON to start 1st pass IR
    Reconfiguration complete
    Mon Feb 19 17:18:14 2007
    ALTER DATABASE MOUNT
    Mon Feb 19 17:18:14 2007
    Starting background process ASMB
    ASMB started with pid=19, OS id=16182
    Starting background process RBAL
    RBAL started with pid=20, OS id=16186
    Loaded ASM Library - Generic Linux, version 2.0.2 (KABI_V2) library for asmlib interface
    Mon Feb 19 17:18:18 2007
    SUCCESS: diskgroup ORCL_DATA1 was mounted
    Mon Feb 19 17:18:23 2007
    Setting recovery target incarnation to 1
    Mon Feb 19 17:18:23 2007
    Successful mount of redo thread 1, with mount id 3309012054
    Mon Feb 19 17:18:23 2007
    Database mounted in Exclusive Mode
    Completed: ALTER DATABASE MOUNT
    Mon Feb 19 17:18:23 2007
    alter database archivelog
    Completed: alter database archivelog
    Mon Feb 19 17:18:23 2007
    alter database open
    Mon Feb 19 17:18:23 2007
    LGWR: STARTING ARCH PROCESSES
    ARC0 started with pid=22, OS id=16393
    Mon Feb 19 17:18:23 2007
    ARC0: Archival started
    ARC1: Archival started
    LGWR: STARTING ARCH PROCESSES COMPLETE
    ARC1 started with pid=23, OS id=16395
    Mon Feb 19 17:18:23 2007
    Thread 1 opened at log sequence 31
    Current log# 1 seq# 31 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
    Successful open of redo thread 1
    Mon Feb 19 17:18:23 2007
    MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
    Mon Feb 19 17:18:23 2007
    SMON: enabling cache recovery
    Mon Feb 19 17:18:23 2007
    ARC0: STARTING ARCH PROCESSES
    Mon Feb 19 17:18:23 2007
    ARC1: Becoming the 'no FAL' ARCH
    ARC1: Becoming the 'no SRL' ARCH
    Mon Feb 19 17:18:23 2007
    ARC2: Archival started
    ARC0: STARTING ARCH PROCESSES COMPLETE
    ARC0: Becoming the heartbeat ARCH
    ARC2 started with pid=24, OS id=16422
    Mon Feb 19 17:18:23 2007
    Successfully onlined Undo Tablespace 1.
    Mon Feb 19 17:18:23 2007
    SMON: enabling tx recovery
    Mon Feb 19 17:18:23 2007
    Database Characterset is WE8ISO8859P1
    replication_dependency_tracking turned off (no async multimaster replication found)
    Starting background process QMNC
    QMNC started with pid=25, OS id=16449
    Mon Feb 19 17:18:25 2007
    Completed: alter database open
    Mon Feb 19 17:18:25 2007
    ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 3 SIZE 51200K,
    GROUP 4 SIZE 51200K
    Completed: ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 3 SIZE 51200K,
    GROUP 4 SIZE 51200K
    Mon Feb 19 17:18:28 2007
    ALTER DATABASE ENABLE PUBLIC THREAD 2
    Completed: ALTER DATABASE ENABLE PUBLIC THREAD 2
    SUCCESS: diskgroup BSA_DATA1 was mounted
    SUCCESS: diskgroup BSA_DATA1 was dismounted
    Mon Feb 19 17:18:29 2007
    Starting background process EMN0
    EMN0 started with pid=26, OS id=16564
    Mon Feb 19 17:18:29 2007
    Shutting down instance: further logons disabled
    Mon Feb 19 17:18:29 2007
    Stopping background process QMNC
    Mon Feb 19 17:18:29 2007
    Stopping background process CJQ0
    Mon Feb 19 17:18:30 2007
    Stopping background process MMNL
    Mon Feb 19 17:18:31 2007
    Stopping background process MMON
    Mon Feb 19 17:18:32 2007
    Shutting down instance (immediate)
    License high water mark = 3
    Mon Feb 19 17:18:32 2007
    Stopping Job queue slave processes
    Mon Feb 19 17:18:32 2007
    Job queue slave processes stopped
    All dispatchers and shared servers shutdown
    Mon Feb 19 17:18:34 2007
    ALTER DATABASE CLOSE NORMAL
    Mon Feb 19 17:18:34 2007
    SMON: disabling tx recovery
    SMON: disabling cache recovery
    Mon Feb 19 17:18:34 2007
    Shutting down archive processes
    Archiving is disabled
    Mon Feb 19 17:18:39 2007
    ARCH shutting down
    ARC2: Archival stopped
    Mon Feb 19 17:18:44 2007
    ARCH shutting down
    ARC1: Archival stopped
    Mon Feb 19 17:18:49 2007
    ARCH shutting down
    ARC0: Archival stopped
    Mon Feb 19 17:18:50 2007
    Thread 1 closed at log sequence 31
    Successful close of redo thread 1
    Mon Feb 19 17:18:51 2007
    Completed: ALTER DATABASE CLOSE NORMAL
    Mon Feb 19 17:18:51 2007
    ALTER DATABASE DISMOUNT
    Mon Feb 19 17:18:51 2007
    SUCCESS: diskgroup ORCL_DATA1 was dismounted
    Mon Feb 19 17:18:51 2007
    Completed: ALTER DATABASE DISMOUNT
    ARCH: Archival disabled due to shutdown: 1089
    Shutting down archive processes
    Archiving is disabled
    Archive process shutdown avoided: 0 active
    ARCH: Archival disabled due to shutdown: 1089
    Shutting down archive processes
    Archiving is disabled
    Archive process shutdown avoided: 0 active
    Mon Feb 19 17:18:56 2007
    freeing rdom 0
    Mon Feb 19 17:18:59 2007
    Starting ORACLE instance (normal)
    LICENSE_MAX_SESSION = 0
    LICENSE_SESSIONS_WARNING = 0
    Interface type 1 eth1 10.250.90.0 configured from OCR for use as a cluster interconnect
    Interface type 1 eth0 10.250.90.0 configured from OCR for use as a public interface
    Picked latch-free SCN scheme 2
    Autotune of undo retention is turned on.
    LICENSE_MAX_USERS = 0
    SYS auditing is disabled
    ksdpec: called for event 13740 prior to event group initialization
    Starting up ORACLE RDBMS Version: 10.2.0.2.0.
    System parameters with non-default values:
    processes = 300
    sessions = 335
    sga_max_size = 1073741824
    spfile = +BSA_DATA1/bsaprod/spfilebsaprod.ora
    sga_target = 1073741824
    control_files = +ORCL_DATA1/bsaprod/controlfile/current.256.614969167
    db_block_size = 8192
    compatible = 10.2.0.2.0
    log_archive_dest_1 = LOCATION=+BSA_DATA1/bsaprod/
    log_archive_format = %t_%s_%r.dbf
    db_file_multiblock_read_count= 16
    cluster_database = TRUE
    cluster_database_instances= 2
    db_create_file_dest = +ORCL_DATA1
    thread = 1
    instance_number = 1
    undo_management = AUTO
    undo_tablespace = UNDOTBS1
    remote_login_passwordfile= EXCLUSIVE
    db_domain =
    dispatchers = (PROTOCOL=TCP) (SERVICE=bsaprodXDB)
    remote_listener = LISTENERS_BSAPROD
    job_queue_processes = 10
    background_dump_dest = /u01/app/oracle/oracle/admin/bsaprod/bdump
    user_dump_dest = /u01/app/oracle/oracle/admin/bsaprod/udump
    core_dump_dest = /u01/app/oracle/oracle/admin/bsaprod/cdump
    audit_file_dest = /u01/app/oracle/oracle/admin/bsaprod/adump
    db_name = bsaprod
    open_cursors = 300
    pga_aggregate_target = 262144000
    Cluster communication is configured to use the following interface(s) for this instance
    10.250.90.107
    Mon Feb 19 17:19:00 2007
    cluster interconnect IPC version:Oracle UDP/IP (generic)
    IPC Vendor 1 proto 2
    PMON started with pid=2, OS id=17373
    DIAG started with pid=3, OS id=17375
    PSP0 started with pid=4, OS id=17377
    LMON started with pid=5, OS id=17379
    LMD0 started with pid=6, OS id=17381
    LMS0 started with pid=7, OS id=17383
    LMS1 started with pid=8, OS id=17387
    MMAN started with pid=9, OS id=17391
    DBW0 started with pid=10, OS id=17393
    LGWR started with pid=11, OS id=17395
    CKPT started with pid=12, OS id=17397
    SMON started with pid=13, OS id=17399
    RECO started with pid=14, OS id=17412
    CJQ0 started with pid=15, OS id=17428
    MMON started with pid=16, OS id=17430
    Mon Feb 19 17:19:00 2007
    starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
    MMNL started with pid=17, OS id=17432
    Mon Feb 19 17:19:00 2007
    starting up 1 shared server(s) ...
    Mon Feb 19 17:19:01 2007
    lmon registered with NM - instance id 1 (internal mem no 0)
    Mon Feb 19 17:19:01 2007
    Reconfiguration started (old inc 0, new inc 2)
    List of nodes:
    0
    Global Resource Directory frozen
    * allocate domain 0, invalid = TRUE
    Communication channels reestablished
    Master broadcasted resource hash value bitmaps
    Non-local Process blocks cleaned out
    Mon Feb 19 17:19:01 2007
    LMS 1: 0 GCS shadows cancelled, 0 closed
    Mon Feb 19 17:19:01 2007
    LMS 0: 0 GCS shadows cancelled, 0 closed
    Set master node info
    Submitted all remote-enqueue requests
    Dwn-cvts replayed, VALBLKs dubious
    All grantable enqueues granted
    Post SMON to start 1st pass IR
    Mon Feb 19 17:19:01 2007
    LMS 1: 0 GCS shadows traversed, 0 replayed
    Mon Feb 19 17:19:01 2007
    LMS 0: 0 GCS shadows traversed, 0 replayed
    Mon Feb 19 17:19:01 2007
    Submitted all GCS remote-cache requests
    Post SMON to start 1st pass IR
    Fix write in gcs resources
    Reconfiguration complete
    LCK0 started with pid=20, OS id=17460
    Mon Feb 19 17:19:02 2007
    ALTER DATABASE MOUNT
    Mon Feb 19 17:19:02 2007
    This instance was first to mount
    Mon Feb 19 17:19:02 2007
    Starting background process ASMB
    ASMB started with pid=22, OS id=17466
    Starting background process RBAL
    RBAL started with pid=23, OS id=17470
    Loaded ASM Library - Generic Linux, version 2.0.2 (KABI_V2) library for asmlib interface
    Mon Feb 19 17:19:06 2007
    SUCCESS: diskgroup ORCL_DATA1 was mounted
    Mon Feb 19 17:19:09 2007
    SUCCESS: diskgroup BSA_DATA1 was mounted
    SUCCESS: diskgroup BSA_DATA1 was dismounted
    SUCCESS: diskgroup BSA_DATA1 was mounted
    SUCCESS: diskgroup BSA_DATA1 was dismounted
    Mon Feb 19 17:19:10 2007
    Setting recovery target incarnation to 1
    Mon Feb 19 17:19:10 2007
    Successful mount of redo thread 1, with mount id 3308997510
    Mon Feb 19 17:19:10 2007
    Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)
    Completed: ALTER DATABASE MOUNT
    Mon Feb 19 17:19:11 2007
    ALTER DATABASE OPEN
    This instance was first to open
    Picked broadcast on commit scheme to generate SCNs
    Mon Feb 19 17:19:11 2007
    LGWR: STARTING ARCH PROCESSES
    ARC0 started with pid=26, OS id=17679
    Mon Feb 19 17:19:11 2007
    ARC0: Archival started
    ARC1: Archival started
    LGWR: STARTING ARCH PROCESSES COMPLETE
    ARC1 started with pid=27, OS id=17706
    Mon Feb 19 17:19:11 2007
    Thread 1 opened at log sequence 31
    Current log# 1 seq# 31 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
    Successful open of redo thread 1
    Mon Feb 19 17:19:11 2007
    MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
    Mon Feb 19 17:19:11 2007
    ARC0: STARTING ARCH PROCESSES
    Mon Feb 19 17:19:11 2007
    ARC1: Becoming the 'no FAL' ARCH
    ARC1: Becoming the 'no SRL' ARCH
    Mon Feb 19 17:19:11 2007
    SMON: enabling cache recovery
    Mon Feb 19 17:19:11 2007
    ARC2: Archival started
    ARC0: STARTING ARCH PROCESSES COMPLETE
    ARC0: Becoming the heartbeat ARCH
    ARC2 started with pid=28, OS id=17708
    Mon Feb 19 17:19:11 2007
    Successfully onlined Undo Tablespace 1.
    Mon Feb 19 17:19:11 2007
    SMON: enabling tx recovery
    Mon Feb 19 17:19:11 2007
    Database Characterset is WE8ISO8859P1
    Mon Feb 19 17:19:11 2007
    Instance recovery: looking for dead threads
    Instance recovery: lock domain invalid but no dead threads
    Mon Feb 19 17:19:12 2007
    SUCCESS: diskgroup BSA_DATA1 was mounted
    SUCCESS: diskgroup BSA_DATA1 was dismounted
    SUCCESS: diskgroup BSA_DATA1 was mounted
    SUCCESS: diskgroup BSA_DATA1 was dismounted
    SUCCESS: diskgroup BSA_DATA1 was mounted
    SUCCESS: diskgroup BSA_DATA1 was dismounted
    SUCCESS: diskgroup BSA_DATA1 was mounted
    SUCCESS: diskgroup BSA_DATA1 was dismounted
    SUCCESS: diskgroup BSA_DATA1 was mounted
    SUCCESS: diskgroup BSA_DATA1 was dismounted
    Mon Feb 19 17:19:12 2007
    replication_dependency_tracking turned off (no async multimaster replication found)
    Starting background process QMNC
    QMNC started with pid=29, OS id=17735
    Mon Feb 19 17:19:13 2007
    Completed: ALTER DATABASE OPEN
    Mon Feb 19 17:20:11 2007
    Shutting down archive processes
    Mon Feb 19 17:20:16 2007
    ARCH shutting down
    ARC2: Archival stopped
    Mon Feb 19 17:23:35 2007
    Starting background process EMN0
    EMN0 started with pid=74, OS id=28350
    Mon Feb 19 17:23:35 2007
    Shutting down instance: further logons disabled
    Mon Feb 19 17:23:36 2007
    Stopping background process QMNC
    Mon Feb 19 17:23:36 2007
    Stopping background process CJQ0
    Mon Feb 19 17:23:38 2007
    Stopping background process MMNL
    Mon Feb 19 17:23:39 2007
    Stopping background process MMON
    Mon Feb 19 17:23:40 2007
    Shutting down instance (immediate)
    License high water mark = 34
    Mon Feb 19 17:23:40 2007
    Stopping Job queue slave processes
    Mon Feb 19 17:23:40 2007
    Job queue slave processes stopped
    All dispatchers and shared servers shutdown
    Mon Feb 19 17:24:07 2007
    ALTER DATABASE CLOSE NORMAL
    Mon Feb 19 17:24:07 2007
    SMON: disabling tx recovery
    SMON: disabling cache recovery
    Mon Feb 19 17:24:08 2007
    Shutting down archive processes
    Archiving is disabled
    Mon Feb 19 17:24:18 2007
    ARC1: Becoming the 'no FAL' ARCH
    ARC1: Becoming the 'no SRL' ARCH
    ARC1: Archiving disabled
    ARCH shutting down
    Mon Feb 19 17:24:18 2007
    ARC1: Archival stopped
    Mon Feb 19 17:24:18 2007
    ARC0: Becoming the heartbeat ARCH
    Mon Feb 19 17:24:18 2007
    ARC0: Archiving disabled
    ARCH shutting down
    ARC0: Archival stopped
    Mon Feb 19 17:24:19 2007
    Thread 1 closed at log sequence 31
    Successful close of redo thread 1
    Mon Feb 19 17:24:19 2007
    Completed: ALTER DATABASE CLOSE NORMAL
    Mon Feb 19 17:24:19 2007
    ALTER DATABASE DISMOUNT
    Mon Feb 19 17:24:19 2007
    SUCCESS: diskgroup ORCL_DATA1 was dismounted
    Mon Feb 19 17:24:19 2007
    Completed: ALTER DATABASE DISMOUNT
    ARCH: Archival disabled due to shutdown: 1089
    Shutting down archive processes
    Archiving is disabled
    Archive process shutdown avoided: 0 active
    ARCH: Archival disabled due to shutdown: 1089
    Shutting down archive processes
    Archiving is disabled
    Archive process shutdown avoided: 0 active
    Mon Feb 19 17:24:25 2007
    freeing rdom 0
    Mon Feb 19 17:24:35 2007
    Starting ORACLE instance (normal)
    LICENSE_MAX_SESSION = 0
    LICENSE_SESSIONS_WARNING = 0
    Interface type 1 eth1 10.250.90.0 configured from OCR for use as a cluster interconnect
    Interface type 1 eth0 10.250.90.0 configured from OCR for use as a public interface
    Picked latch-free SCN scheme 2
    Autotune of undo retention is turned on.
    LICENSE_MAX_USERS = 0
    SYS auditing is disabled
    ksdpec: called for event 13740 prior to event group initialization
    Starting up ORACLE RDBMS Version: 10.2.0.2.0.
    System parameters with non-default values:
    processes = 300
    sessions = 335
    sga_max_size = 1073741824
    __shared_pool_size = 255852544
    __large_pool_size = 4194304
    __java_pool_size = 4194304
    __streams_pool_size = 0
    spfile = +BSA_DATA1/bsaprod/spfilebsaprod.ora
    sga_target = 1073741824
    control_files = +ORCL_DATA1/bsaprod/controlfile/current.256.614969167
    db_block_size = 8192
    __db_cache_size = 801112064
    compatible = 10.2.0.2.0
    log_archive_dest_1 = LOCATION=+BSA_DATA1/bsaprod/
    log_archive_format = %t_%s_%r.dbf
    db_file_multiblock_read_count= 16
    cluster_database = TRUE
    cluster_database_instances= 2
    db_create_file_dest = +ORCL_DATA1
    thread = 1
    instance_number = 1
    undo_management = AUTO
    undo_tablespace = UNDOTBS1
    remote_login_passwordfile= EXCLUSIVE
    db_domain =
    dispatchers = (PROTOCOL=TCP) (SERVICE=bsaprodXDB)
    remote_listener = LISTENERS_BSAPROD
    job_queue_processes = 10
    background_dump_dest = /u01/app/oracle/oracle/admin/bsaprod/bdump
    user_dump_dest = /u01/app/oracle/oracle/admin/bsaprod/udump
    core_dump_dest = /u01/app/oracle/oracle/admin/bsaprod/cdump
    audit_file_dest = /u01/app/oracle/oracle/admin/bsaprod/adump
    db_name = bsaprod
    open_cursors = 300
    pga_aggregate_target = 262144000
    Cluster communication is configured to use the following interface(s) for this instance
    10.250.90.107
    Mon Feb 19 17:24:35 2007
    cluster interconnect IPC version:Oracle UDP/IP (generic)
    IPC Vendor 1 proto 2
    PMON started with pid=2, OS id=29950
    DIAG started with pid=3, OS id=29952
    PSP0 started with pid=4, OS id=29954
    LMON started with pid=5, OS id=29956
    LMD0 started with pid=6, OS id=29958
    LMS0 started with pid=7, OS id=29960
    LMS1 started with pid=8, OS id=29964
    MMAN started with pid=9, OS id=29968
    DBW0 started with pid=10, OS id=29970
    LGWR started with pid=11, OS id=29972
    CKPT started with pid=12, OS id=29974
    SMON started with pid=13, OS id=29976
    RECO started with pid=14, OS id=29978
    CJQ0 started with pid=15, OS id=29980
    MMON started with pid=16, OS id=29982
    Mon Feb 19 17:24:36 2007
    starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
    MMNL started with pid=17, OS id=29984
    Mon Feb 19 17:24:36 2007
    starting up 1 shared server(s) ...
    Mon Feb 19 17:24:36 2007
    lmon registered with NM - instance id 1 (internal mem no 0)
    Mon Feb 19 17:24:39 2007
    Reconfiguration started (old inc 0, new inc 4)
    List of nodes:
    0 1
    Global Resource Directory frozen
    * allocate domain 0, invalid = TRUE
    Communication channels reestablished
    * domain 0 not valid according to instance 1
    * domain 0 valid = 0 according to instance 1
    Mon Feb 19 17:24:39 2007
    Master broadcasted resource hash value bitmaps
    Non-local Process blocks cleaned out
    Mon Feb 19 17:24:39 2007
    LMS 1: 0 GCS shadows cancelled, 0 closed
    Mon Feb 19 17:24:39 2007
    LMS 0: 0 GCS shadows cancelled, 0 closed
    Set master node info
    Submitted all remote-enqueue requests
    Dwn-cvts replayed, VALBLKs dubious
    All grantable enqueues granted
    Mon Feb 19 17:24:39 2007
    LMS 0: 0 GCS shadows traversed, 0 replayed
    Mon Feb 19 17:24:39 2007
    LMS 1: 0 GCS shadows traversed, 0 replayed
    Mon Feb 19 17:24:39 2007
    Submitted all GCS remote-cache requests
    Post SMON to start 1st pass IR
    Fix write in gcs resources
    Reconfiguration complete
    LCK0 started with pid=20, OS id=30069
    Mon Feb 19 17:24:40 2007
    ALTER DATABASE MOUNT
    Mon Feb 19 17:24:41 2007
    Starting background process ASMB
    ASMB started with pid=22, OS id=30121
    Starting background process RBAL
    RBAL started with pid=23, OS id=30125
    Loaded ASM Library - Generic Linux, version 2.0.2 (KABI_V2) library for asmlib interface
    Mon Feb 19 17:24:45 2007
    SUCCESS: diskgroup ORCL_DATA1 was mounted
    Mon Feb 19 17:24:49 2007
    Setting recovery target incarnation to 1
    Mon Feb 19 17:24:49 2007
    Successful mount of redo thread 1, with mount id 3309017062
    Mon Feb 19 17:24:49 2007
    Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)
    Mon Feb 19 17:24:51 2007
    SUCCESS: diskgroup BSA_DATA1 was mounted
    SUCCESS: diskgroup BSA_DATA1 was dismounted
    SUCCESS: diskgroup BSA_DATA1 was mounted
    SUCCESS: diskgroup BSA_DATA1 was dismounted
    SUCCESS: diskgroup BSA_DATA1 was mounted
    SUCCESS: diskgroup BSA_DATA1 was dismounted
    SUCCESS: diskgroup BSA_DATA1 was mounted
    SUCCESS: diskgroup BSA_DATA1 was dismounted
    SUCCESS: diskgroup BSA_DATA1 was mounted
    SUCCESS: diskgroup BSA_DATA1 was dismounted
    Mon Feb 19 17:24:52 2007
    Completed: ALTER DATABASE MOUNT
    Mon Feb 19 17:24:52 2007
    ALTER DATABASE OPEN
    Picked broadcast on commit scheme to generate SCNs
    Mon Feb 19 17:24:52 2007
    LGWR: STARTING ARCH PROCESSES
    ARC0 started with pid=25, OS id=30406
    Mon Feb 19 17:24:53 2007
    ARC0: Archival started
    ARC1: Archival started
    LGWR: STARTING ARCH PROCESSES COMPLETE
    ARC1 started with pid=26, OS id=30422
    Mon Feb 19 17:24:53 2007
    Thread 1 opened at log sequence 31
    Current log# 1 seq# 31 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
    Successful open of redo thread 1
    Mon Feb 19 17:24:53 2007
    MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
    Mon Feb 19 17:24:53 2007
    ARC0: STARTING ARCH PROCESSES
    Mon Feb 19 17:24:53 2007
    ARC1: Becoming the 'no FAL' ARCH
    ARC1: Becoming the 'no SRL' ARCH
    Mon Feb 19 17:24:53 2007
    SMON: enabling cache recovery
    Mon Feb 19 17:24:53 2007
    ARC2: Archival started
    ARC0: STARTING ARCH PROCESSES COMPLETE
    ARC0: Becoming the heartbeat ARCH
    ARC2 started with pid=27, OS id=30436
    Mon Feb 19 17:25:07 2007
    Successfully onlined Undo Tablespace 1.
    Mon Feb 19 17:25:07 2007
    SMON: enabling tx recovery
    Mon Feb 19 17:25:07 2007
    Database Characterset is WE8ISO8859P1
    replication_dependency_tracking turned off (no async multimaster replication found)
    Starting background process QMNC
    QMNC started with pid=29, OS id=30882
    Mon Feb 19 17:26:31 2007
    Completed: ALTER DATABASE OPEN
    Mon Feb 19 17:26:38 2007
    ALTER SYSTEM SET service_names='bsaprod','bsatest' SCOPE=MEMORY SID='bsaprod1';
    Mon Feb 19 17:26:53 2007
    Shutting down archive processes
    Mon Feb 19 17:26:58 2007
    ARCH shutting down
    ARC2: Archival stopped
    Mon Feb 19 17:32:16 2007
    Error: KGXGN aborts the instance (6)
    Mon Feb 19 17:32:16 2007
    Errors in file /u01/app/oracle/oracle/admin/bsaprod/bdump/bsaprod1_lmon_29956.trc:
    ORA-29702: error occurred in Cluster Group Service operation
    LMON: terminating instance due to error 29702
    Mon Feb 19 17:32:17 2007
    System state dump is made for local instance
    System State dumped to trace file /u01/app/oracle/oracle/admin/bsaprod/bdump/bsaprod1_diag_29952.trc
    Mon Feb 19 17:32:18 2007
    Trace dumping is performing id=[cdmp_20070219173217]
    Mon Feb 19 17:32:21 2007
    Instance terminated by LMON, pid = 29956
    and also the output of crs_stat -t in first node is....
    Name Type Target State Host
    ora....d1.inst application ONLINE OFFLINE
    ora....d2.inst application ONLINE OFFLINE
    ora....od1.srv application ONLINE OFFLINE
    ora....od2.srv application ONLINE OFFLINE
    ora....test.cs application ONLINE OFFLINE
    ora.bsaprod.db application ONLINE OFFLINE
    ora....SM1.asm application ONLINE UNKNOWN dsrvbd003
    ora....ER.lsnr application ONLINE UNKNOWN dsrvbd003
    ora....03.lsnr application ONLINE UNKNOWN dsrvbd003
    ora....003.gsd application ONLINE UNKNOWN dsrvbd003
    ora....003.ons application ONLINE UNKNOWN dsrvbd003
    ora....003.vip application ONLINE ONLINE dsrvbd003
    ora....SM2.asm application ONLINE OFFLINE
    ora....04.lsnr application ONLINE OFFLINE
    ora....004.gsd application ONLINE OFFLINE
    ora....004.ons application ONLINE OFFLINE
    ora....004.vip application ONLINE OFFLINE

  • Post server reboot , instance 1 crashes on starting the second one

    Hi ,
    We had a server reboot ( Solaris SPARC 64 bit) and had to shutdown the application running on RAC system on 2 servers.
    now post the activity , we are trying to bring up the instances but whenever one comes up the other goes down.
    If we try to start the second one , first goes down.
    Oracle database version : 10.2.0.3.0
    Alert logs for both have been mentioned below :
    instance1:
    Mon Jul 23 16:17:22 2012
    Starting ORACLE instance (normal)
    LICENSE_MAX_SESSION = 0
    LICENSE_SESSIONS_WARNING = 0
    Interface type 1 bge3 10.0.0.0 configured from OCR for use as a cluster interconnect
    Interface type 1 bge0 58.2.35.0 configured from OCR for use as  a public interface
    Interface type 1 bge1 58.2.35.0 configured from OCR for use as  a public interface
    Picked latch-free SCN scheme 3
    Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/orainf/product/10.2.0/dbs/arch
    Autotune of undo retention is turned on.
    LICENSE_MAX_USERS = 0
    SYS auditing is disabled
    ksdpec: called for event 13740 prior to event group initialization
    Starting up ORACLE RDBMS Version: 10.2.0.3.0.
    System parameters with non-default values:
      processes                = 400
      __shared_pool_size       = 956301312
      shared_pool_size         = 201326592
      __large_pool_size        = 16777216
      __java_pool_size         = 150994944
      java_pool_size           = 150994944
      __streams_pool_size      = 16777216
      streams_pool_size        = 16777216
      sga_target               = 1610612736
      control_files            = /db01/asdb/asdb/control01.ctl, /db01/asdb/asdb/control02.ctl, /db01/asdb/asdb/control03.ctl
      db_block_size            = 8192
      __db_cache_size          = 452984832
      db_cache_size            = 167772160
      compatible               = 10.2.0.3.0
      db_file_multiblock_read_count= 16
      cluster_database         = TRUE
      cluster_database_instances= 2
      thread                   = 1
      instance_number          = 1
      undo_management          = AUTO
      undo_tablespace          = UNDOTBS1
      ldap_directory_access    = PASSWORD
      remote_login_passwordfile= EXCLUSIVE
      db_domain                = intranet.genpact.com
      remote_listener          = LISTENERS_ASDB
      job_queue_processes      = 10
      background_dump_dest     = /u01/app/orainf/admin/asdb/bdump
      user_dump_dest           = /u01/app/orainf/admin/asdb/udump
      core_dump_dest           = /u01/app/orainf/admin/asdb/cdump
      audit_file_dest          = /u01/app/orainf/admin/asdb/adump
      db_name                  = asdb
      open_cursors             = 300
      pga_aggregate_target     = 848297984
      aq_tm_processes          = 1
    Mon Jul 23 16:17:24 2012
    Oracle instance running with ODM: Veritas 5.0.30.00 ODM Library, Version 1.1
    cluster interconnect IPC version:
            VERITAS IPC '5.0MP3' 07:30:15 Jul 29 2008
    IPC Vendor 86 proto 76
      Version 1.0
    PMON started with pid=2, OS id=17279
    DIAG started with pid=3, OS id=17281
    Mon Jul 23 16:17:24 2012
    Errors in file /u01/app/orainf/admin/asdb/bdump/asdb1_diag_17281.trc:
    ORA-07445: exception encountered: core dump [_kill()+8] [SIGIOT] [unknown code] [0x438100000000] [] []
    PSP0 started with pid=4, OS id=17283
    LMON started with pid=5, OS id=17285
    LMD0 started with pid=6, OS id=17287
    LMS0 started with pid=7, OS id=17289
    LMS1 started with pid=8, OS id=17293
    MMAN started with pid=9, OS id=17297
    DBW0 started with pid=10, OS id=17299
    LGWR started with pid=11, OS id=17301
    CKPT started with pid=12, OS id=17303
    SMON started with pid=13, OS id=17333
    RECO started with pid=14, OS id=17338
    CJQ0 started with pid=15, OS id=17340
    MMON started with pid=16, OS id=17342
    MMNL started with pid=17, OS id=17344
    Mon Jul 23 16:17:26 2012
    Errors in file /u01/app/orainf/admin/asdb/bdump/asdb1_lmon_17285.trc:
    ORA-27550: Target ID protocol check failed. tid vers=%d, type=%d, remote instance number=%d, local instance number=%d
    Mon Jul 23 16:17:26 2012
    LMON: terminating instance due to error 27550
    Instance terminated by LMON, pid = 17285
    Mon Jul 23 16:18:33 2012
    Starting ORACLE instance (normal)
    LICENSE_MAX_SESSION = 0
    LICENSE_SESSIONS_WARNING = 0
    Interface type 1 bge3 10.0.0.0 configured from OCR for use as a cluster interconnect
    Interface type 1 bge0 58.2.35.0 configured from OCR for use as  a public interface
    Interface type 1 bge1 58.2.35.0 configured from OCR for use as  a public interface
    Picked latch-free SCN scheme 3
    Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/orainf/product/10.2.0/dbs/arch
    Autotune of undo retention is turned on.
    LICENSE_MAX_USERS = 0
    SYS auditing is disabled
    ksdpec: called for event 13740 prior to event group initialization
    Starting up ORACLE RDBMS Version: 10.2.0.3.0.
    System parameters with non-default values:
      processes                = 400
      __shared_pool_size       = 956301312
      shared_pool_size         = 201326592
      __large_pool_size        = 16777216
      __java_pool_size         = 150994944
      java_pool_size           = 150994944
      __streams_pool_size      = 16777216
      streams_pool_size        = 16777216
      sga_target               = 1610612736
      control_files            = /db01/asdb/asdb/control01.ctl, /db01/asdb/asdb/control02.ctl, /db01/asdb/asdb/control03.ctl
      db_block_size            = 8192
      __db_cache_size          = 452984832
      db_cache_size            = 167772160
      compatible               = 10.2.0.3.0
      db_file_multiblock_read_count= 16
      cluster_database         = TRUE
      cluster_database_instances= 2
      thread                   = 1
      instance_number          = 1
      undo_management          = AUTO
      undo_tablespace          = UNDOTBS1
      remote_login_passwordfile= EXCLUSIVE
      db_domain                = intranet.genpact.com
      job_queue_processes      = 10
      background_dump_dest     = /u01/app/orainf/admin/asdb/bdump
      user_dump_dest           = /u01/app/orainf/admin/asdb/udump
      core_dump_dest           = /u01/app/orainf/admin/asdb/cdump
      audit_file_dest          = /u01/app/orainf/admin/asdb/adump
      db_name                  = asdb
      open_cursors             = 300
      pga_aggregate_target     = 848297984
      aq_tm_processes          = 1
    Mon Jul 23 16:18:35 2012
    Oracle instance running with ODM: Veritas 5.0.30.00 ODM Library, Version 1.1
    cluster interconnect IPC version:
            VERITAS IPC '5.0MP3' 07:30:15 Jul 29 2008
    IPC Vendor 86 proto 76
      Version 1.0
    PMON started with pid=2, OS id=19523
    DIAG started with pid=3, OS id=19525
    Mon Jul 23 16:18:36 2012
    Errors in file /u01/app/orainf/admin/asdb/bdump/asdb1_diag_19525.trc:
    ORA-07445: exception encountered: core dump [_kill()+8] [SIGIOT] [unknown code] [0x4C4500000000] [] []
    PSP0 started with pid=4, OS id=19527
    LMON started with pid=5, OS id=19529
    LMD0 started with pid=6, OS id=19531
    LMS0 started with pid=7, OS id=19533
    LMS1 started with pid=8, OS id=19537
    MMAN started with pid=9, OS id=19557
    DBW0 started with pid=10, OS id=19573
    LGWR started with pid=11, OS id=19576
    CKPT started with pid=12, OS id=19578
    SMON started with pid=13, OS id=19580
    RECO started with pid=14, OS id=19582
    CJQ0 started with pid=15, OS id=19584
    MMON started with pid=16, OS id=19586
    MMNL started with pid=17, OS id=19588
    Mon Jul 23 16:18:37 2012
    Errors in file /u01/app/orainf/admin/asdb/bdump/asdb1_lmon_19529.trc:
    ORA-27550: Target ID protocol check failed. tid vers=%d, type=%d, remote instance number=%d, local instance number=%d
    Mon Jul 23 16:18:37 2012
    LMON: terminating instance due to error 27550
    Instance terminated by LMON, pid = 19529
    Mon Jul 23 16:20:16 2012
    Starting ORACLE instance (normal)
    LICENSE_MAX_SESSION = 0
    LICENSE_SESSIONS_WARNING = 0
    Interface type 1 bge3 10.0.0.0 configured from OCR for use as a cluster interconnect
    Interface type 1 bge0 58.2.35.0 configured from OCR for use as  a public interface
    Interface type 1 bge1 58.2.35.0 configured from OCR for use as  a public interface
    Picked latch-free SCN scheme 3
    Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/orainf/product/10.2.0/dbs/arch
    Autotune of undo retention is turned on.
    LICENSE_MAX_USERS = 0
    SYS auditing is disabled
    ksdpec: called for event 13740 prior to event group initialization
    Starting up ORACLE RDBMS Version: 10.2.0.3.0.
    System parameters with non-default values:
      processes                = 400
      __shared_pool_size       = 956301312
      shared_pool_size         = 201326592
      __large_pool_size        = 16777216
      __java_pool_size         = 150994944
      java_pool_size           = 150994944
      __streams_pool_size      = 16777216
      streams_pool_size        = 16777216
      sga_target               = 1610612736
      control_files            = /db01/asdb/asdb/control01.ctl, /db01/asdb/asdb/control02.ctl, /db01/asdb/asdb/control03.ctl
      db_block_size            = 8192
      __db_cache_size          = 452984832
      db_cache_size            = 167772160
      compatible               = 10.2.0.3.0
      db_file_multiblock_read_count= 16
      cluster_database         = TRUE
      cluster_database_instances= 2
      thread                   = 1
      instance_number          = 1
      undo_management          = AUTO
      undo_tablespace          = UNDOTBS1
      db_domain                = intranet.genpact.com
      job_queue_processes      = 10
      background_dump_dest     = /u01/app/orainf/admin/asdb/bdump
      user_dump_dest           = /u01/app/orainf/admin/asdb/udump
      core_dump_dest           = /u01/app/orainf/admin/asdb/cdump
      audit_file_dest          = /u01/app/orainf/admin/asdb/adump
      db_name                  = asdb
      open_cursors             = 300
      pga_aggregate_target     = 848297984
    Mon Jul 23 16:20:18 2012
    Oracle instance running with ODM: Veritas 5.0.30.00 ODM Library, Version 1.1
    cluster interconnect IPC version:
            VERITAS IPC '5.0MP3' 07:30:15 Jul 29 2008
    IPC Vendor 86 proto 76
      Version 1.0
    PMON started with pid=2, OS id=22860
    DIAG started with pid=3, OS id=22870
    Mon Jul 23 16:20:18 2012
    Errors in file /u01/app/orainf/admin/asdb/bdump/asdb1_diag_22870.trc:
    ORA-07445: exception encountered: core dump [_kill()+8] [SIGIOT] [unknown code] [0x595600000000] [] []
    PSP0 started with pid=4, OS id=22888
    LMON started with pid=5, OS id=22892
    LMD0 started with pid=6, OS id=22899
    LMS0 started with pid=7, OS id=22901
    LMS1 started with pid=8, OS id=22905
    MMAN started with pid=9, OS id=22909
    DBW0 started with pid=10, OS id=22911
    LGWR started with pid=11, OS id=22913
    CKPT started with pid=12, OS id=22915
    SMON started with pid=13, OS id=22917
    RECO started with pid=14, OS id=22919
    CJQ0 started with pid=15, OS id=22921
    MMON started with pid=16, OS id=22923
    MMNL started with pid=17, OS id=22933
    Mon Jul 23 16:20:20 2012
    Errors in file /u01/app/orainf/admin/asdb/bdump/asdb1_lmon_22892.trc:
    ORA-27550: Target ID protocol check failed. tid vers=%d, type=%d, remote instance number=%d, local instance number=%d
    Mon Jul 23 16:20:20 2012
    LMON: terminating instance due to error 27550
    Instance terminated by LMON, pid = 22892instance 2:
    Mon Jul 23 16:20:30 2012
    Starting ORACLE instance (normal)
    LICENSE_MAX_SESSION = 0
    LICENSE_SESSIONS_WARNING = 0
    Interface type 1 bge3 10.0.0.0 configured from OCR for use as a cluster interconnect
    WARNING 10.0.0.0 could not be translated to a network address error 1
    Interface type 1 bge0 58.2.35.0 configured from OCR for use as  a public interface
    Interface type 1 bge1 58.2.35.0 configured from OCR for use as  a public interface
      WARNING: No cluster interconnect has been specified. Depending on
               the communication driver configured Oracle cluster traffic
               may be directed to the public interface of this machine.
               Oracle recommends that RAC clustered databases be configured
               with a private interconnect for enhanced security and
               performance.
    Picked latch-free SCN scheme 3
    Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/orainf/product/10.2.0/dbs/arch
    Autotune of undo retention is turned on.
    LICENSE_MAX_USERS = 0
    SYS auditing is disabled
    ksdpec: called for event 13740 prior to event group initialization
    Starting up ORACLE RDBMS Version: 10.2.0.3.0.
    System parameters with non-default values:
      processes                = 400
      __shared_pool_size       = 721420288
      shared_pool_size         = 201326592
      __large_pool_size        = 16777216
      __java_pool_size         = 150994944
      java_pool_size           = 150994944
      __streams_pool_size      = 16777216
      streams_pool_size        = 16777216
      spfile                   = /db01/asdb/asdb/spfileasdb.ora
      sga_target               = 1610612736
      control_files            = /db01/asdb/asdb/control01.ctl, /db01/asdb/asdb/control02.ctl, /db01/asdb/asdb/control03.ctl
      db_block_size            = 8192
      __db_cache_size          = 687865856
      db_cache_size            = 167772160
      compatible               = 10.2.0.3.0
      db_file_multiblock_read_count= 16
      cluster_database         = TRUE
      cluster_database_instances= 2
      thread                   = 2
      instance_number          = 2
      undo_management          = AUTO
      undo_tablespace          = UNDOTBS2
      ldap_directory_access    = PASSWORD
      remote_login_passwordfile= EXCLUSIVE
      db_domain                = intranet.genpact.com
      dispatchers              = (PROTOCOL=TCP) (SERVICE=asdbXDB)
      remote_listener          = LISTENERS_ASDB
      job_queue_processes      = 10
      background_dump_dest     = /u01/app/orainf/admin/asdb/bdump
      user_dump_dest           = /u01/app/orainf/admin/asdb/udump
      core_dump_dest           = /u01/app/orainf/admin/asdb/cdump
      audit_file_dest          = /u01/app/orainf/admin/asdb/adump
      db_name                  = asdb
      open_cursors             = 300
      pga_aggregate_target     = 848297984
      aq_tm_processes          = 1
    Cluster communication is configured to use the following interface(s) for this instance
      58.2.35.93
    Mon Jul 23 16:20:32 2012
    Oracle instance running with ODM: Veritas 5.0.30.00 ODM Library, Version 1.1
    cluster interconnect IPC version:Oracle UDP/IP (generic)
    IPC Vendor 1 proto 2
    PMON started with pid=2, OS id=28537
    DIAG started with pid=3, OS id=28539
    PSP0 started with pid=4, OS id=28541
    LMON started with pid=5, OS id=28543
    Mon Jul 23 16:20:33 2012
    WARNING: Failed to set buffer limit on IPC interconnect socket
    Oracle requires that the SocketReceive buffer size be tunable upto 1MB
    Please make sure the kernel parameterwhich limits SO_RCVBUF value set by
    applications is atleast 1MB
    LMD0 started with pid=6, OS id=28545
    LMS0 started with pid=7, OS id=28547
    LMS1 started with pid=8, OS id=28551
    MMAN started with pid=9, OS id=28555
    DBW0 started with pid=10, OS id=28557
    LGWR started with pid=11, OS id=28567
    CKPT started with pid=12, OS id=28587
    SMON started with pid=13, OS id=28594
    RECO started with pid=14, OS id=28596
    CJQ0 started with pid=15, OS id=28599
    MMON started with pid=16, OS id=28605
    MMNL started with pid=17, OS id=28608
    Mon Jul 23 16:20:34 2012
    starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
    starting up 1 shared server(s) ...
    Mon Jul 23 16:20:34 2012
    lmon registered with NM - instance id 2 (internal mem no 1)
    Mon Jul 23 16:20:35 2012
    Reconfiguration started (old inc 0, new inc 2)
    List of nodes:
    1
    Global Resource Directory frozen
    * allocate domain 0, invalid = TRUE
    Communication channels reestablished
    Master broadcasted resource hash value bitmaps
    Non-local Process blocks cleaned out
    Mon Jul 23 16:20:35 2012
    LMS 0: 0 GCS shadows cancelled, 0 closed
    Mon Jul 23 16:20:35 2012
    LMS 1: 0 GCS shadows cancelled, 0 closed
    Set master node info
    Submitted all remote-enqueue requests
    Dwn-cvts replayed, VALBLKs dubious
    All grantable enqueues granted
    Post SMON to start 1st pass IR
    Mon Jul 23 16:20:35 2012
    LMS 1: 0 GCS shadows traversed, 0 replayed
    Mon Jul 23 16:20:35 2012
    LMS 0: 0 GCS shadows traversed, 0 replayed
    Mon Jul 23 16:20:35 2012
    Submitted all GCS remote-cache requests
    Fix write in gcs resources
    Reconfiguration complete
    LCK0 started with pid=20, OS id=28770
    Mon Jul 23 16:20:36 2012
    ALTER DATABASE   MOUNT
    Mon Jul 23 16:20:36 2012
    This instance was first to mount
    Setting recovery target incarnation to 2
    Mon Jul 23 16:20:40 2012
    Successful mount of redo thread 2, with mount id 2392310356
    Mon Jul 23 16:20:40 2012
    Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)
    Completed: ALTER DATABASE   MOUNT
    Mon Jul 23 16:20:40 2012
    ALTER DATABASE OPEN
    This instance was first to open
    Mon Jul 23 16:20:40 2012
    Beginning crash recovery of 1 threads
    parallel recovery started with 2 processes
    Mon Jul 23 16:20:41 2012
    Started redo scan
    Mon Jul 23 16:20:41 2012
    Completed redo scan
    509 redo blocks read, 149 data blocks need recovery
    Mon Jul 23 16:20:41 2012
    Started redo application at
    Thread 2: logseq 1492, block 3
    Mon Jul 23 16:20:41 2012
    Recovery of Online Redo Log: Thread 2 Group 7 Seq 1492 Reading mem 0
      Mem# 0: /db01/asdb/asdb/redo07a.log
      Mem# 1: /db01/asdb/asdb/redo07b.log
    Mon Jul 23 16:20:41 2012
    Completed redo application
    Mon Jul 23 16:20:41 2012
    Completed crash recovery at
    Thread 2: logseq 1492, block 512, scn 10435264045528
    149 data blocks read, 149 data blocks written, 509 redo blocks read
    Picked broadcast on commit scheme to generate SCNs
    Mon Jul 23 16:20:42 2012
    Thread 2 advanced to log sequence 1493
    Thread 2 opened at log sequence 1493
      Current log# 8 seq# 1493 mem# 0: /db01/asdb/asdb/redo08a.log
      Current log# 8 seq# 1493 mem# 1: /db01/asdb/asdb/redo08b.log
    Successful open of redo thread 2
    Mon Jul 23 16:20:42 2012
    MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
    Mon Jul 23 16:20:42 2012
    SMON: enabling cache recovery
    Mon Jul 23 16:20:42 2012
    Successfully onlined Undo Tablespace 5.
    Mon Jul 23 16:20:42 2012
    SMON: enabling tx recovery
    Mon Jul 23 16:20:43 2012
    Database Characterset is AL32UTF8
    replication_dependency_tracking turned off (no async multimaster replication found)
    Starting background process QMNC
    QMNC started with pid=24, OS id=29088
    Mon Jul 23 16:20:46 2012
    Completed: ALTER DATABASE OPENPlease suggest
    Regards
    KK

    Hi ,
    I have used the command ifconfig -a
    below is the output :
    chaning the original IP's with alphabets for security concerns
    instance 1:
    bash-2.05$ ifconfig -a
    lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1
            inet 127.0.0.1 netmask ff000000
    bge0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
            inet XY.M.CD.93 netmask ffffff00 broadcast XY.M.CD.255
            groupname orapub
    bge0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 2
            inet XY.M.CD.94 netmask ffffff00 broadcast XY.M.CD.255
    bge0:3: flags=1040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4> mtu 1500 index 2
            inet XY.M.CD.151 netmask ffffff00 broadcast XY.M.CD.255
    bge1: flags=39040803<UP,BROADCAST,MULTICAST,DEPRECATED,IPv4,NOFAILOVER,FAILED,STANDBY> mtu 1500 index 3
            inet XY.M.CD.95 netmask ffffff00 broadcast XY.M.CD.255
            groupname orapub
    bge2: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 4
            inet ZZ.0.0.1 netmask ff000000 broadcast ZZ.255.255.255
    bge3: flags=1000842<BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 5
            inet 0.0.0.0 netmask 0instance 2:
    bash-2.05$ ifconfig -a
    lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1
            inet 127.0.0.1 netmask ff000000
    bge0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
            inet XY.M.CD.96 netmask ffffff00 broadcast XY.M.CD.255
            groupname orapub
    bge0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 2
            inet XY.M.CD.97 netmask ffffff00 broadcast XY.M.CD.255
    bge0:2: flags=1040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4> mtu 1500 index 2
            inet XY.M.CD.152 netmask ffffff00 broadcast XY.M.CD.255
    bge1: flags=69040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER,STANDBY,INACTIVE> mtu 1500 index 3
            inet XY.M.CD.98 netmask ffffff00 broadcast XY.M.CD.255
            groupname orapub
    bge3: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 5
            inet 0.0.0.0 netmask ff000000
    bge3:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 5
            inet 10.0.0.3 netmask ff000000 broadcast ZZ.255.255.255Regards

  • Can not start Rac database

    Hi,
    Oracle RAC database 10.2.0.3/RedHat4 with 2 nodes.
    In the begining we had an error ORA-600[12803] so only sys can connect to database I find the note 1026653.6 this note said that we need to create AUDSES$ sequence but befor that we have to restart the database.
    When we stop the datanbase we had another ORA-600 and it's impossible to start it!!
    Here is a coppy of our alert file:
    Picked latch-free SCN scheme 2
    Autotune of undo retention is turned on.
    LICENSE_MAX_USERS = 0
    SYS auditing is disabled
    ksdpec: called for event 13740 prior to event group initialization
    Starting up ORACLE RDBMS Version: 10.2.0.3.0.
    System parameters with non-default values:
    processes = 300
    sessions = 335
    sga_max_size = 524288000
    __shared_pool_size = 310378496
    __large_pool_size = 4194304
    __java_pool_size = 8388608
    __streams_pool_size = 8388608
    spfile = +DATA/osista/spfileosista.ora
    nls_language = FRENCH
    nls_territory = FRANCE
    nls_length_semantics = CHAR
    sga_target = 524288000
    control_files = DATA/osista/controlfile/control01.ctl, DATA/osista/controlfile/control02.ctl
    db_block_size = 8192
    __db_cache_size = 184549376
    compatible = 10.2.0.3.0
    log_archive_dest_1 = LOCATION=USE_DB_RECOVERY_FILE_DEST
    db_file_multiblock_read_count= 16
    cluster_database = TRUE
    cluster_database_instances= 2
    db_create_file_dest = +DATA
    db_recovery_file_dest = +FLASH
    db_recovery_file_dest_size= 68543315968
    thread = 2
    instance_number = 2
    undo_management = AUTO
    undo_tablespace = UNDOTBS2
    undo_retention = 29880
    remote_login_passwordfile= EXCLUSIVE
    db_domain =
    dispatchers = (PROTOCOL=TCP) (SERVICE=OSISTAXDB)
    local_listener = (address=(protocol=tcp)(port=1521)(host=132.147.160.243))
    remote_listener = LISTENERS_OSISTA
    job_queue_processes = 10
    background_dump_dest = /oracle/product/admin/OSISTA/bdump
    user_dump_dest = /oracle/product/admin/OSISTA/udump
    core_dump_dest = /oracle/product/admin/OSISTA/cdump
    audit_file_dest = /oracle/product/admin/OSISTA/adump
    db_name = OSISTA
    open_cursors = 300
    pga_aggregate_target = 104857600
    aq_tm_processes = 1
    Cluster communication is configured to use the following interface(s) for this instance
    172.16.0.2
    Wed Jun 13 11:04:30 2012
    cluster interconnect IPC version:Oracle UDP/IP (generic)
    IPC Vendor 1 proto 2
    PMON started with pid=2, OS id=8560
    DIAG started with pid=3, OS id=8562
    PSP0 started with pid=4, OS id=8566
    LMON started with pid=5, OS id=8570
    LMD0 started with pid=6, OS id=8574
    LMS0 started with pid=7, OS id=8576
    LMS1 started with pid=8, OS id=8580
    MMAN started with pid=9, OS id=8584
    DBW0 started with pid=10, OS id=8586
    LGWR started with pid=11, OS id=8588
    CKPT started with pid=12, OS id=8590
    SMON started with pid=13, OS id=8592
    RECO started with pid=14, OS id=8594
    CJQ0 started with pid=15, OS id=8596
    MMON started with pid=16, OS id=8598
    Wed Jun 13 11:04:31 2012
    starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
    MMNL started with pid=17, OS id=8600
    Wed Jun 13 11:04:31 2012
    starting up 1 shared server(s) ...
    Wed Jun 13 11:04:31 2012
    lmon registered with NM - instance id 2 (internal mem no 1)
    Wed Jun 13 11:04:31 2012
    Reconfiguration started (old inc 0, new inc 2)
    List of nodes:
    1
    Global Resource Directory frozen
    * allocate domain 0, invalid = TRUE
    Communication channels reestablished
    Master broadcasted resource hash value bitmaps
    Non-local Process blocks cleaned out
    Wed Jun 13 11:04:31 2012
    LMS 0: 0 GCS shadows cancelled, 0 closed
    Wed Jun 13 11:04:31 2012
    LMS 1: 0 GCS shadows cancelled, 0 closed
    Set master node info
    Submitted all remote-enqueue requests
    Dwn-cvts replayed, VALBLKs dubious
    All grantable enqueues granted
    Post SMON to start 1st pass IR
    Wed Jun 13 11:04:31 2012
    LMS 0: 0 GCS shadows traversed, 0 replayed
    Wed Jun 13 11:04:31 2012
    LMS 1: 0 GCS shadows traversed, 0 replayed
    Wed Jun 13 11:04:31 2012
    Submitted all GCS remote-cache requests
    Fix write in gcs resources
    Reconfiguration complete
    LCK0 started with pid=20, OS id=8877
    Wed Jun 13 11:04:43 2012
    alter database mount
    Wed Jun 13 11:04:43 2012
    This instance was first to mount
    Wed Jun 13 11:04:43 2012
    Starting background process ASMB
    ASMB started with pid=25, OS id=10068
    Starting background process RBAL
    RBAL started with pid=26, OS id=10072
    Wed Jun 13 11:04:47 2012
    SUCCESS: diskgroup DATA was mounted
    Wed Jun 13 11:04:51 2012
    Setting recovery target incarnation to 1
    Wed Jun 13 11:04:52 2012
    Successful mount of redo thread 2, with mount id 3005749259
    Wed Jun 13 11:04:52 2012
    Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)
    Completed: alter database mount
    Wed Jun 13 11:05:06 2012
    alter database open
    Wed Jun 13 11:05:06 2012
    This instance was first to open
    Wed Jun 13 11:05:06 2012
    Beginning crash recovery of 1 threads
    parallel recovery started with 2 processes
    Wed Jun 13 11:05:07 2012
    Started redo scan
    Wed Jun 13 11:05:07 2012
    Completed redo scan
    61 redo blocks read, 4 data blocks need recovery
    Wed Jun 13 11:05:07 2012
    Started redo application at
    Thread 1: logseq 7924, block 3, scn 506098125
    Wed Jun 13 11:05:07 2012
    Recovery of Online Redo Log: Thread 1 Group 2 Seq 7924 Reading mem 0
    Mem# 0: +DATA/osista/onlinelog/group_2.372.742132543
    Wed Jun 13 11:05:07 2012
    Completed redo application
    Wed Jun 13 11:05:07 2012
    Completed crash recovery at
    Thread 1: logseq 7924, block 64, scn 506118186
    4 data blocks read, 4 data blocks written, 61 redo blocks read
    Switch log for thread 1 to sequence 7925
    Picked broadcast on commit scheme to generate SCNs
    db_recovery_file_dest_size of 65368 MB is 0.61% used. This is a
    user-specified limit on the amount of space that will be used by this
    database for recovery-related files, and does not reflect the amount of
    space available in the underlying filesystem or ASM diskgroup.
    SUCCESS: diskgroup FLASH was mounted
    SUCCESS: diskgroup FLASH was dismounted
    Thread 1 advanced to log sequence 7926
    SUCCESS: diskgroup FLASH was mounted
    SUCCESS: diskgroup FLASH was dismounted
    Thread 1 advanced to log sequence 7927
    Wed Jun 13 11:05:11 2012
    LGWR: STARTING ARCH PROCESSES
    ARC0 started with pid=31, OS id=12747
    Wed Jun 13 11:05:11 2012
    ARC0: Archival started
    ARC1: Archival started
    LGWR: STARTING ARCH PROCESSES COMPLETE
    ARC1 started with pid=32, OS id=12749
    Wed Jun 13 11:05:12 2012
    Thread 2 opened at log sequence 7176
    Current log# 4 seq# 7176 mem# 0: +DATA/osista/onlinelog/group_4.289.742134597
    Wed Jun 13 11:05:12 2012
    ARC1: Becoming the 'no FAL' ARCH
    ARC1: Becoming the 'no SRL' ARCH
    Wed Jun 13 11:05:12 2012
    Successful open of redo thread 2
    Wed Jun 13 11:05:12 2012
    MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
    Wed Jun 13 11:05:12 2012
    ARC0: Becoming the heartbeat ARCH
    Wed Jun 13 11:05:12 2012
    SMON: enabling cache recovery
    Wed Jun 13 11:05:15 2012
    Successfully onlined Undo Tablespace 20.
    Wed Jun 13 11:05:15 2012
    SMON: enabling tx recovery
    Wed Jun 13 11:05:15 2012
    Database Characterset is AL32UTF8
    Wed Jun 13 11:05:16 2012
    Errors in file /oracle/product/admin/OSISTA/udump/osista2_ora_9174.trc:
    ORA-00600: code d'erreur interne, arguments : [kokiasg1], [], [], [], [], [], [], []
    Wed Jun 13 11:05:16 2012
    Errors in file /oracle/product/admin/OSISTA/udump/osista2_ora_9174.trc:
    ORA-00600: code d'erreur interne, arguments : [kokiasg1], [], [], [], [], [], [], []
    Error 600 happened during db open, shutting down database
    USER: terminating instance due to error 600
    Instance terminated by USER, pid = 9174
    ORA-1092 signalled during: alter database open...
    Wed Jun 13 11:06:16 2012
    Starting ORACLE instance (normal)
    LICENSE_MAX_SESSION = 0
    LICENSE_SESSIONS_WARNING = 0
    Interface type 1 eth0 172.16.0.0 configured from OCR for use as a cluster interconnect
    Interface type 1 bond0 132.147.160.0 configured from OCR for use as a public interface
    Picked latch-free SCN scheme 2
    Autotune of undo retention is turned on.
    LICENSE_MAX_USERS = 0
    SYS auditing is disabled
    ksdpec: called for event 13740 prior to event group initialization
    Starting up ORACLE RDBMS Version: 10.2.0.3.0.
    System parameters with non-default values:
    processes = 300
    sessions = 335
    sga_max_size = 524288000
    __shared_pool_size = 314572800
    __large_pool_size = 4194304
    __java_pool_size = 8388608
    __streams_pool_size = 8388608
    spfile = +DATA/osista/spfileosista.ora
    nls_language = FRENCH
    nls_territory = FRANCE
    nls_length_semantics = CHAR
    sga_target = 524288000
    control_files = DATA/osista/controlfile/control01.ctl, DATA/osista/controlfile/control02.ctl
    db_block_size = 8192
    __db_cache_size = 180355072
    compatible = 10.2.0.3.0
    log_archive_dest_1 = LOCATION=USE_DB_RECOVERY_FILE_DEST
    db_file_multiblock_read_count= 16
    cluster_database = TRUE
    cluster_database_instances= 2
    db_create_file_dest = +DATA
    db_recovery_file_dest = +FLASH
    db_recovery_file_dest_size= 68543315968
    thread = 2
    instance_number = 2
    undo_management = AUTO
    undo_tablespace = UNDOTBS2
    undo_retention = 29880
    remote_login_passwordfile= EXCLUSIVE
    db_domain =
    dispatchers = (PROTOCOL=TCP) (SERVICE=OSISTAXDB)
    local_listener = (address=(protocol=tcp)(port=1521)(host=132.147.160.243))
    remote_listener = LISTENERS_OSISTA
    job_queue_processes = 10
    background_dump_dest = /oracle/product/admin/OSISTA/bdump
    user_dump_dest = /oracle/product/admin/OSISTA/udump
    core_dump_dest = /oracle/product/admin/OSISTA/cdump
    audit_file_dest = /oracle/product/admin/OSISTA/adump
    db_name = OSISTA
    open_cursors = 300
    pga_aggregate_target = 104857600
    aq_tm_processes = 1
    Cluster communication is configured to use the following interface(s) for this instance
    172.16.0.2
    Wed Jun 13 11:06:16 2012
    cluster interconnect IPC version:Oracle UDP/IP (generic)
    IPC Vendor 1 proto 2
    PMON started with pid=2, OS id=18682
    DIAG started with pid=3, OS id=18684
    PSP0 started with pid=4, OS id=18695
    LMON started with pid=5, OS id=18704
    LMD0 started with pid=6, OS id=18721
    LMS0 started with pid=7, OS id=18735
    LMS1 started with pid=8, OS id=18753
    MMAN started with pid=9, OS id=18767
    DBW0 started with pid=10, OS id=18788
    LGWR started with pid=11, OS id=18796
    CKPT started with pid=12, OS id=18799
    SMON started with pid=13, OS id=18801
    RECO started with pid=14, OS id=18803
    CJQ0 started with pid=15, OS id=18805
    MMON started with pid=16, OS id=18807
    Wed Jun 13 11:06:17 2012
    starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
    MMNL started with pid=17, OS id=18809
    Wed Jun 13 11:06:17 2012
    starting up 1 shared server(s) ...
    Wed Jun 13 11:06:17 2012
    lmon registered with NM - instance id 2 (internal mem no 1)
    Wed Jun 13 11:06:17 2012
    Reconfiguration started (old inc 0, new inc 2)
    List of nodes:
    1
    Global Resource Directory frozen
    * allocate domain 0, invalid = TRUE
    Communication channels reestablished
    Master broadcasted resource hash value bitmaps
    Non-local Process blocks cleaned out
    Wed Jun 13 11:06:18 2012
    LMS 0: 0 GCS shadows cancelled, 0 closed
    Wed Jun 13 11:06:18 2012
    LMS 1: 0 GCS shadows cancelled, 0 closed
    Set master node info
    Submitted all remote-enqueue requests
    Dwn-cvts replayed, VALBLKs dubious
    All grantable enqueues granted
    Post SMON to start 1st pass IR
    Wed Jun 13 11:06:18 2012
    LMS 0: 0 GCS shadows traversed, 0 replayed
    Wed Jun 13 11:06:18 2012
    LMS 1: 0 GCS shadows traversed, 0 replayed
    Wed Jun 13 11:06:18 2012
    Submitted all GCS remote-cache requests
    Fix write in gcs resources
    Reconfiguration complete
    LCK0 started with pid=20, OS id=18816
    Wed Jun 13 11:06:18 2012
    ALTER DATABASE MOUNT
    Wed Jun 13 11:06:18 2012
    This instance was first to mount
    Wed Jun 13 11:06:18 2012
    Reconfiguration started (old inc 2, new inc 4)
    List of nodes:
    0 1
    Wed Jun 13 11:06:18 2012
    Starting background process ASMB
    Wed Jun 13 11:06:18 2012
    Global Resource Directory frozen
    Communication channels reestablished
    ASMB started with pid=22, OS id=18913
    Starting background process RBAL
    * domain 0 valid = 0 according to instance 0
    Wed Jun 13 11:06:18 2012
    Master broadcasted resource hash value bitmaps
    Non-local Process blocks cleaned out
    Wed Jun 13 11:06:18 2012
    LMS 0: 0 GCS shadows cancelled, 0 closed
    Wed Jun 13 11:06:18 2012
    LMS 1: 0 GCS shadows cancelled, 0 closed
    Set master node info
    Submitted all remote-enqueue requests
    Dwn-cvts replayed, VALBLKs dubious
    All grantable enqueues granted
    Wed Jun 13 11:06:18 2012
    LMS 0: 0 GCS shadows traversed, 0 replayed
    Wed Jun 13 11:06:18 2012
    LMS 1: 0 GCS shadows traversed, 0 replayed
    Wed Jun 13 11:06:18 2012
    Submitted all GCS remote-cache requests
    Fix write in gcs resources
    RBAL started with pid=23, OS id=18917
    Reconfiguration complete
    Wed Jun 13 11:06:22 2012
    SUCCESS: diskgroup DATA was mounted
    Wed Jun 13 11:06:26 2012
    Setting recovery target incarnation to 1
    Wed Jun 13 11:06:26 2012
    Successful mount of redo thread 2, with mount id 3005703530
    Wed Jun 13 11:06:26 2012
    Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)
    Completed: ALTER DATABASE MOUNT
    Wed Jun 13 11:06:27 2012
    ALTER DATABASE OPEN
    This instance was first to open
    Wed Jun 13 11:06:27 2012
    Beginning crash recovery of 1 threads
    parallel recovery started with 2 processes
    Wed Jun 13 11:06:27 2012
    Started redo scan
    Wed Jun 13 11:06:27 2012
    Completed redo scan
    61 redo blocks read, 4 data blocks need recovery
    Wed Jun 13 11:06:28 2012
    Started redo application at
    Thread 2: logseq 7176, block 3
    Wed Jun 13 11:06:28 2012
    Recovery of Online Redo Log: Thread 2 Group 4 Seq 7176 Reading mem 0
    Mem# 0: +DATA/osista/onlinelog/group_4.289.742134597
    Wed Jun 13 11:06:28 2012
    Completed redo application
    Wed Jun 13 11:06:28 2012
    Completed crash recovery at
    Thread 2: logseq 7176, block 64, scn 506138248
    4 data blocks read, 4 data blocks written, 61 redo blocks read
    Picked broadcast on commit scheme to generate SCNs
    Wed Jun 13 11:06:28 2012
    LGWR: STARTING ARCH PROCESSES
    ARC0 started with pid=28, OS id=19692
    Wed Jun 13 11:06:28 2012
    ARC0: Archival started
    ARC1: Archival started
    LGWR: STARTING ARCH PROCESSES COMPLETE
    ARC1 started with pid=29, OS id=19695
    Wed Jun 13 11:06:28 2012
    Thread 2 advanced to log sequence 7177
    Thread 2 opened at log sequence 7177
    Current log# 3 seq# 7177 mem# 0: +DATA/osista/onlinelog/group_3.291.742134597
    Successful open of redo thread 2
    Wed Jun 13 11:06:28 2012
    MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
    Wed Jun 13 11:06:28 2012
    ARC0: Becoming the 'no FAL' ARCH
    ARC0: Becoming the 'no SRL' ARCH
    Wed Jun 13 11:06:28 2012
    ARC1: Becoming the heartbeat ARCH
    Wed Jun 13 11:06:28 2012
    SMON: enabling cache recovery
    Wed Jun 13 11:06:28 2012
    db_recovery_file_dest_size of 65368 MB is 0.61% used. This is a
    user-specified limit on the amount of space that will be used by this
    database for recovery-related files, and does not reflect the amount of
    space available in the underlying filesystem or ASM diskgroup.
    SUCCESS: diskgroup FLASH was mounted
    SUCCESS: diskgroup FLASH was dismounted
    Wed Jun 13 11:06:31 2012
    Successfully onlined Undo Tablespace 20.
    Wed Jun 13 11:06:31 2012
    SMON: enabling tx recovery
    Wed Jun 13 11:06:31 2012
    Database Characterset is AL32UTF8
    Wed Jun 13 11:06:31 2012
    Errors in file /oracle/product/admin/OSISTA/udump/osista2_ora_19596.trc:
    ORA-00600: code d'erreur interne, arguments : [kokiasg1], [], [], [], [], [], [], []
    Wed Jun 13 11:06:32 2012
    Errors in file /oracle/product/admin/OSISTA/udump/osista2_ora_19596.trc:
    ORA-00600: code d'erreur interne, arguments : [kokiasg1], [], [], [], [], [], [], []
    Error 600 happened during db open, shutting down database
    USER: terminating instance due to error 600
    Instance terminated by USER, pid = 19596
    ORA-1092 signalled during: ALTER DATABASE OPEN...
    Wed Jun 13 11:11:35 2012
    Starting ORACLE instance (normal)
    LICENSE_MAX_SESSION = 0
    LICENSE_SESSIONS_WARNING = 0
    Interface type 1 eth0 172.16.0.0 configured from OCR for use as a cluster interconnect
    Interface type 1 bond0 132.147.160.0 configured from OCR for use as a public interface
    Picked latch-free SCN scheme 2
    Autotune of undo retention is turned on.
    LICENSE_MAX_USERS = 0
    SYS auditing is disabled
    ksdpec: called for event 13740 prior to event group initialization
    Starting up ORACLE RDBMS Version: 10.2.0.3.0.
    System parameters with non-default values:
    processes = 300
    sessions = 335
    sga_max_size = 524288000
    __shared_pool_size = 318767104
    __large_pool_size = 4194304
    __java_pool_size = 8388608
    __streams_pool_size = 8388608
    spfile = +DATA/osista/spfileosista.ora
    nls_language = FRENCH
    nls_territory = FRANCE
    nls_length_semantics = CHAR
    sga_target = 524288000
    control_files = DATA/osista/controlfile/control01.ctl, DATA/osista/controlfile/control02.ctl
    db_block_size = 8192
    __db_cache_size = 176160768
    compatible = 10.2.0.3.0
    log_archive_dest_1 = LOCATION=USE_DB_RECOVERY_FILE_DEST
    db_file_multiblock_read_count= 16
    cluster_database = TRUE
    cluster_database_instances= 2
    db_create_file_dest = +DATA
    db_recovery_file_dest = +FLASH
    db_recovery_file_dest_size= 68543315968
    thread = 2
    instance_number = 2
    undo_management = AUTO
    undo_tablespace = UNDOTBS2
    undo_retention = 29880
    remote_login_passwordfile= EXCLUSIVE
    db_domain =
    dispatchers = (PROTOCOL=TCP) (SERVICE=OSISTAXDB)
    local_listener = (address=(protocol=tcp)(port=1521)(host=132.147.160.243))
    remote_listener = LISTENERS_OSISTA
    job_queue_processes = 10
    background_dump_dest = /oracle/product/admin/OSISTA/bdump
    user_dump_dest = /oracle/product/admin/OSISTA/udump
    core_dump_dest = /oracle/product/admin/OSISTA/cdump
    audit_file_dest = /oracle/product/admin/OSISTA/adump
    db_name = OSISTA
    open_cursors = 300
    pga_aggregate_target = 104857600
    aq_tm_processes = 1
    Cluster communication is configured to use the following interface(s) for this instance
    172.16.0.2
    Wed Jun 13 11:11:35 2012
    cluster interconnect IPC version:Oracle UDP/IP (generic)
    IPC Vendor 1 proto 2
    PMON started with pid=2, OS id=16101
    DIAG started with pid=3, OS id=16103
    PSP0 started with pid=4, OS id=16105
    LMON started with pid=5, OS id=16107
    LMD0 started with pid=6, OS id=16110
    LMS0 started with pid=7, OS id=16112
    LMS1 started with pid=8, OS id=16116
    MMAN started with pid=9, OS id=16120
    DBW0 started with pid=10, OS id=16132
    LGWR started with pid=11, OS id=16148
    CKPT started with pid=12, OS id=16169
    SMON started with pid=13, OS id=16185
    RECO started with pid=14, OS id=16203
    CJQ0 started with pid=15, OS id=16219
    MMON started with pid=16, OS id=16227
    Wed Jun 13 11:11:36 2012
    starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
    MMNL started with pid=17, OS id=16229
    Wed Jun 13 11:11:36 2012
    starting up 1 shared server(s) ...
    Wed Jun 13 11:11:36 2012
    lmon registered with NM - instance id 2 (internal mem no 1)
    Wed Jun 13 11:11:36 2012
    Reconfiguration started (old inc 0, new inc 2)
    List of nodes:
    1
    Global Resource Directory frozen
    * allocate domain 0, invalid = TRUE
    Communication channels reestablished
    Master broadcasted resource hash value bitmaps
    Non-local Process blocks cleaned out
    Wed Jun 13 11:11:36 2012
    LMS 0: 0 GCS shadows cancelled, 0 closed
    Wed Jun 13 11:11:36 2012
    LMS 1: 0 GCS shadows cancelled, 0 closed
    Set master node info
    Submitted all remote-enqueue requests
    Dwn-cvts replayed, VALBLKs dubious
    All grantable enqueues granted
    Post SMON to start 1st pass IR
    Wed Jun 13 11:11:36 2012
    LMS 1: 0 GCS shadows traversed, 0 replayed
    Wed Jun 13 11:11:36 2012
    LMS 0: 0 GCS shadows traversed, 0 replayed
    Wed Jun 13 11:11:36 2012
    Submitted all GCS remote-cache requests
    Fix write in gcs resources
    Reconfiguration complete
    LCK0 started with pid=20, OS id=16235
    Wed Jun 13 11:11:37 2012
    ALTER DATABASE MOUNT
    Wed Jun 13 11:11:37 2012
    This instance was first to mount
    Wed Jun 13 11:11:37 2012
    Starting background process ASMB
    ASMB started with pid=22, OS id=16343
    Starting background process RBAL
    RBAL started with pid=23, OS id=16347
    Wed Jun 13 11:11:44 2012
    SUCCESS: diskgroup DATA was mounted
    Wed Jun 13 11:11:49 2012
    Setting recovery target incarnation to 1
    Wed Jun 13 11:11:49 2012
    Successful mount of redo thread 2, with mount id 3005745065
    Wed Jun 13 11:11:49 2012
    Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)
    Completed: ALTER DATABASE MOUNT
    Wed Jun 13 11:22:25 2012
    alter database open
    This instance was first to open
    Wed Jun 13 11:22:26 2012
    Beginning crash recovery of 1 threads
    parallel recovery started with 2 processes
    Wed Jun 13 11:22:26 2012
    Started redo scan
    Wed Jun 13 11:22:26 2012
    Completed redo scan
    61 redo blocks read, 4 data blocks need recovery
    Wed Jun 13 11:22:26 2012
    Started redo application at
    Thread 1: logseq 7927, block 3
    Wed Jun 13 11:22:26 2012
    Recovery of Online Redo Log: Thread 1 Group 1 Seq 7927 Reading mem 0
    Mem# 0: +DATA/osista/onlinelog/group_1.283.742132543
    Wed Jun 13 11:22:26 2012
    Completed redo application
    Wed Jun 13 11:22:26 2012
    Completed crash recovery at
    Thread 1: logseq 7927, block 64, scn 506178382
    4 data blocks read, 4 data blocks written, 61 redo blocks read
    Switch log for thread 1 to sequence 7928
    Picked broadcast on commit scheme to generate SCNs
    Wed Jun 13 11:22:27 2012
    LGWR: STARTING ARCH PROCESSES
    ARC0 started with pid=31, OS id=13010
    Wed Jun 13 11:22:27 2012
    ARC0: Archival started
    ARC1: Archival started
    LGWR: STARTING ARCH PROCESSES COMPLETE
    ARC1 started with pid=32, OS id=13033
    Wed Jun 13 11:22:27 2012
    Thread 2 opened at log sequence 7178
    Current log# 4 seq# 7178 mem# 0: +DATA/osista/onlinelog/group_4.289.742134597
    Successful open of redo thread 2
    Wed Jun 13 11:22:27 2012
    MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
    Wed Jun 13 11:22:27 2012
    ARC0: Becoming the 'no FAL' ARCH
    ARC0: Becoming the 'no SRL' ARCH
    Wed Jun 13 11:22:27 2012
    ARC1: Becoming the heartbeat ARCH
    Wed Jun 13 11:22:27 2012
    SMON: enabling cache recovery
    Wed Jun 13 11:22:30 2012
    db_recovery_file_dest_size of 65368 MB is 0.61% used. This is a
    user-specified limit on the amount of space that will be used by this
    database for recovery-related files, and does not reflect the amount of
    space available in the underlying filesystem or ASM diskgroup.
    SUCCESS: diskgroup FLASH was mounted
    SUCCESS: diskgroup FLASH was dismounted
    Wed Jun 13 11:22:31 2012
    Successfully onlined Undo Tablespace 20.
    Wed Jun 13 11:22:31 2012
    SMON: enabling tx recovery
    Wed Jun 13 11:22:32 2012
    Database Characterset is AL32UTF8
    Wed Jun 13 11:22:32 2012
    Errors in file /oracle/product/admin/OSISTA/udump/osista2_ora_11751.trc:
    ORA-00600: code d'erreur interne, arguments : [kokiasg1], [], [], [], [], [], [], []
    Wed Jun 13 11:22:33 2012
    Errors in file /oracle/product/admin/OSISTA/udump/osista2_ora_11751.trc:
    ORA-00600: code d'erreur interne, arguments : [kokiasg1], [], [], [], [], [], [], []
    Error 600 happened during db open, shutting down database
    USER: terminating instance due to error 600
    Instance terminated by USER, pid = 11751
    ORA-1092 signalled during: alter database open...
    regards,

    Hi;
    Errors in file /oracle/product/admin/OSISTA/udump/osista2_ora_9174.trc:Did you check trc file?
    ORA-00600: code d'erreur interne, arguments : [kokiasg1], [], [], [], [], [], [], []You are getting oracle internal error(ORA 600) which mean you could need to work wiht oracle support team. Please see below note, if its not help than i suggest log a sr:
    Troubleshoot an ORA-600 or ORA-7445 Error Using the Error Lookup Tool [ID 153788.1]
    for your future rac issue please use Forum Home » High Availability » RAC, ASM & Clusterware Installation which is RAC dedicated forum site.
    Regard
    Helios

  • Rac node restart

    Hello everyone,
    I have met an error,that is our RAC node auto restart with below messages.
    #/u01/app/oracle/diag/rdbms/odsdb/odsdb1/trace/alert_odsdb1.log
    Fri Jun 07 12:23:42 2013
    Thread 1 cannot allocate new log, sequence 58363
    Checkpoint not complete
    Current log# 2 seq# 58362 mem# 0: +DATA/odsdb/onlinelog/group_2.265.812288839
    Current log# 2 seq# 58362 mem# 1: +DATA/odsdb/onlinelog/group_2.266.812288839
    Fri Jun 07 12:23:42 2013
    NOTE: ASMB terminating
    Errors in file /u01/app/oracle/diag/rdbms/odsdb/odsdb1/trace/odsdb1_asmb_32641.trc:
    ORA-15064: ? ASM ??????
    ORA-03113: ?????????
    ?? ID:
    ?? ID: 2047 ???: 5
    Errors in file /u01/app/oracle/diag/rdbms/odsdb/odsdb1/trace/odsdb1_asmb_32641.trc:
    ORA-15064: ? ASM ??????
    ORA-03113: ?????????
    ?? ID:
    ?? ID: 2047 ???: 5
    ASMB (ospid: 32641): terminating the instance due to error 15064
    Fri Jun 07 12:23:44 2013
    ORA-1092 : opitsk aborting process
    Fri Jun 07 12:23:46 2013
    ORA-1092 : opitsk aborting process
    Instance terminated by ASMB, pid = 32641
    Fri Jun 07 12:25:02 2013
    Starting ORACLE instance (normal)
    Fri Jun 07 12:25:23 2013
    LICENSE_MAX_SESSION = 0
    LICENSE_SESSIONS_WARNING = 0
    Private Interface 'eth1:1' configured from GPnP for use as a private interconnect.
    [name='eth1:1', type=1, ip=169.254.37.103, mac=00-26-55-eb-61-89, net=169.254.0.0/16, mask=255.255.0.0, use=haip:cluster_interconnect/62]
    Public Interface 'eth0' configured from GPnP for use as a public interface.
    [name='eth0', type=1, ip=135.33.2.8, mac=00-26-55-eb-61-88, net=135.33.2.0/27, mask=255.255.255.224, use=public/1]
    Public Interface 'eth0:1' configured from GPnP for use as a public interface.
    [name='eth0:1', type=1, ip=135.33.2.13, mac=00-26-55-eb-61-88, net=135.33.2.0/27, mask=255.255.255.224, use=public/1]
    Picked latch-free SCN scheme 3
    Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.2.0/dbhome_2/dbs/arch
    Autotune of undo retention is turned on.
    LICENSE_MAX_USERS = 0
    SYS auditing is disabled
    Starting up:
    Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
    With the Partitioning, Real Application Clusters, OLAP, Data Mining
    and Real Application Testing options.
    ORACLE_HOME = /u01/app/oracle/product/11.2.0/dbhome_2
    System name:     Linux
    Node name:     odsdb1
    Release:     2.6.18-308.el5
    Version:     #1 SMP Fri Jan 27 17:17:51 EST 2012
    Machine:     x86_64
    Using parameter settings in server-side pfile /u01/app/oracle/product/11.2.0/dbhome_2/dbs/initodsdb1.ora
    System parameters with non-default values:
    processes = 4500
    sessions = 6784
    event = ""
    spfile = "+DATA/odsdb/spfileodsdb.ora"
    nls_language = "SIMPLIFIED CHINESE"
    nls_territory = "CHINA"
    memory_target = 170G
    control_files = "+DATA/odsdb/controlfile/current.262.812288837"
    control_files = "+DATA/odsdb/controlfile/current.261.812288837"
    db_block_size = 8192
    compatible = "11.2.0.0.0"
    db_files = 4096
    cluster_database = TRUE
    db_create_file_dest = "+DATA"
    db_recovery_file_dest = ""
    db_recovery_file_dest_size= 38820M
    thread = 1
    undo_tablespace = "UNDOTBS1"
    instance_number = 1
    remote_login_passwordfile= "EXCLUSIVE"
    db_domain = ""
    dispatchers = "(PROTOCOL=TCP) (SERVICE=odsdbXDB)"
    remote_listener = "odsdb-cluster-scan:1521"
    job_queue_processes = 1000
    audit_file_dest = "/u01/app/oracle/admin/odsdb/adump"
    audit_trail = "DB"
    db_name = "odsdb"
    open_cursors = 300
    diagnostic_dest = "/u01/app/oracle"
    Cluster communication is configured to use the following interface(s) for this instance
    169.254.37.103
    cluster interconnect IPC version:Oracle UDP/IP (generic)
    IPC Vendor 1 proto 2
    Fri Jun 07 12:25:33 2013
    PMON started with pid=2, OS id=22959
    Fri Jun 07 12:25:33 2013
    PSP0 started with pid=3, OS id=22962
    Fri Jun 07 12:25:34 2013
    VKTM started with pid=4, OS id=22971 at elevated priority
    VKTM running at (1)millisec precision with DBRM quantum (100)ms
    Fri Jun 07 12:25:34 2013
    GEN0 started with pid=5, OS id=22977
    Fri Jun 07 12:25:34 2013
    DIAG started with pid=6, OS id=22979
    Fri Jun 07 12:25:35 2013
    DBRM started with pid=7, OS id=22981
    Fri Jun 07 12:25:35 2013
    PING started with pid=8, OS id=22983
    Fri Jun 07 12:25:35 2013
    ACMS started with pid=9, OS id=22985
    Fri Jun 07 12:25:35 2013
    DIA0 started with pid=10, OS id=22987
    Fri Jun 07 12:25:35 2013
    LMON started with pid=11, OS id=22989
    Fri Jun 07 12:25:35 2013
    LMD0 started with pid=12, OS id=22991
    * Load Monitor used for high load check
    * New Low - High Load Threshold Range = [61440 - 81920]
    Fri Jun 07 12:25:35 2013
    LMS0 started with pid=13, OS id=22994 at elevated priority
    Fri Jun 07 12:25:35 2013
    LMS1 started with pid=14, OS id=22998 at elevated priority
    Fri Jun 07 12:25:35 2013
    LMS2 started with pid=15, OS id=23002 at elevated priority
    Fri Jun 07 12:25:35 2013
    LMS3 started with pid=16, OS id=23006 at elevated priority
    Fri Jun 07 12:25:35 2013
    RMS0 started with pid=17, OS id=23010
    Fri Jun 07 12:25:35 2013
    LMHB started with pid=18, OS id=23013
    Fri Jun 07 12:25:35 2013
    MMAN started with pid=19, OS id=23015
    Fri Jun 07 12:25:35 2013
    DBW0 started with pid=20, OS id=23017
    Fri Jun 07 12:25:35 2013
    DBW1 started with pid=21, OS id=23019
    Fri Jun 07 12:25:35 2013
    DBW2 started with pid=22, OS id=23022
    Fri Jun 07 12:25:35 2013
    DBW3 started with pid=23, OS id=23024
    Fri Jun 07 12:25:35 2013
    DBW4 started with pid=24, OS id=23026
    Fri Jun 07 12:25:35 2013
    DBW5 started with pid=25, OS id=23028
    Fri Jun 07 12:25:35 2013
    DBW6 started with pid=26, OS id=23031
    Fri Jun 07 12:25:35 2013
    DBW7 started with pid=27, OS id=23033
    Fri Jun 07 12:25:35 2013
    LGWR started with pid=28, OS id=23035
    Fri Jun 07 12:25:35 2013
    CKPT started with pid=29, OS id=23037
    Fri Jun 07 12:25:35 2013
    SMON started with pid=30, OS id=23039
    Fri Jun 07 12:25:35 2013
    RECO started with pid=31, OS id=23041
    Fri Jun 07 12:25:35 2013
    RBAL started with pid=32, OS id=23043
    Fri Jun 07 12:25:35 2013
    ASMB started with pid=33, OS id=23045
    Fri Jun 07 12:25:35 2013
    MMON started with pid=34, OS id=23048
    Fri Jun 07 12:25:35 2013
    MMNL started with pid=35, OS id=23052
    Fri Jun 07 12:25:35 2013
    starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
    NOTE: initiating MARK startup
    starting up 1 shared server(s) ...
    Starting background process MARK
    Fri Jun 07 12:25:35 2013
    MARK started with pid=37, OS id=23056
    NOTE: MARK has subscribed
    lmon registered with NM - instance number 1 (internal mem no 0)
    Reconfiguration started (old inc 0, new inc 119)
    List of instances:
    1 2 (myinst: 1)
    Global Resource Directory frozen
    * allocate domain 0, invalid = TRUE
    Communication channels reestablished
    * domain 0 valid according to instance 2
    * domain 0 valid = 1 according to instance 2
    Master broadcasted resource hash value bitmaps
    Non-local Process blocks cleaned out
    LMS 3: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
    LMS 1: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
    LMS 2: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
    LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
    Set master node info
    Submitted all remote-enqueue requests
    Dwn-cvts replayed, VALBLKs dubious
    All grantable enqueues granted
    Submitted all GCS remote-cache requests
    Fix write in gcs resources
    Reconfiguration started (old inc 119, new inc 121)
    List of instances:
    1 2 (myinst: 1)
    Nested reconfiguration detected.
    Global Resource Directory frozen
    Communication channels reestablished
    Master broadcasted resource hash value bitmaps
    Non-local Process blocks cleaned out
    LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
    LMS 3: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
    LMS 2: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
    LMS 1: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
    Set master node info
    Submitted all remote-enqueue requests
    Dwn-cvts replayed, VALBLKs dubious
    All grantable enqueues granted
    Fri Jun 07 12:25:45 2013
    Submitted all GCS remote-cache requests
    Fri Jun 07 12:26:08 2013
    Fix write in gcs resources
    Reconfiguration complete
    Fri Jun 07 12:26:10 2013
    LCK0 started with pid=40, OS id=23632
    Fri Jun 07 12:26:10 2013
    Starting background process RSMN
    Fri Jun 07 12:26:10 2013
    RSMN started with pid=41, OS id=23646
    ORACLE_BASE not set in environment. It is recommended
    that ORACLE_BASE be set in the environment
    Reusing ORACLE_BASE from an earlier startup = /u01/app/oracle
    Fri Jun 07 12:26:11 2013
    ALTER SYSTEM SET local_listener=' (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=135.33.2.13)(PORT=1521))))' SCOPE=MEMORY SID='odsdb1';
    ALTER DATABASE MOUNT /* db agent *//* {1:9971:2} */
    Fri Jun 07 12:26:11 2013
    NOTE: Loaded library: System
    Fri Jun 07 12:26:11 2013
    SUCCESS: diskgroup DATA was mounted
    Fri Jun 07 12:26:11 2013
    NOTE: dependency between database odsdb and diskgroup resource ora.DATA.dg is established
    Fri Jun 07 12:26:16 2013
    Successful mount of redo thread 1, with mount id 3452000551
    Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)
    Lost write protection disabled
    Completed: ALTER DATABASE MOUNT /* db agent *//* {1:9971:2} */
    ALTER DATABASE OPEN /* db agent *//* {1:9971:2} */
    Picked broadcast on commit scheme to generate SCNs
    Thread 1 advanced to log sequence 58364 (thread open)
    Thread 1 opened at log sequence 58364
    Current log# 2 seq# 58364 mem# 0: +DATA/odsdb/onlinelog/group_2.265.812288839
    Current log# 2 seq# 58364 mem# 1: +DATA/odsdb/onlinelog/group_2.266.812288839
    Successful open of redo thread 1
    MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
    Fri Jun 07 12:26:21 2013
    SMON: enabling cache recovery
    Fri Jun 07 12:26:23 2013
    minact-scn: Inst 1 is a slave inc#:121 mmon proc-id:23048 status:0x2
    minact-scn status: grec-scn:0x0000.00000000 gmin-scn:0x0000.00000000 gcalc-scn:0x0000.00000000
    Fri Jun 07 12:26:34 2013
    [23651] Successfully onlined Undo Tablespace 2.
    Undo initialization finished serial:0 start:2061372614 end:2061384964 diff:12350 (123 seconds)
    Verifying file header compatibility for 11g tablespace encryption..
    Verifying 11g file header compatibility for tablespace encryption completed
    Fri Jun 07 12:26:34 2013
    SMON: enabling tx recovery
    Database Characterset is ZHS16GBK
    No Resource Manager plan active
    Starting background process GTX0
    Fri Jun 07 12:26:35 2013
    GTX0 started with pid=45, OS id=23931
    Starting background process RCBG
    Fri Jun 07 12:26:35 2013
    RCBG started with pid=46, OS id=23933
    replication_dependency_tracking turned off (no async multimaster replication found)
    Starting background process QMNC
    Fri Jun 07 12:26:35 2013
    QMNC started with pid=48, OS id=23940
    Completed: ALTER DATABASE OPEN /* db agent *//* {1:9971:2} */
    Fri Jun 07 12:26:38 2013
    Starting background process CJQ0
    Fri Jun 07 12:26:38 2013
    CJQ0 started with pid=55, OS id=23977
    Fri Jun 07 12:27:56 2013
    Thread 1 advanced to log sequence 58365 (LGWR switch)
    Current log# 1 seq# 58365 mem# 0: +DATA/odsdb/onlinelog/group_1.263.812288839
    Current log# 1 seq# 58365 mem# 1: +DATA/odsdb/onlinelog/group_1.264.812288839
    Fri Jun 07 12:28:18 2013
    Starting background process SMCO
    Fri Jun 07 12:28:18 2013
    SMCO started with pid=70, OS id=25166
    Fri Jun 07 12:29:01 2013
    Thread 1 cannot allocate new log, sequence 58366
    Trace file /u01/app/oracle/diag/rdbms/odsdb/odsdb1/trace/odsdb1_asmb_32641.trc
    Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
    With the Partitioning, Real Application Clusters, OLAP, Data Mining
    and Real Application Testing options
    ORACLE_HOME = /u01/app/oracle/product/11.2.0/dbhome_2
    System name: Linux
    Node name: odsdb1
    Release: 2.6.18-308.el5
    Version: #1 SMP Fri Jan 27 17:17:51 EST 2012
    Machine: x86_64
    Instance name: odsdb1
    Redo thread mounted by this instance: 0 <none>
    Oracle process number: 33
    Unix process pid: 32641, image: oracle@odsdb1 (ASMB)
    *** 2013-05-14 15:37:08.705
    *** SESSION ID:(3499.1) 2013-05-14 15:37:08.705
    *** CLIENT ID:() 2013-05-14 15:37:08.705
    *** SERVICE NAME:() 2013-05-14 15:37:08.705
    *** MODULE NAME:() 2013-05-14 15:37:08.705
    *** ACTION NAME:() 2013-05-14 15:37:08.705
    NOTE: initiating MARK startup
    *** 2013-05-14 15:37:16.835
    instance health monitoring reports instance shutting down
    *** 2013-06-07 12:23:42.700
    NOTE: ASMB terminating
    ORA-15064: ? ASM ??????
    ORA-03113: ?????????
    ?? ID:
    ?? ID: 2047 ???: 5
    error 15064 detected in background process
    ORA-15064: ? ASM ??????
    ORA-03113: ?????????
    ?? ID:
    ?? ID: 2047 ???: 5
    kjzduptcctx: Notifying DIAG for crash event
    ----- Abridged Call Stack Trace -----
    ksedsts()+461<-kjzdssdmp()+267<-kjzduptcctx()+232<-kjzdicrshnfy()+53<-ksuitm()+1332<-ksbrdp()+3344<-opirip()+623<-opidrv()+603<-sou2o()+103<-opimai_real()+266<-ssthrdmain()+252<-main()+201<-__libc_start_main()+244<-_start()+36
    ----- End of Abridged Call Stack Trace -----
    *** 2013-06-07 12:23:42.783
    ASMB (ospid: 32641): terminating the instance due to error 15064
    /u01/app/grid/diag/asm/+asm/+ASM1/trace/alert_+ASM1.log
    NOTE: ASMB process exiting, either shutdown is in progress
    NOTE: or foreground connected to ASMB was killed.
    Fri Jun 07 12:23:42 2013
    NOTE: client exited [14808]
    Fri Jun 07 12:23:44 2013
    Received an instance abort message from instance 2
    Please check instance 2 alert and LMON trace files for detail.
    Fri Jun 07 12:23:44 2013
    Received an instance abort message from instance 2
    Please check instance 2 alert and LMON trace files for detail.
    LMD0 (ospid: 31201): terminating the instance due to error 481
    Instance terminated by LMD0, pid = 31201
    Fri Jun 07 12:24:30 2013
    * instance_number obtained from CSS = 1, checking for the existence of node 0...
    * node 0 does not exist. instance_number = 1
    Starting ORACLE instance (normal)
    LICENSE_MAX_SESSION = 0
    LICENSE_SESSIONS_WARNING = 0
    Private Interface 'eth1:1' configured from GPnP for use as a private interconnect.
    [name='eth1:1', type=1, ip=169.254.37.103, mac=00-26-55-eb-61-89, net=169.254.0.0/16, mask=255.255.0.0, use=haip:cluster_interconnect/62]
    Public Interface 'eth0' configured from GPnP for use as a public interface.
    [name='eth0', type=1, ip=135.33.2.8, mac=00-26-55-eb-61-88, net=135.33.2.0/27, mask=255.255.255.224, use=public/1]
    Picked latch-free SCN scheme 3
    Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/11.2.0.2/grid/dbs/arch
    Autotune of undo retention is turned on.
    LICENSE_MAX_USERS = 0
    [grid@odsdb1 cssd]$ file core.30481
    core.30481: ELF 64-bit LSB core file AMD x86-64, version 1 (SYSV), SVR4-style, from 'ocssd.bin'
    [grid@odsdb1 cssd]$ gdb
    gdb gdbserver gdbtui
    [grid@odsdb1 cssd]$ gdb ocssd.bin core.30481
    GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-42.el5)
    Copyright (C) 2009 Free Software Foundation, Inc.
    License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
    This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law. Type "show copying"
    and "show warranty" for details.
    This GDB was configured as "x86_64-redhat-linux-gnu".
    For bug reporting instructions, please see:
    <http://www.gnu.org/software/gdb/bugs/>...
    Reading symbols from /u01/app/11.2.0.2/grid/bin/ocssd.bin...(no debugging symbols found)...done.
    [New Thread 30486]
    [New Thread 30530]
    [New Thread 30526]
    [New Thread 30525]
    [New Thread 30523]
    [New Thread 30522]
    [New Thread 30521]
    [New Thread 30520]
    [New Thread 30519]
    [New Thread 30504]
    [New Thread 30503]
    [New Thread 30495]
    [New Thread 30485]
    [New Thread 30484]
    [New Thread 30483]
    [New Thread 30481]
    Reading symbols from /u01/app/11.2.0.2/grid/lib/libhasgen11.so...(no debugging symbols found)...done.
    Loaded symbols for /u01/app/11.2.0.2/grid/lib/libhasgen11.so
    Reading symbols from /u01/app/11.2.0.2/grid/lib/libocr11.so...(no debugging symbols found)...done.
    Loaded symbols for /u01/app/11.2.0.2/grid/lib/libocr11.so
    Reading symbols from /u01/app/11.2.0.2/grid/lib/libocrb11.so...(no debugging symbols found)...done.
    Loaded symbols for /u01/app/11.2.0.2/grid/lib/libocrb11.so
    Reading symbols from /u01/app/11.2.0.2/grid/lib/libocrutl11.so...(no debugging symbols found)...done.
    Loaded symbols for /u01/app/11.2.0.2/grid/lib/libocrutl11.so
    Reading symbols from /u01/app/11.2.0.2/grid/lib/libclntsh.so.11.1...(no debugging symbols found)...done.
    Loaded symbols for /u01/app/11.2.0.2/grid/lib/libclntsh.so.11.1
    Reading symbols from /u01/app/11.2.0.2/grid/lib/libskgxn2.so...(no debugging symbols found)...done.
    Loaded symbols for /u01/app/11.2.0.2/grid/lib/libskgxn2.so
    Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
    Loaded symbols for /lib64/libdl.so.2
    Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
    Loaded symbols for /lib64/libm.so.6
    Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
    [Thread debugging using libthread_db enabled]
    Loaded symbols for /lib64/libpthread.so.0
    Reading symbols from /lib64/libnsl.so.1...(no debugging symbols found)...done.
    Loaded symbols for /lib64/libnsl.so.1
    Reading symbols from /u01/app/11.2.0.2/grid/lib/libasmclntsh11.so...(no debugging symbols found)...done.
    Loaded symbols for /u01/app/11.2.0.2/grid/lib/libasmclntsh11.so
    Reading symbols from /u01/app/11.2.0.2/grid/lib/libcell11.so...(no debugging symbols found)...done.
    Loaded symbols for /u01/app/11.2.0.2/grid/lib/libcell11.so
    Reading symbols from /u01/app/11.2.0.2/grid/lib/libskgxp11.so...(no debugging symbols found)...done.
    Loaded symbols for /u01/app/11.2.0.2/grid/lib/libskgxp11.so
    Reading symbols from /u01/app/11.2.0.2/grid/lib/libnnz11.so...(no debugging symbols found)...done.
    Loaded symbols for /u01/app/11.2.0.2/grid/lib/libnnz11.so
    Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
    Loaded symbols for /lib64/libc.so.6
    Reading symbols from /usr/lib64/libaio.so.1...(no debugging symbols found)...done.
    Loaded symbols for /usr/lib64/libaio.so.1
    Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
    Loaded symbols for /lib64/ld-linux-x86-64.so.2
    Reading symbols from /u01/app/11.2.0.2/grid/lib/libnque11.so...(no debugging symbols found)...done.
    Loaded symbols for /u01/app/11.2.0.2/grid/lib/libnque11.so
    Reading symbols from /opt/oracle/extapi/64/asm/orcl/1/libasm.so...(no debugging symbols found)...done.
    Loaded symbols for /opt/oracle/extapi/64/asm/orcl/1/libasm.so
    warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff505fd000
    Core was generated by `/u01/app/11.2.0.2/grid/bin/ocssd.bin '.
    Program terminated with signal 6, Aborted.
    #0 0x000000369ea30265 in raise () from /lib64/libc.so.6
    (gdb) where
    #0 0x000000369ea30265 in raise () from /lib64/libc.so.6
    #1 0x000000369ea31d10 in abort () from /lib64/libc.so.6
    #2 0x00002afc67f9aeda in scls_abort (flags=0) at scls.c:7088
    #3 0x000000000040babd in clssscExit (thrd=0x10d325a0, status=clssscreasonSHUTNORM) at clsssc.c:2155
    #4 0x0000000000446221 in clssgmClientShutdown (thrd=0x10d325a0, cmInfo=0x10b40090) at clssgmc.c:6415
    #5 0x0000000000436707 in clssgmProcClientReqs (thrd=0x10d325a0, clctx=0x10b40630) at clssgmc.c:704
    #6 0x0000000000436405 in clssgmclientlsnr (thrd=0x10d325a0) at clssgmc.c:644
    #7 0x000000000040ac2f in clssscthrdmain (thrd=0x10d325a0) at clsssc.c:1716
    #8 0x000000369fa0677d in start_thread () from /lib64/libpthread.so.0
    #9 0x000000369ead49ad in clone () from /lib64/libc.so.6
    (gdb)
    2013-06-07 12:19:37.377: [    CSSD][1085888832]clssscSelect: cookie accept request 0x10b40630
    2013-06-07 12:19:37.377: [    CSSD][1085888832]clssgmAllocProc: (0x2aaab0133ea0) allocated
    2013-06-07 12:19:37.379: [    CSSD][1085888832]clssgmClientConnectMsg: properties of cmProc 0x2aaab0133ea0 - 1,2,3,4,5
    2013-06-07 12:19:37.379: [    CSSD][1085888832]clssgmClientConnectMsg: Connect from con(0x6ae44fa) proc(0x2aaab0133ea0) pid(14139/14139) version 11:2:1:4, properties: 1,2,3,4,5
    2013-06-07 12:19:37.379: [    CSSD][1085888832]clssgmClientConnectMsg: msg flags 0x0000
    2013-06-07 12:19:37.384: [    CSSD][1085888832]clssscSelect: cookie accept request 0x2aaab0133ea0
    2013-06-07 12:19:37.384: [    CSSD][1085888832]clssscevtypSHRCON: getting client with cmproc 0x2aaab0133ea0
    2013-06-07 12:19:37.384: [    CSSD][1085888832]clssgmRegisterClient: proc(69/0x2aaab0133ea0), client(1/0x2aaab010c5c0)
    2013-06-07 12:19:37.385: [    CSSD][1085888832]clssgmRegisterShared: grp DBODSDB, mbr 0, type 1
    2013-06-07 12:19:37.385: [    CSSD][1085888832]clssgmQueueShare: (0x2aaab0085790) target global grock DBODSDB member 0 type 1 queued from client (0x2aaab010c5c0), global grock DBODSDB, refcount 23
    2013-06-07 12:19:37.385: [    CSSD][1085888832]clssgmRegisterShared: global grock DBODSDB member 0 share type 1, refcount 23
    2013-06-07 12:19:37.391: [    CSSD][1085888832]clssscSelect: cookie accept request 0x2aaab0133ea0
    2013-06-07 12:19:37.391: [    CSSD][1085888832]clssscevtypSHRCON: getting client with cmproc 0x2aaab0133ea0
    2013-06-07 12:19:37.391: [    CSSD][1085888832]clssgmRegisterClient: proc(69/0x2aaab0133ea0), client(2/0x2aaab0061f10)
    what is the problem
    Edited by: 徐振富 on 2013-6-7 下午6:38
    Edited by: 徐振富 on 2013-6-7 下午6:45

    is your ASM instance up?
    If not, trying bring up ASM instance up just by itself and see if it throws any error?
    Post status of crsctl status cluster -all

  • Archive Generation in RAC environment.

    Gems,
    I have one doubt it was not able to solve from my mind i.e.
    As per my knowledge,
    1) Every instance has its own thread so Thread is instance specific. My questions are, How the SCN will be maintained in ARCHIVES? will thread 1 & thread2 contains the same SCN?
    2) If i have primary database with two nodes, and standby as single stand alone, How it will apply the archives on standby? I monitored ALERT log file of standby, Mostly it applies first thread 1 sequence, then thread2 sequence then thread1 sequence and then thread 2 sequence and so on.. Let me know if my assumption is wrong.
    3) If i cloned from RAC to NON-RAC then, If in case if i performed SCN based recovery as
    run
    set until scn 10000;
    restore database;
    recover database;
    then scn *10000* will contain in both the archives or only in one archives? or it contains only in one sequence? Please explain me.
    Appriciated for the right answers, If you are unable to understand let me know, I will try to clear up more clearly.
    Thanks

    Hi,
    first of all there are 2 kinds of SCNs. A "global" SCN and a local SCN per instance. Since 10g the SCNs are advanced with a mechanism called BoC (Broadcast on Commit).
    Each time the LGWR writes to the redo log, LGWR sends a message to update the global SCN in all instances and to update the local SCN in all active instances.
    1.) So each RedoLog entry knows the global SCN. So even though you have multiple instances, the SCN is unique (and in order).
    2.) That depends on how the redlogs were written. If e.g. one instance did not write redologs for a while, it could be that 2 logs from instance 1 are needed before the log from instance 2 is needed. But in gerenall it will be "parts of redolog 1 then parts of redolog 2" etc.
    3.) Yes you can, since the (global) SCNs are unique and will be used.
    Regards
    Sebastian

  • [代发帖子]10.2.0.4.0版本rac 报IPC Send timeout detected.Sender 错误问题

    以下为代网友发帖，内容为
    环境：db：10.2.0.4.0 system：aix 6.1
    请各位大师帮忙确定是什么原因导致的
    node1
    Wed Aug 22 09:59:17 2012
    IPC Send timeout detected.Sender: ospid 25821502
    Receiver: inst 2 binc 1412232506 ospid 27066946
    Wed Aug 22 09:59:17 2012
    IPC Send timeout detected.Sender: ospid 24838548
    Receiver: inst 2 binc 1412232506 ospid 27066946
    Wed Aug 22 09:59:18 2012
    IPC Send timeout detected.Sender: ospid 18743888
    Receiver: inst 2 binc 1412232506 ospid 27066946
    Wed Aug 22 09:59:18 2012
    IPC Send timeout detected.Sender: ospid 8586514
    Receiver: inst 2 binc 1412232506 ospid 27066946
    Wed Aug 22 09:59:19 2012
    IPC Send timeout detected.Sender: ospid 21365006
    Receiver: inst 2 binc 1412232506 ospid 27066946
    Wed Aug 22 09:59:20 2012
    IPC Send timeout detected.Sender: ospid 18088896
    Receiver: inst 2 binc 1412232506 ospid 27066946
    Wed Aug 22 09:59:20 2012
    IPC Send timeout detected.Sender: ospid 15467098
    Receiver: inst 2 binc 1412232506 ospid 27066946
    Wed Aug 22 09:59:20 2012
    Wed Aug 22 09:59:44 2012
    IPC Send timeout detected.Sender: ospid 9635304
    Receiver: inst 2 binc 1412232544 ospid 25362610
    Wed Aug 22 09:59:44 2012
    IPC Send timeout detected.Sender: ospid 1704160
    Receiver: inst 2 binc 1412232544 ospid 25362610
    Wed Aug 22 09:59:44 2012
    IPC Send timeout detected.Sender: ospid 2687916
    Receiver: inst 2 binc 1412232544 ospid 25362610
    Wed Aug 22 09:59:44 2012
    IPC Send timeout detected.Sender: ospid 1836306
    Receiver: inst 2 binc 1412232506 ospid 27066946
    Wed Aug 22 09:59:45 2012
    IPC Send timeout detected.Sender: ospid 11273456
    Wed Aug 22 10:00:03 2012
    IPC Send timeout detected.Sender: ospid 4982212
    Receiver: inst 2 binc 1412232544 ospid 25362610
    Wed Aug 22 10:00:03 2012
    IPC Send timeout to 1.1 inc 28 for msg type 32 from opid 629
    Wed Aug 22 10:00:03 2012
    IPC Send timeout detected.Sender: ospid 15270298
    Receiver: inst 2 binc 1412232506 ospid 27066946
    Wed Aug 22 10:00:03 2012
    IPC Send timeout to 1.5 inc 28 for msg type 36 from opid 374
    Wed Aug 22 10:00:03 2012
    IPC Send timeout to 1.5 inc 28 for msg type 32 from opid 619
    Wed Aug 22 10:00:03 2012
    IPC Send timeout detected.Sender: ospid 16712906
    Receiver: inst 2 binc 1412232506 ospid 27066946
    Wed Aug 22 10:00:03 2012
    IPC Send timeout to 1.1 inc 28 for msg type 32 from opid 626
    Wed Aug 22 10:00:03 2012
    IPC Send timeout to 1.5 inc 28 for msg type 32 from opid 376
    Wed Aug 22 10:00:03 2012
    IPC Send timeout to 1.1 inc 28 for msg type 32 from opid 260
    Wed Aug 22 10:00:03 2012
    IPC Send timeout detected.Sender: ospid 12125594
    Receiver: inst 2 binc 1412232506 ospid 27066946
    Wed Aug 22 10:00:04 2012
    IPC Send timeout detected.Sender: ospid 22086174
    Receiver: inst 2 binc 1412232506 ospid 27066946
    Wed Aug 22 10:00:04 2012
    IPC Send timeout detected.Sender: ospid 8979076
    Receiver: inst 2 binc 1412232544 ospid 25362610
    Wed Aug 22 10:00:04 2012
    IPC Send timeout detected.Sender: ospid 23004052
    Receiver: inst 2 binc 1412232506 ospid 27066946
    Wed Aug 22 10:00:04 2012
    IPC Send timeout to 1.5 inc 28 for msg type 32 from opid 640
    Wed Aug 22 10:00:04 2012
    IPC Send timeout to 1.5 inc 28 for msg type 32 from opid 114
    Wed Aug 22 10:00:04 2012
    IPC Send timeout to 1.5 inc 28 for msg type 36 from opid 642
    Wed Aug 22 10:00:04 2012
    IPC Send timeout to 1.1 inc 28 for msg type 32 from opid 644
    Wed Aug 22 10:00:04 2012
    IPC Send timeout to 1.5 inc 28 for msg type 32 from opid 228
    Wed Aug 22 10:00:04 2012
    IPC Send timeout detected.Sender: ospid 13436000
    Receiver: inst 2 binc 1412232506 ospid 27066946
    Wed Aug 22 10:00:05 2012
    IPC Send timeout detected.Sender: ospid 4850956
    Receiver: inst 2 binc 1412232544 ospid 25362610
    Wed Aug 22 10:00:05 2012
    IPC Send timeout detected.Sender: ospid 21627618
    Receiver: inst 2 binc 1412232544 ospid 25362610
    Wed Aug 22 10:00:05 2012
    IPC Send timeout to 1.1 inc 28 for msg type 36 from opid 637
    Wed Aug 22 10:00:05 2012
    IPC Send timeout to 1.5 inc 28 for msg type 32 from opid 648
    Wed Aug 22 10:00:05 2012
    IPC Send timeout to 1.1 inc 28 for msg type 36 from opid 628
    Wed Aug 22 10:00:05 2012
    IPC Send timeout to 1.1 inc 28 for msg type 32 from opid 647
    Wed Aug 22 10:00:05 2012
    IPC Send timeout detected.Sender: ospid 23724892
    Receiver: inst 2 binc 1412232506 ospid 27066946
    Wed Aug 22 10:00:05 2012
    IPC Send timeout detected.Sender: ospid 14353416
    Receiver: inst 2 binc 1412232506 ospid 27066946
    Wed Aug 22 10:00:05 2012
    IPC Send timeout detected.Sender: ospid 15270628
    Receiver: inst 2 binc 1412232506 ospid 27066946
    Wed Aug 22 10:00:06 2012
    IPC Send timeout to 1.1 inc 28 for msg type 32 from opid 634
    Wed Aug 22 10:00:06 2012
    IPC Send timeout to 1.1 inc 28 for msg type 32 from opid 235
    Wed Aug 22 10:00:06 2012
    IPC Send timeout detected.Sender: ospid 10749150
    Receiver: inst 2 binc 1412232506 ospid 27066946
    Wed Aug 22 10:00:06 2012
    IPC Send timeout to 1.1 inc 28 for msg type 32 from opid 72
    Wed Aug 22 10:00:06 2012
    IPC Send timeout to 1.1 inc 28 for msg type 32 from opid 654
    Wed Aug 22 10:00:06 2012
    IPC Send timeout detected.Sender: ospid 16123646
    Receiver: inst 2 binc 1412232544 ospid 25362610
    Wed Aug 22 10:00:06 2012
    IPC Send timeout detected.Sender: ospid 19005890
    Receiver: inst 2 binc 1412232506 ospid 27066946
    Wed Aug 22 10:00:06 2012
    IPC Send timeout to 1.1 inc 28 for msg type 32 from opid 655
    Wed Aug 22 10:00:06 2012
    IPC Send timeout detected.Sender: ospid 1311244
    Receiver: inst 2 binc 1412232506 ospid 27066946
    Wed Aug 22 10:00:06 2012
    IPC Send timeout to 1.5 inc 28 for msg type 36 from opid 658
    Wed Aug 22 10:00:07 2012
    IPC Send timeout detected.Sender: ospid 14091918
    Receiver: inst 2 binc 1412232506 ospid 27066946
    Wed Aug 22 10:00:07 2012
    IPC Send timeout to 1.5 inc 28 for msg type 32 from opid 424
    Wed Aug 22 10:00:07 2012
    IPC Send timeout to 1.5 inc 28 for msg type 32 from opid 631
    Wed Aug 22 10:00:07 2012
    IPC Send timeout detected.Sender: ospid 14876874
    Receiver: inst 2 binc 1412232506 ospid 27066946
    Wed Aug 22 10:00:07 2012
    IPC Send timeout to 1.1 inc 28 for msg type 32 from opid 661
    Wed Aug 22 10:00:07 2012
    IPC Send timeout detected.Sender: ospid 18613208
    Receiver: inst 2 binc 1412232506 ospid 27066946
    Wed Aug 22 10:00:07 2012
    IPC Send timeout to 1.5 inc 28 for msg type 32 from opid 664
    Wed Aug 22 10:00:07 2012
    IPC Send timeout to 1.1 inc 28 for msg type 32 from opid 603
    Wed Aug 22 10:00:07 2012
    IPC Send timeout detected.Sender: ospid 2818720
    Receiver: inst 2 binc 1412232544 ospid 25362610
    Wed Aug 22 10:03:54 2012
    IPC Send timeout to 1.1 inc 28 for msg type 36 from opid 1219
    Wed Aug 22 10:04:05 2012
    IPC Send timeout to 1.5 inc 28 for msg type 32 from opid 2256
    Wed Aug 22 10:04:08 2012
    IPC Send timeout to 1.1 inc 28 for msg type 36 from opid 1908
    Wed Aug 22 10:04:10 2012
    IPC Send timeout to 1.1 inc 28 for msg type 32 from opid 1476
    Wed Aug 22 10:04:12 2012
    IPC Send timeout to 1.5 inc 28 for msg type 32 from opid 1653
    Wed Aug 22 10:04:12 2012
    IPC Send timeout to 1.5 inc 28 for msg type 32 from opid 1266
    Wed Aug 22 10:04:13 2012
    IPC Send timeout to 1.1 inc 28 for msg type 32 from opid 1345
    Wed Aug 22 10:04:13 2012
    IPC Send timeout to 1.1 inc 28 for msg type 32 from opid 1202
    Wed Aug 22 10:04:13 2012
    IPC Send timeout to 1.1 inc 28 for msg type 32 from opid 1203
    Wed Aug 22 10:04:13 2012
    IPC Send timeout to 1.1 inc 28 for msg type 32 from opid 1239
    Wed Aug 22 10:04:13 2012
    IPC Send timeout to 1.1 inc 28 for msg type 32 from opid 1237
    Wed Aug 22 10:04:13 2012
    IPC Send timeout to 1.5 inc 28 for msg type 32 from opid 1207
    Wed Aug 22 10:04:13 2012
    IPC Send timeout to 1.5 inc 28 for msg type 32 from opid 1197
    Wed Aug 22 10:04:13 2012
    IPC Send timeout to 1.5 inc 28 for msg type 32 from opid 1313
    Wed Aug 22 10:04:13 2012
    IPC Send timeout to 1.5 inc 28 for msg type 32 from opid 1274
    Wed Aug 22 10:04:13 2012
    IPC Send timeout to 1.1 inc 28 for msg type 32 from opid 1140
    Wed Aug 22 10:04:13 2012
    IPC Send timeout to 1.1 inc Wed Aug 22 10:25:55 2012Node2:
    Wed Aug 22 09:47:57 2012
    Error 2056 trapped in 2PC on transaction 20.16.21521114. Cleaning up.
    Error stack returned to user:
    ORA-02056: 2PC: k2lpre: bad two-phase command number 0 from coord
    Wed Aug 22 09:48:28 2012
    Error 2056 trapped in 2PC on transaction 78.44.4484418. Cleaning up.
    Error stack returned to user:
    ORA-02056: 2PC: k2lpre: bad two-phase command number 0 from coord
    Error 2056 trapped in 2PC on transaction 17.42.11081684. Cleaning up.
    Error stack returned to user:
    ORA-02056: 2PC: k2lpre: bad two-phase command number 0 from coord
    Wed Aug 22 09:48:42 2012
    Error 2056 trapped in 2PC on transaction 75.15.10618335. Cleaning up.
    Error stack returned to user:
    ORA-02056: 2PC: k2lpre: bad two-phase command number 0 from coord
    Wed Aug 22 09:48:58 2012
    Error 2056 trapped in 2PC on transaction 62.28.20411900. Cleaning up.
    Error stack returned to user:
    ORA-02056: 2PC: k2lpre: bad two-phase command number 0 from coord
    Wed Aug 22 09:49:04 2012
    Error 2056 trapped in 2PC on transaction 81.34.7287924. Cleaning up.
    Error stack returned to user:
    ORA-02056: 2PC: k2lpre: bad two-phase command number 0 from coord
    Wed Aug 22 09:50:01 2012
    Thread 2 advanced to log sequence 53611 (LGWR switch)
      Current log# 5 seq# 53611 mem# 0: /dev/rredo05_rzdb
    Wed Aug 22 09:50:34 2012
    Error 2056 trapped in 2PC on transaction 71.15.14657575. Cleaning up.
    Error stack returned to user:
    ORA-02056: 2PC: k2lpre: bad two-phase command number 0 from coord
    Wed Aug 22 09:50:46 2012
    Error 2056 trapped in 2PC on transaction 84.12.11181366. Cleaning up.
    Error stack returned to user:
    ORA-02056: 2PC: k2lpre: bad two-phase command number 0 from coord
    Wed Aug 22 09:50:48 2012
    Errors in file /oracle/opt/product/10.2.0/db/admin/crmrz/udump/crmrz2_ora_23856588.trc:
    ORA-00600: internal error code, arguments: [ksqcmi:res2], [], [], [], [], [], [], []
    Wed Aug 22 09:50:49 2012
    Trace dumping is performing id=[cdmp_20120822095049]
    Wed Aug 22 09:59:16 2012
    IPC Send timeout detected. Receiver ospid 27066946
    Receiver is waiting for a latch dumping latch state for receiver -17356
    Wed Aug 22 09:59:16 2012
    Errors in file /oracle/opt/product/10.2.0/db/admin/crmrz/bdump/crmrz2_lms4_25362610.trc:
    Wed Aug 22 09:59:38 2012
    Errors in file /oracle/opt/product/10.2.0/db/admin/crmrz/bdump/crmrz2_lms0_27066946.trc:
    Wed Aug 22 09:59:38 2012
    Trace dumping is performing id=[cdmp_20120822095938]
    Wed Aug 22 10:02:09 2012
    kkjcre1p: unable to spawn jobq slave process
    Wed Aug 22 10:02:09 2012
    Errors in file /oracle/opt/product/10.2.0/db/admin/crmrz/bdump/crmrz2_cjq0_25100728.trc:
    Wed Aug 22 10:02:32 2012
    Errors in file /oracle/opt/product/10.2.0/db/admin/crmrz/bdump/crmrz2_lmon_25493740.trc:
    ORA-29740: evicted by member 0, group incarnation 30
    Wed Aug 22 10:02:32 2012
    LMON: terminating instance due to error 29740
    Wed Aug 22 10:02:32 2012
    Errors in file /oracle/opt/product/10.2.0/db/admin/crmrz/bdump/crmrz2_lms2_24904066.trc:
    ORA-29740: evicted by member , group incarnation
    Wed Aug 22 10:02:32 2012
    Errors in file /oracle/opt/product/10.2.0/db/admin/crmrz/bdump/crmrz2_lms6_25626088.trc:
    ORA-29740: evicted by member , group incarnation
    Wed Aug 22 10:02:32 2012
    Errors in file /oracle/opt/product/10.2.0/db/admin/crmrz/bdump/crmrz2_lms3_25559280.trc:
    ORA-29740: evicted by member , group incarnation
    Wed Aug 22 10:02:32 2012
    Errors in file /oracle/opt/product/10.2.0/db/admin/crmrz/bdump/crmrz2_lms5_22873874.trc:
    ORA-29740: evicted by member , group incarnation
    Wed Aug 22 10:02:32 2012
    Errors in file /oracle/opt/product/10.2.0/db/admin/crmrz/bdump/crmrz2_lms1_24905100.trc:
    ORA-29740: evicted by member , group incarnation
    Wed Aug 22 10:02:32 2012
    Errors in file /oracle/opt/product/10.2.0/db/admin/crmrz/bdump/crmrz2_arc1_26543176.trc:
    ORA-29740: evicted by member , group incarnation
    Wed Aug 22 10:02:32 2012
    Errors in file /oracle/opt/product/10.2.0/db/admin/crmrz/bdump/crmrz2_arc1_26543176.trc:
    ORA-29740: evicted by member , group incarnation
    Wed Aug 22 10:02:32 2012
    Errors in file /oracle/opt/product/10.2.0/db/admin/crmrz/bdump/crmrz2_arc1_26543176.trc:
    ORA-29740: evicted by member , group incarnation
    Wed Aug 22 10:02:32 2012
    Master background archival failure: 29740
    Wed Aug 22 10:02:32 2012
    Errors in file /oracle/opt/product/10.2.0/db/admin/crmrz/bdump/crmrz2_lgwr_26477376.trc:
    ORA-29740: evicted by member , group incarnation
    Wed Aug 22 10:02:32 2012
    Errors in file /oracle/opt/product/10.2.0/db/admin/crmrz/bdump/crmrz2_lck0_22677374.trc:
    ORA-29740: evicted by member , group incarnation
    Wed Aug 22 10:02:32 2012
    Errors in file /oracle/opt/product/10.2.0/db/admin/crmrz/bdump/crmrz2_lmd0_27132530.trc:
    ORA-29740: evicted by member , group incarnation
    Wed Aug 22 10:02:32 2012
    Errors in file /oracle/opt/product/10.2.0/db/admin/crmrz/bdump/crmrz2_pmon_25494950.trc:
    ORA-29740: evicted by member , group incarnation
    Wed Aug 22 10:02:33 2012
    Errors in file /oracle/opt/product/10.2.0/db/admin/crmrz/bdump/crmrz2_mman_21235492.trc:
    ORA-29740: evicted by member , group incarnation
    Wed Aug 22 10:02:33 2012
    Errors in file /oracle/opt/product/10.2.0/db/admin/crmrz/bdump/crmrz2_arc0_25036272.trc:
    ORA-29740: evicted by member , group incarnation
    Wed Aug 22 10:02:33 2012
    Errors in file /oracle/opt/product/10.2.0/db/admin/crmrz/bdump/crmrz2_arc0_25036272.trc:
    ORA-29740: evicted by member , group incarnation
    Wed Aug 22 10:02:33 2012
    Errors in file /oracle/opt/product/10.2.0/db/admin/crmrz/bdump/crmrz2_arc0_25036272.trc:
    ORA-29740: evicted by member , group incarnation
    Wed Aug 22 10:02:33 2012
    Master background archival failure: 29740
    Wed Aug 22 10:02:33 2012
    Errors in file /oracle/opt/product/10.2.0/db/admin/crmrz/bdump/crmrz2_smon_22743004.trc:
    ORA-29740: evicted by member , group incarnation
    Wed Aug 22 10:02:34 2012
    Shutting down instance (abort)
    License high water mark = 1722
    Wed Aug 22 10:02:38 2012
    Instance terminated by LMON, pid = 25493740
    Wed Aug 22 10:02:39 2012
    Instance terminated by USER, pid = 22873880
    Wed Aug 22 10:03:09 2012
    Starting ORACLE instance (normal)
    sskgpgetexecname failed to get name
    LICENSE_MAX_SESSION = 0
    LICENSE_SESSIONS_WARNING = 0
    Interface type 1 en10 192.100.100.0 configured from OCR for use as a cluster interconnect
    Interface type 1 en9 134.224.106.0 configured from OCR for use as  a public interface
    Picked latch-free SCN scheme 3
    Autotune of undo retention is turned on.
    LICENSE_MAX_USERS = 0
    SYS auditing is disabled
    ksdpec: called for event 13740 prior to event group initialization
    Starting up ORACLE RDBMS Version: 10.2.0.4.0.
    System parameters with non-default values:
      processes                = 3000
      sessions                 = 3305
      resource_limit           = TRUE
      event                    =
      sga_max_size             = 51539607552
      __shared_pool_size       = 4227858432
      __large_pool_size        = 16777216
      __java_pool_size         = 33554432
      __streams_pool_size      = 0
      spfile                   = /dev/rspfile_rzdb
      sga_target               = 51539607552
      control_files            = /dev/rcntrl01_rzdb, /dev/rcntrl02_rzdb, /dev/rcntrl03_rzdb
      db_block_size            = 8192
      __db_cache_size          = 46707769344
      db_keep_cache_size       = 536870912
      compatible               = 10.2.0.3.0
      log_archive_dest_1       = location=/arch02
      db_files                 = 1000
      db_file_multiblock_read_count= 16
      cluster_database         = TRUE
      cluster_database_instances= 2
      thread                   = 2
      instance_number          = 2
      undo_management          = AUTO
      undo_tablespace          = UNDOTBS2
      undo_retention           = 900
      recyclebin               = OFF
      remote_login_passwordfile= EXCLUSIVE
      db_domain                =
      dispatchers              = (PROTOCOL=TCP) (SERVICE=crmrzXDB)
      local_listener           = (ADDRESS = (PROTOCOL =TCP)(HOST = 134.224.106.126)(PORT = 1521))
      remote_listener          = LISTENERS_CRMRZ
      utl_file_dir             = /arch01
      job_queue_processes      = 20
      cursor_sharing           = SIMILAR
      background_dump_dest     = /oracle/opt/product/10.2.0/db/admin/crmrz/bdump
      user_dump_dest           = /oracle/opt/product/10.2.0/db/admin/crmrz/udump
      core_dump_dest           = /oracle/opt/product/10.2.0/db/admin/crmrz/cdump
      audit_file_dest          = /oracle/opt/product/10.2.0/db/admin/crmrz/adump
      db_name                  = crmrz
      open_cursors             = 1500
      pga_aggregate_target     = 10726932480
    Cluster communication is configured to use the following interface(s) for this instance
      192.100.100.2
    Wed Aug 22 10:03:11 2012
    cluster interconnect IPC version:Oracle UDP/IP (generic)
    IPC Vendor 1 proto 2
    PMON started with pid=2, OS id=24379686
    DIAG started with pid=3, OS id=22743022
    PSP0 started with pid=4, OS id=24773700
    LMON started with pid=5, OS id=25429322
    LMD0 started with pid=6, OS id=25494130
    LMS0 started with pid=7, OS id=24904068
    LMS1 started with pid=8, OS id=9437246
    LMS2 started with pid=9, OS id=13304862
    LMS3 started with pid=10, OS id=24969498
    LMS4 started with pid=11, OS id=26936330
    LMS5 started with pid=12, OS id=26935868
    LMS6 started with pid=13, OS id=25167108
    MMAN started with pid=14, OS id=27132604
    DBW0 started with pid=15, OS id=26739208
    DBW1 started with pid=16, OS id=23790482
    DBW2 started with pid=17, OS id=25626104
    DBW3 started with pid=18, OS id=25494956
    LGWR started with pid=19, OS id=26673758
    CKPT started with pid=20, OS id=26542676
    SMON started with pid=21, OS id=20383626
    RECO started with pid=22, OS id=26477394
    CJQ0 started with pid=23, OS id=26411882
    MMON started with pid=24, OS id=26346430
    Wed Aug 22 10:03:14 2012
    starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
    MMNL started with pid=25, OS id=25624582
    Wed Aug 22 10:03:14 2012
    starting up 1 shared server(s) ...
    Wed Aug 22 10:03:19 2012
    lmon registered with NM - instance id 2 (internal mem no 1)
    Wed Aug 22 10:03:26 2012
    Reconfiguration started (old inc 0, new inc 33)
    List of nodes:
    0 1
    Global Resource Directory frozen
    * allocate domain 0, invalid = TRUE
    Communication channels reestablished
    * domain 0 valid = 0 according to instance 0
    Wed Aug 22 10:03:26 2012
    Master broadcasted resource hash value bitmaps
    Non-local Process blocks cleaned out
    Wed Aug 22 10:03:26 2012
    LMS 4: 0 GCS shadows cancelled, 0 closed
    Wed Aug 22 10:03:26 2012
    LMS 6: 0 GCS shadows cancelled, 0 closed
    Wed Aug 22 10:03:26 2012
    LMS 2: 0 GCS shadows cancelled, 0 closed
    Wed Aug 22 10:03:26 2012
    LMS 1: 0 GCS shadows cancelled, 0 closed
    Wed Aug 22 10:03:26 2012
    LMS 5: 0 GCS shadows cancelled, 0 closed
    Wed Aug 22 10:03:26 2012
    LMS 3: 0 GCS shadows cancelled, 0 closed
    Wed Aug 22 10:03:26 2012
    LMS 0: 0 GCS shadows cancelled, 0 closed
    Set master node info
    Submitted all remote-enqueue requests
    Dwn-cvts replayed, VALBLKs dubious
    All grantable enqueues granted
    Wed Aug 22 10:03:28 2012
    LMS 5: 0 GCS shadows traversed, 0 replayed
    Wed Aug 22 10:03:28 2012
    LMS 6: 0 GCS shadows traversed, 0 replayed
    Wed Aug 22 10:03:28 2012
    LMS 4: 0 GCS shadows traversed, 0 replayed
    Wed Aug 22 10:03:28 2012
    LMS 2: 0 GCS shadows traversed, 0 replayed
    Wed Aug 22 10:03:28 2012
    LMS 1: 0 GCS shadows traversed, 0 replayed
    Wed Aug 22 10:03:28 2012
    LMS 3: 0 GCS shadows traversed, 0 replayed
    Wed Aug 22 10:03:28 2012
    LMS 0: 0 GCS shadows traversed, 0 replayed
    Wed Aug 22 10:03:28 2012
    Submitted all GCS remote-cache requests
    Fix write in gcs resources
    Reconfiguration complete
    LCK0 started with pid=28, OS id=26215276
    Wed Aug 22 10:03:33 2012
    ALTER DATABASE   MOUNT
    Wed Aug 22 10:03:37 2012
    Setting recovery target incarnation to 1
    Wed Aug 22 10:03:37 2012
    Successful mount of redo thread 2, with mount id 3477695295
    Wed Aug 22 10:03:37 2012
    Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)
    Completed: ALTER DATABASE   MOUNT
    Wed Aug 22 10:03:38 2012
    ALTER DATABASE OPEN
    Picked broadcast on commit scheme to generate SCNs
    Wed Aug 22 10:03:59 2012
    Trace dumping is performing id=[cdmp_20120822100344]
    IPC Send timeout detected. Receiver ospid 24904068
    Wed Aug 22 10:04:03 2012
    Errors in file /oracle/opt/product/10.2.0/db/admin/crmrz/bdump/crmrz2_lms0_24904068.trc:
    Wed Aug 22 10:04:59 2012
    Trace dumping is performing id=[cdmp_20120822100458]
    Wed Aug 22 10:05:19 2012
    LGWR: STARTING ARCH PROCESSES
    ARC0 started with pid=32, OS id=26739814
    Wed Aug 22 10:05:19 2012
    ARC0: Archival started
    ARC1: Archival started
    LGWR: STARTING ARCH PROCESSES COMPLETE
    ARC1 started with pid=31, OS id=24445226
    Wed Aug 22 10:05:20 2012
    IPC Send timeout detected. Receiver ospid 24904068
    Wed Aug 22 10:05:20 2012
    Errors in file /oracle/opt/product/10.2.0/db/admin/crmrz/bdump/crmrz2_lms0_24904068.trc:
    Wed Aug 22 10:05:21 2012
    Thread 2 opened at log sequence 53612
      Current log# 6 seq# 53612 mem# 0: /dev/rredo06_rzdb
    Wed Aug 22 10:05:21 2012
    ARC1: Becoming the 'no FAL' ARCH
    ARC1: Becoming the 'no SRL' ARCH
    Wed Aug 22 10:05:21 2012
    ARC0: Becoming the heartbeat ARCH
    Wed Aug 22 10:05:21 2012
    Successful open of redo thread 2
    Wed Aug 22 10:05:21 2012
    MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
    Wed Aug 22 10:05:21 2012
    SMON: enabling cache recovery
    Wed Aug 22 10:06:31 2012
    WARNING: inbound connection timed out (ORA-3136)
    Wed Aug 22 10:06:32 2012
    Successfully onlined Undo Tablespace 4.
    Wed Aug 22 10:06:32 2012
    SMON: enabling tx recovery
    Wed Aug 22 10:06:32 2012
    Database Characterset is ZHS16GBK
    Opening with internal Resource Manager plan
    where NUMA PG = 1, CPUs = 28
    replication_dependency_tracking turned off (no async multimaster replication found)
    Starting background process QMNC
    QMNC started with pid=42, OS id=25887794
    Wed Aug 22 10:06:43 2012
    Completed: ALTER DATABASE OPEN
    Wed Aug 22 10:07:00 2012
    WARNING: inbound connection timed out (ORA-3136)
    Wed Aug 22 10:07:13 2012
    WARNING: inbound connection timed out (ORA-3136)
    Wed Aug 22 10:07:25 2012
    WARNING: inbound connection timed out (ORA-3136)
    Wed Aug 22 10:07:31 2012
    Trace dumping is performing id=[cdmp_20120822100731]
    Wed Aug 22 10:09:20 2012
    Trace dumping is performing id=[cdmp_20120822100732]
    Wed Aug 22 10:25:54 2012

    2012-08-22 10:01:23.571: [  CRSEVT][11527]32CAAMonitorHandler :: 0:Could not join /oracle/opt/product/10.2.0/crs/bin/racgwrap(check)
    category: 1234, operation: scls_process_join, loc: childcrash, OS error: 0, other: Abnormal termination of the child
    2012-08-22 10:01:31.473: [  CRSEVT][11527]32CAAMonitorHandler :: 0:Action Script /oracle/opt/product/10.2.0/crs/bin/racgwrap(check) timed out for ora.crmrz01.vip! (timeout=60)
    2012-08-22 10:01:31.474: [  CRSAPP][11527]32CheckResource error for ora.crmrz01.vip error code = -2
    2012-08-22 10:03:04.553: [  CRSEVT][11530]32CAAMonitorHandler :: 0:Could not join /oracle/opt/product/10.2.0/crs/bin/racgwrap(check)
    category: 1234, operation: scls_process_join, loc: childcrash, OS error: 0, other: Abnormal termination of the child
    2012-08-22 10:03:04.553: [  CRSEVT][11530]32CAAMonitorHandler :: 0:Action Script /oracle/opt/product/10.2.0/crs/bin/racgwrap(check) timed out for ora.crmrz01.vip! (timeout=60)
    2012-08-22 10:03:04.554: [  CRSAPP][11530]32CheckResource error for ora.crmrz01.vip error code = -2
    2012-08-22 10:03:39.055: [ COMMCRS][3866]Authorization failed, network error
    2012-08-22 10:03:39.138: [  OCRSRV][3866]th_select_answer: Failure in answer. clsc ret [3]
    2012-08-22 10:03:39.630: [ COMMCRS][3866]Authorization failed, network error
    2012-08-22 10:03:39.814: [  OCRSRV][3866]th_select_answer: Failure in answer. clsc ret [3]
    2012-08-22 10:04:13.506: [ COMMCRS][3866]Authorization failed, network error
    2012-08-22 10:04:13.522: [  OCRSRV][3866]th_select_answer: Failure in answer. clsc ret [3]
    2012-08-22 10:04:13.610: [ COMMCRS][3866]Authorization failed, network error
    2012-08-22 10:04:13.618: [  OCRSRV][3866]th_select_answer: Failure in answer. clsc ret [3]
    2012-08-22 10:04:44.509: [ COMMCRS][3866]Authorization failed, network error
    2012-08-22 10:04:44.525: [  OCRSRV][3866]th_select_answer: Failure in answer. clsc ret [3]
    2012-08-22 10:04:44.898: [ COMMCRS][3866]Authorization failed, network error
    2012-08-22 10:04:44.915: [  OCRSRV][3866]th_select_answer: Failure in answer. clsc ret [3]
    2012-08-22 10:04:58.959: [ COMMCRS][3866]Authorization failed, network error
    2012-08-22 10:04:58.963: [  OCRSRV][3866]th_select_answer: Failure in answer. clsc ret [3]
    2012-08-22 10:04:59.047: [ COMMCRS][3866]Authorization failed, network 2012-08-22 10:12:25.310: [  CRSRES][11569]32startRunnable: setting CLI values
    crs 日志network error 基本上可以确定当时的网络是存在问题

  • RAC 11GR2 + ora-00206 after disable FRA

    Hi all,
    I had a diskgroup FRA_DATA with a size of 600Go and i want to reduce this diskgroup to 300Go.
    I put the database RAC in noarchivelog mode and i have deactivate the FRA.
    I reduce the size of this diskgroup to 300Go.
    Then, when i restart the database, i have this problem :
    Oracle want to find the controlfile of my FRA but it doesn't exist.
    this is an extract of the init file :
    control_files = "+RACDB_DATA/lpm/controlfile/current.256.756570549"
    control_files = "+FRA_DATA/lpm/controlfile/current.256.756570549"
    This is an extract of the alert.log :
    ALTER DATABASE OPEN
    This instance was first to open
    Picked broadcast on commit scheme to generate SCNs
    SUCCESS: diskgroup FRA_DATA was dismounted
    SUCCESS: diskgroup FRA_DATA was dismounted
    Wed Oct 12 13:49:27 2011
    Errors in file /oracle/app/oracle/diag/rdbms/lpm/lpm1/trace/lpm1_ckpt_27957.trc:
    ORA-00206: error in writing (block 3, # blocks 1) of control file
    ORA-00202: control file: '+FRA_DATA/lpm/controlfile/current.256.756570549'
    ORA-15078: ASM diskgroup was forcibly dismounted
    ORA-15078: ASM diskgroup was forcibly dismounted
    Errors in file /oracle/app/oracle/diag/rdbms/lpm/lpm1/trace/lpm1_ckpt_27957.trc:
    ORA-00221: error on write to control file
    ORA-00206: error in writing (block 3, # blocks 1) of control file
    ORA-00202: control file: '+FRA_DATA/lpm/controlfile/current.256.756570549'
    ORA-15078: ASM diskgroup was forcibly dismounted
    ORA-15078: ASM diskgroup was forcibly dismounted
    CKPT (ospid: 27957): terminating the instance due to error 221
    Wed Oct 12 13:49:28 2011
    System state dump is made for local instance
    System State dumped to trace file /oracle/app/oracle/diag/rdbms/lpm/lpm1/trace/lpm1_diag_27743.trc
    Wed Oct 12 13:49:28 2011
    ORA-1092 : opitsk aborting process
    Trace dumping is performing id=[cdmp_20111012134928]
    Wed Oct 12 13:49:28 2011
    License high water mark = 1
    Instance terminated by CKPT, pid = 27957
    USER (ospid: 28062): terminating the instance
    Instance terminated by USER, pid = 28062
    Could you help me plase ?
    Thanks !!!
    Edited by: user12064507 on 12 oct. 2011 05:14

    Please go to anyone one of the rac node where you have the "lpm" db instances available and start the instance as follows :
    Assuming your NODE1 instance name is lpm1. Go to RDBMS home and export the sid lpm1 :
    $ cd $ORACLE_HOME ---> oracle home is the rdbms home here
    $ export ORACLE_SID=lpm1
    $ sqlplus / as sysdba
    SQL> startup nomount ;
    sql> show parameter control_files ;
    Please post the result of above parameter CONTROL_FILES ...
    Regards,
    IMRAN KHAN

  • Production database restarted

    Hi Guru,
    Today morning when i ched my production env i saw it was restrted last night not getting the cause exactley.Here is the log file details any body explain me the cause plz.I need to explane to the cutomer for this.
    Starting ORACLE instance (normal)
    LICENSE_MAX_SESSION = 0
    LICENSE_SESSIONS_WARNING = 0
    Interface type 1 en3 192.168.3.0 configured from OCR for use as a cluster interconnect
    Interface type 1 en2 172.19.21.128 configured from OCR for use as a public interface
    Picked latch-free SCN scheme 3
    WARNING: db_recovery_file_dest is same as db_create_online_log_dest_2
    Autotune of undo retention is turned on.
    LICENSE_MAX_USERS = 0
    SYS auditing is disabled
    ksdpec: called for event 13740 prior to event group initialization
    Starting up ORACLE RDBMS Version: 10.2.0.3.0.
    System parameters with non-default values:
    processes = 150
    __shared_pool_size = 553648128
    __large_pool_size = 16777216
    __java_pool_size = 16777216
    __streams_pool_size = 16777216
    spfile = +SOADATA/soapr/spfilesoapr.ora
    nls_language = ENGLISH
    nls_territory = UNITED KINGDOM
    sga_target = 3221225472
    control_files = SOADATA/soapr/controlfile/current.260.643395251, SOAFLASH/soapr/controlfile/current.256.643395251
    db_block_size = 8192
    __db_cache_size = 2600468480
    compatible = 10.2.0.3.0
    log_archive_dest_1 = LOCATION=+SOAFLASH/SOAPR/
    log_archive_format = %t_%s_%r.dbf
    Cluster communication is configured to use the following interface(s) for this instance
    192.168.3.21
    Thu Jun 11 04:02:14 2009
    cluster interconnect IPC version:Oracle UDP/IP (generic)
    IPC Vendor 1 proto 2
    PSP0 started with pid=4, OS id=90296
    DIAG started with pid=3, OS id=557234
    PMON started with pid=2, OS id=233714
    LMON started with pid=5, OS id=438470
    LMD0 started with pid=6, OS id=520246
    LMS0 started with pid=7, OS id=553174
    LMS1 started with pid=8, OS id=491544
    MMAN started with pid=9, OS id=360658
    DBW0 started with pid=10, OS id=291018
    LGWR started with pid=11, OS id=516102
    CKPT started with pid=12, OS id=282782
    SMON started with pid=13, OS id=544788
    RECO started with pid=14, OS id=221322
    CJQ0 started with pid=15, OS id=573464
    MMON started with pid=16, OS id=585758
    MMNL started with pid=17, OS id=589856
    Thu Jun 11 04:02:15 2009
    starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
    starting up 1 shared server(s) ...
    Thu Jun 11 04:02:15 2009
    lmon registered with NM - instance id 2 (internal mem no 1)
    Thu Jun 11 04:02:16 2009
    Reconfiguration started (old inc 0, new inc 68)
    List of nodes:
    0 1
    Global Resource Directory frozen
    * allocate domain 0, invalid = TRUE
    Communication channels reestablished
    * domain 0 valid = 1 according to instance 0

    Thankes for understanding thats why only i am thinking here is total alert log file.I am also confusing the cause it may be due to communication chennle reestablised?
    Starting ORACLE instance (normal)
    Thu Jun 11 04:02:42 2009
    Specified value of sga_max_size is too small, bumping to 3841982464
    LICENSE_MAX_SESSION = 0
    LICENSE_SESSIONS_WARNING = 0
    Interface type 1 en3 192.168.3.0 configured from OCR for use as a cluster interconnect
    Interface type 1 en2 172.19.21.128 configured from OCR for use as a public interface
    Picked latch-free SCN scheme 3
    WARNING: db_recovery_file_dest is same as db_create_online_log_dest_2
    Autotune of undo retention is turned on.
    LICENSE_MAX_USERS = 0
    SYS auditing is disabled
    ksdpec: called for event 13740 prior to event group initialization
    Starting up ORACLE RDBMS Version: 10.2.0.3.0.
    System parameters with non-default values:
    processes = 100
    sessions = 410
    sga_max_size = 3841982464
    shared_pool_size = 1073741824
    large_pool_size = 117440512
    java_pool_size = 536870912
    spfile = +WEBDATA/webpr/spfilewebpr.ora
    control_files = WEBDATA/webpr/controlfile/current.256.645528657, WEBFLASH/webpr/controlfile/current.259.645528657
    db_block_size = 8192
    db_cache_size = 2097152000
    compatible = 10.2.0.3.0
    log_archive_dest_1 = LOCATION=+WEBDATA/WEBPR/
    log_archive_format = %t_%s_%r.dbf
    db_file_multiblock_read_count= 16
    cluster_database = TRUE
    cluster_database_instances= 2
    db_create_file_dest = +WEBDATA
    db_create_online_log_dest_1= +WEBDATA
    db_create_online_log_dest_2= +WEBFLASH
    db_recovery_file_dest = +WEBFLASH
    db_recovery_file_dest_size= 21474836480
    thread = 2
    instance_number = 2
    undo_management = AUTO
    undo_tablespace = UNDOTBS2
    remote_login_passwordfile= EXCLUSIVE
    db_domain = phones4u.ltd
    service_names = WEBPR.phones4u.ltd
    dispatchers = (PROTOCOL=TCP) (SERVICE=WEBPRXDB)
    remote_listener = LISTENERS_WEBPR
    job_queue_processes = 2
    background_dump_dest = /app/oracle/WEB/web_home/admin/WEBPR/bdump
    user_dump_dest = /app/oracle/WEB/web_home/admin/WEBPR/udump
    core_dump_dest = /app/oracle/WEB/web_home/admin/WEBPR/cdump
    audit_file_dest = /app/oracle/WEB/web_home/admin/WEBPR/adump
    db_name = WEBPR
    open_cursors = 300
    pga_aggregate_target = 2147483648
    aq_tm_processes = 1
    Cluster communication is configured to use the following interface(s) for this instance
    192.168.3.21
    Thu Jun 11 04:02:43 2009
    cluster interconnect IPC version:Oracle UDP/IP (generic)
    IPC Vendor 1 proto 2
    DIAG started with pid=3, OS id=602288
    PMON started with pid=2, OS id=688270
    PSP0 started with pid=4, OS id=409774
    LMON started with pid=5, OS id=733286
    LMD0 started with pid=6, OS id=680058
    LMS0 started with pid=7, OS id=716928
    LMS1 started with pid=8, OS id=725092
    MMAN started with pid=9, OS id=745598
    DBW0 started with pid=10, OS id=749698
    LGWR started with pid=11, OS id=753776
    CKPT started with pid=12, OS id=757874
    SMON started with pid=13, OS id=761972
    RECO started with pid=14, OS id=766070
    CJQ0 started with pid=15, OS id=770168
    MMON started with pid=16, OS id=774266
    MMNL started with pid=17, OS id=778364
    Thu Jun 11 04:02:44 2009
    starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
    starting up 1 shared server(s) ...
    Thu Jun 11 04:02:44 2009
    lmon registered with NM - instance id 2 (internal mem no 1)
    Thu Jun 11 04:02:45 2009
    Reconfiguration started (old inc 0, new inc 20)
    List of nodes:
    0 1
    Global Resource Directory frozen
    * allocate domain 0, invalid = TRUE
    Communication channels reestablished
    * domain 0 valid = 1 according to instance 0
    Thu Jun 11 04:02:45 2009
    Master broadcasted resource hash value bitmaps
    Non-local Process blocks cleaned out
    Thu Jun 11 04:02:45 2009
    LMS 0: 0 GCS shadows cancelled, 0 closed
    Thu Jun 11 04:02:45 2009
    LMS 1: 0 GCS shadows cancelled, 0 closed
    Set master node info
    Submitted all remote-enqueue requests
    Dwn-cvts replayed, VALBLKs dubious
    All grantable enqueues granted
    Thu Jun 11 04:02:45 2009
    LMS 0: 0 GCS shadows traversed, 0 replayed
    Thu Jun 11 04:02:45 2009
    LMS 1: 0 GCS shadows traversed, 0 replayed
    Thu Jun 11 04:02:45 2009
    Submitted all GCS remote-cache requests
    Fix write in gcs resources
    Reconfiguration complete
    LCK0 started with pid=20, OS id=729238
    Thu Jun 11 04:02:47 2009
    ALTER DATABASE MOUNT
    Thu Jun 11 04:02:47 2009
    Starting background process ASMB
    ASMB started with pid=22, OS id=737424
    Starting background process RBAL
    RBAL started with pid=23, OS id=811148
    Thu Jun 11 04:02:52 2009
    SUCCESS: diskgroup WEBDATA was mounted
    SUCCESS: diskgroup WEBFLASH was mounted
    Thu Jun 11 04:02:59 2009
    Setting recovery target incarnation to 1
    Thu Jun 11 04:02:59 2009
    Successful mount of redo thread 2, with mount id 843760124
    Thu Jun 11 04:02:59 2009
    Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)
    Completed: ALTER DATABASE MOUNT
    Thu Jun 11 04:03:00 2009
    ALTER DATABASE OPEN
    Picked broadcast on commit scheme to generate SCNs
    Thu Jun 11 04:03:00 2009
    LGWR: STARTING ARCH PROCESSES
    ARC0 started with pid=26, OS id=921634
    Thu Jun 11 04:03:00 2009
    ARC0: Archival started
    ARC1 started with pid=27, OS id=950398
    Thu Jun 11 04:03:00 2009
    ARC1: Archival started
    LGWR: STARTING ARCH PROCESSES COMPLETE
    Thread 2 opened at log sequence 1427
    Current log# 3 seq# 1427 mem# 0: +WEBDATA/webpr/onlinelog/group_3.275.645530383
    Current log# 3 seq# 1427 mem# 1: +WEBFLASH/webpr/onlinelog/group_3.256.645530383
    Successful open of redo thread 2
    Thu Jun 11 04:03:01 2009
    MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
    Thu Jun 11 04:03:01 2009
    ARC1: Becoming the 'no FAL' ARCH
    ARC1: Becoming the 'no SRL' ARCH
    Thu Jun 11 04:03:01 2009
    ARC0: Becoming the heartbeat ARCH
    Thu Jun 11 04:03:01 2009
    SMON: enabling cache recovery
    Thu Jun 11 04:03:02 2009
    Successfully onlined Undo Tablespace 4.
    Thu Jun 11 04:03:02 2009
    SMON: enabling tx recovery
    Thu Jun 11 04:03:02 2009
    Database Characterset is WE8ISO8859P1
    replication_dependency_tracking turned off (no async multimaster replication found)
    Starting background process QMNC
    QMNC started with pid=29, OS id=925830
    Thu Jun 11 04:03:12 2009
    Completed: ALTER DATABASE OPEN

  • Database was down

    Hi Gurus,
    When i checked the alert log file in my prod environment i saw morning my database was down and automaticaly it was up.please let me know ehat might be the cause of this.
    Tue Jan 13 08:49:34 2009
    lmon registered with NM - instance id 1 (internal mem no 0)
    Tue Jan 13 08:49:38 2009
    Reconfiguration started (old inc 0, new inc 8)
    List of nodes:
    0 1
    Global Resource Directory frozen
    * allocate domain 0, invalid = TRUE
    Communication channels reestablished
    * domain 0 valid according to instance 1
    * domain 0 valid = 1 according to instance 1
    Tue Jan 13 08:49:38 2009
    Master broadcasted resource hash value bitmaps
    Non-local Process blocks cleaned out
    Tue Jan 13 08:49:38 2009
    LMS 1: 0 GCS shadows cancelled, 0 closed
    Tue Jan 13 08:49:38 2009
    LMS 0: 0 GCS shadows cancelled, 0 closed
    Tue Jan 13 08:49:38 2009
    LMS 2: 0 GCS shadows cancelled, 0 closed
    Set master node info
    Submitted all remote-enqueue requests
    Dwn-cvts replayed, VALBLKs dubious
    All grantable enqueues granted
    Tue Jan 13 08:49:40 2009
    LMS 0: 0 GCS shadows traversed, 0 replayed
    Tue Jan 13 08:49:40 2009
    LMS 1: 0 GCS shadows traversed, 0 replayed
    Tue Jan 13 08:49:40 2009
    LMS 2: 0 GCS shadows traversed, 0 replayed
    Tue Jan 13 08:49:40 2009
    Submitted all GCS remote-cache requests
    Post SMON to start 1st pass IR
    Fix write in gcs resources
    Reconfiguration complete
    LCK0 started with pid=20, OS id=905386
    Tue Jan 13 08:49:43 2009
    ALTER DATABASE MOUNT
    Tue Jan 13 08:49:43 2009
    Starting background process ASMB
    ASMB started with pid=22, OS id=2711718
    Starting background process RBAL
    RBAL started with pid=23, OS id=2768918
    Tue Jan 13 08:49:50 2009
    SUCCESS: diskgroup ASM_DATA was mounted
    Tue Jan 13 08:49:55 2009
    Setting recovery target incarnation to 4
    Tue Jan 13 08:49:55 2009
    Successful mount of redo thread 1, with mount id 3534255914
    Tue Jan 13 08:49:55 2009
    Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)
    Completed: ALTER DATABASE MOUNT
    Tue Jan 13 08:49:56 2009
    ALTER DATABASE OPEN
    Tue Jan 13 08:49:56 2009
    Picked broadcast on commit scheme to generate SCNs
    SUCCESS: diskgroup ASM_FLASH was mounted
    SUCCESS: diskgroup ASM_FLASH was dismounted
    SUCCESS: diskgroup ASM_FLASH was mounted
    SUCCESS: diskgroup ASM_FLASH was dismounted
    SUCCESS: diskgroup ASM_FLASH was mounted
    SUCCESS: diskgroup ASM_FLASH was dismounted
    SUCCESS: diskgroup ASM_FLASH was mounted
    SUCCESS: diskgroup ASM_FLASH was dismounted
    SUCCESS: diskgroup ASM_FLASH was mounted
    SUCCESS: diskgroup ASM_FLASH was dismounted
    Thanks,
    Preet

    Hi,
    this is RAC?
    I dont think that db was down. In aler.log isnt any information about stopping backgroud processes.
    There should be samething like "shutting down instance" or "alter database close"
    There was only reconfiguration started.
    Regards,
    Tom
    http://oracledba.cz

  • ORA-01034: Message 1034 not found; No message file for product=RDBMS

    Hi!
    After creating a RAC Database (10.2.0.5) on Redhat Enterprise Linux 64-bit, we checked: Listeners, Processes and Environment variables.
    Everything was set up properly and running.
    When trying to connect as sysdba the following message appeared:
    $sqlplus / as sysdba
    sql>select * from v$instance;
    select * from v$instance
    ERROR at line 1:
    ORA-01034: Message 1034 not found; No message file for product=RDBMS,
    facility=ORA
    A shutdown and startup did not help.
    What could be the issue here?
    Thanks!

    Here is an excerpt of the alert log. No ora-
    ALTER DATABASE OPEN
    Picked broadcast on commit scheme to generate SCNs
    Fri Feb 24 11:26:50 CET 2012
    LGWR: STARTING ARCH PROCESSES
    ARC0 started with pid=32, OS id=4930
    Fri Feb 24 11:26:50 CET 2012
    ARC0: Archival started
    ARC1: Archival started
    LGWR: STARTING ARCH PROCESSES COMPLETE
    ARC1 started with pid=33, OS id=4932
    Fri Feb 24 11:26:50 CET 2012
    SUCCESS: diskgroup LOG01 was mounted
    SUCCESS: diskgroup LOG02 was mounted
    Thread 1 opened at log sequence 10
      Current log# 11 seq# 10 mem# 0: +LOG01/omdemo/onlinelog/group_11.265.776085353
      Current log# 11 seq# 10 mem# 1: +LOG02/omdemo/onlinelog/group_11.265.776085353
    Successful open of redo thread 1
    Fri Feb 24 11:26:50 CET 2012
    ARC0: Becoming the 'no FAL' ARCH
    ARC0: Becoming the 'no SRL' ARCH
    Fri Feb 24 11:26:50 CET 2012
    ARC1: Becoming the heartbeat ARCH
    Fri Feb 24 11:26:50 CET 2012
    SMON: enabling cache recovery
    Fri Feb 24 11:26:50 CET 2012
    Successfully onlined Undo Tablespace 1.
    Fri Feb 24 11:26:50 CET 2012
    SMON: enabling tx recovery
    Fri Feb 24 11:26:50 CET 2012
    Database Characterset is WE8MSWIN1252
    replication_dependency_tracking turned off (no async multimaster replication found)
    Starting background process QMNC
    QMNC started with pid=34, OS id=4934
    Fri Feb 24 11:26:52 CET 2012
    Completed: ALTER DATABASE OPEN
    Sat Feb 25 02:00:23 CET 2012
    Thread 1 advanced to log sequence 11 (LGWR switch)
      Current log# 12 seq# 11 mem# 0: +LOG01/omdemo/onlinelog/group_12.266.776085355
      Current log# 12 seq# 11 mem# 1: +LOG02/omdemo/onlinelog/group_12.266.776085355$ps -ef|grep mon
    oracle    4694     1  0 Feb24 ?        00:00:10 /oracle/app/oracle/product/10.2.0/db_1/bin/racgimon startd OMDEMO
    oracle    4773     1  0 Feb24 ?        00:00:04 ora_pmon_OMDEMO1
    oracle    4779     1  0 Feb24 ?        00:00:03 ora_lmon_OMDEMO1
    oracle    4819     1  0 Feb24 ?        00:00:05 ora_smon_OMDEMO1
    oracle    4825     1  0 Feb24 ?        00:00:00 ora_mmon_OMDEMO1
    root     10429     1  0 Jan24 ?        03:29:11 /oracle/app/grid/11.2.0/grid/bin/osysmond.bin
    root     10475     1  0 Jan24 ?        00:01:44 /oracle/app/grid/11.2.0/grid/bin/cssdmonitor
    grid     11214     1  0 Jan24 ?        00:00:46 asm_pmon_+ASM1
    grid     11301     1  0 Jan24 ?        00:00:19 asm_lmon_+ASM1
    grid     11335     1  0 Jan24 ?        00:00:00 asm_smon_+ASM1
    grid     11343     1  0 Jan24 ?        00:00:00 asm_gmon_+ASM1
    grid     11347     1  0 Jan24 ?        00:00:00 asm_mmon_+ASM1
    dbus     11972     1  0 Jan17 ?        00:00:03 dbus-daemon --system
    xfs      14493     1  0 Jan17 ?        00:00:00 xfs -droppriv -daemon
    avahi    14594     1  0 Jan17 ?        00:00:00 avahi-daemon: running [at1srv0080.local]
    avahi    14595 14594  0 Jan17 ?        00:00:00 avahi-daemon: chroot helper
    root     14671     1  0 Jan17 ?        00:00:00 /usr/sbin/gdm-binary -nodaemon
    root     14736 14671  0 Jan17 ?        00:00:00 /usr/sbin/gdm-binary -nodaemon
    oracle   17059 12052  0 10:42 pts/1    00:00:00 grep mon
    [oracle@at1srv0080 /]$ sqlplus / as sysdba
    SQL*Plus: Release 10.2.0.5.0 - Production on Mon Feb 27 10:46:51 2012
    Copyright (c) 1982, 2010, Oracle.  All Rights Reserved.
    Connected to an idle instance.
    SQL> startup
    ORA-01078: failure in processing system parameters
    LRM-00109: could not open parameter file '/oracle/app/oracle/product/10.2.0/db_1/dbs/initOCTA1.ora'

  • My one cluster crashed ,Plz help me

    Hi
    I have oracle 10AS and oracle 10gDS running on HP-UX 11i platform configured with real application cluster.
    In two different machines we are running our database Server and in other two different machines we are running application server.
    Today due to short circuit our one unix machine containing the database server suffer
    With hardware crash so now database is not getting up.
    Please tell me the command and how I can up my one remaining cluster with other database machine and after repair of the damaged database sever machine how can I synchronize my both the cluster.
    Thanks and Regards
    Deep

    (1)Database was shutdown
    (2) System data was not configured right
    Sat Jun 17 00:11:59 2006
    Starting ORACLE instance (normal)
    LICENSE_MAX_SESSION = 0
    LICENSE_SESSIONS_WARNING = 0
    Interface type 1 lan0 80.0.0.0 configured from OCR for use as a cluster interconnect
    Interface type 1 lan900 198.100.0.0 configured from OCR for use as a public interface
    Picked latch-free SCN scheme 1
    Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/10.2.0/db_1/dbs/arch
    Autotune of undo retention is turned on.
    LICENSE_MAX_USERS = 0
    SYS auditing is disabled
    ksdpec: called for event 13740 prior to event group initialization
    Starting up ORACLE RDBMS Version: 10.2.0.1.0.
    System parameters with non-default values:
    processes = 150
    __shared_pool_size = 583008256
    __large_pool_size = 4194304
    __java_pool_size = 4194304
    __streams_pool_size = 0
    spfile = /dev/vg_rac/rkumaondb_spfilel_raw
    sga_target = 612368384
    control_files = /dev/vg_rac/rkumaondb_control1_raw, /dev/vg_rac/rkumaondb_control2_raw
    db_block_size = 8192
    __db_cache_size = 16777216
    compatible = 10.2.0.1.0
    db_file_multiblock_read_count= 16
    cluster_database = TRUE
    cluster_database_instances= 2
    thread = 2
    instance_number = 2
    undo_management = AUTO
    undo_tablespace = UNDOTBS2
    remote_login_passwordfile= EXCLUSIVE
    db_domain =
    dispatchers = (PROTOCOL=TCP) (SERVICE=kumaondbXDB)
    remote_listener = LISTENERS_KUMAONDB
    job_queue_processes = 10
    background_dump_dest = /u01/app/oracle/product/10.2.0/db_1/admin/kumaondb/bdump
    user_dump_dest = /u01/app/oracle/product/10.2.0/db_1/admin/kumaondb/udump
    core_dump_dest = /u01/app/oracle/product/10.2.0/db_1/admin/kumaondb/cdump
    audit_file_dest = /u01/app/oracle/product/10.2.0/db_1/admin/kumaondb/adump
    db_name = kumaondb
    open_cursors = 300
    pga_aggregate_target = 203423744
    Cluster communication is configured to use the following interface(s) for this instance
    80.0.0.2
    Sat Jun 17 00:12:00 2006
    cluster interconnect IPC version:Oracle UDP/IP
    IPC Vendor 1 proto 2
    PMON started with pid=2, OS id=4013
    DIAG started with pid=3, OS id=4032
    PSP0 started with pid=4, OS id=4044
    LMON started with pid=5, OS id=4046
    LMD0 started with pid=6, OS id=4048
    LMS0 started with pid=7, OS id=4050
    MMAN started with pid=8, OS id=4052
    DBW0 started with pid=9, OS id=4054
    LGWR started with pid=10, OS id=4057
    CKPT started with pid=11, OS id=4059
    SMON started with pid=12, OS id=4061
    RECO started with pid=13, OS id=4063
    CJQ0 started with pid=14, OS id=4065
    MMON started with pid=15, OS id=4067
    Sat Jun 17 00:12:01 2006
    starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
    MMNL started with pid=16, OS id=4096
    Sat Jun 17 00:12:01 2006
    starting up 1 shared server(s) ...
    Sat Jun 17 00:12:02 2006
    lmon registered with NM - instance id 2 (internal mem no 1)
    Sat Jun 17 00:12:02 2006
    Reconfiguration started (old inc 0, new inc 2)
    pseudo shared rm latch used
    List of nodes:
    1
    Global Resource Directory frozen
    * allocate domain 0, invalid = TRUE
    Communication channels reestablished
    Master broadcasted resource hash value bitmaps
    Non-local Process blocks cleaned out
    Sat Jun 17 00:12:02 2006
    LMS 0: 0 GCS shadows cancelled, 0 closed
    Set master node info
    Submitted all remote-enqueue requests
    Dwn-cvts replayed, VALBLKs dubious
    All grantable enqueues granted
    Post SMON to start 1st pass IR
    Sat Jun 17 00:12:02 2006
    LMS 0: 0 GCS shadows traversed, 0 replayed
    Sat Jun 17 00:12:02 2006
    Submitted all GCS remote-cache requests
    Post SMON to start 1st pass IR
    Fix write in gcs resources
    Reconfiguration complete
    LCK0 started with pid=19, OS id=4102
    Sat Jun 17 00:12:03 2006
    ALTER DATABASE MOUNT
    Sat Jun 17 00:12:03 2006
    This instance was first to mount
    Setting recovery target incarnation to 2
    Sat Jun 17 00:12:07 2006
    Successful mount of redo thread 2, with mount id 4106916051
    Sat Jun 17 00:12:07 2006
    Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)
    Completed: ALTER DATABASE MOUNT
    Sat Jun 17 00:12:08 2006
    ALTER DATABASE OPEN
    This instance was first to open
    Picked broadcast on commit scheme to generate SCNs
    Sat Jun 17 00:12:08 2006
    Thread 2 opened at log sequence 34
    Current log# 4 seq# 34 mem# 0: /dev/vg_rac/rkumaondb_redo2_2_raw
    Successful open of redo thread 2
    Sat Jun 17 00:12:08 2006
    MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
    Sat Jun 17 00:12:08 2006
    SMON: enabling cache recovery
    Sat Jun 17 00:12:08 2006
    Successfully onlined Undo Tablespace 5.
    Sat Jun 17 00:12:08 2006
    SMON: enabling tx recovery
    Sat Jun 17 00:12:08 2006
    Database Characterset is WE8ISO8859P1
    Sat Jun 17 00:12:08 2006
    Instance recovery: looking for dead threads
    Instance recovery: lock domain invalid but no dead threads
    Sat Jun 17 00:12:09 2006
    replication_dependency_tracking turned off (no async multimaster replication found)
    Starting background process QMNC
    QMNC started with pid=21, OS id=4295
    Sat Jun 17 00:12:11 2006
    Completed: ALTER DATABASE OPEN
    Sat Jun 17 00:12:21 2006
    ALTER SYSTEM SET service_names='kumaondb','orcl' SCOPE=MEMORY SID='kumaondb2';
    Sat Jun 17 00:15:25 2006
    Starting ORACLE instance (normal)
    Sat Jun 17 00:15:30 2006
    Starting background process EMN0
    EMN0 started with pid=28, OS id=9811
    Sat Jun 17 00:15:30 2006
    Shutting down instance: further logons disabled
    Sat Jun 17 00:15:30 2006
    Stopping background process CJQ0
    Sat Jun 17 00:15:30 2006
    Stopping background process MMNL
    Sat Jun 17 00:15:30 2006
    Stopping background process MMON
    Sat Jun 17 00:15:31 2006
    Shutting down instance (normal)
    License high water mark = 5
    Sat Jun 17 00:15:31 2006
    Stopping Job queue slave processes
    Sat Jun 17 00:15:31 2006
    Job queue slave processes stopped
    Sat Jun 17 00:15:32 2006
    ALTER SYSTEM SET service_names='kumaondb' SCOPE=MEMORY SID='kumaondb2';
    All dispatchers and shared servers shutdown
    Sat Jun 17 00:15:37 2006
    ALTER DATABASE CLOSE NORMAL
    Sat Jun 17 00:15:37 2006
    SMON: disabling tx recovery
    SMON: disabling cache recovery
    Sat Jun 17 00:15:37 2006
    Shutting down archive processes
    Archiving is disabled
    Archive process shutdown avoided: 0 active
    Thread 2 closed at log sequence 34
    Successful close of redo thread 2
    Sat Jun 17 00:15:37 2006
    Completed: ALTER DATABASE CLOSE NORMAL
    Sat Jun 17 00:15:37 2006
    ALTER DATABASE DISMOUNT
    Completed: ALTER DATABASE DISMOUNT
    ARCH: Archival disabled due to shutdown: 1090
    Shutting down archive processes
    Archiving is disabled
    Archive process shutdown avoided: 0 active
    ARCH: Archival disabled due to shutdown: 1090
    Shutting down archive processes
    Archiving is disabled
    Archive process shutdown avoided: 0 active
    Sat Jun 17 00:15:38 2006
    freeing rdom 0
    Sat Jun 17 00:15:46 2006
    Starting ORACLE instance (normal)
    LICENSE_MAX_SESSION = 0
    LICENSE_SESSIONS_WARNING = 0
    Interface type 1 lan0 80.0.0.0 configured from OCR for use as a cluster interconnect
    Interface type 1 lan900 198.100.0.0 configured from OCR for use as a public interface
    Picked latch-free SCN scheme 1
    Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/10.2.0/db_1/dbs/arch
    Autotune of undo retention is turned on.
    LICENSE_MAX_USERS = 0
    SYS auditing is disabled
    ksdpec: called for event 13740 prior to event group initialization
    Starting up ORACLE RDBMS Version: 10.2.0.1.0.
    System parameters with non-default values:
    processes = 150
    __shared_pool_size = 583008256
    __large_pool_size = 4194304
    __java_pool_size = 4194304
    __streams_pool_size = 0
    spfile = /dev/vg_rac/rkumaondb_spfilel_raw
    sga_target = 612368384
    control_files = /dev/vg_rac/rkumaondb_control1_raw, /dev/vg_rac/rkumaondb_control2_raw
    db_block_size = 8192
    __db_cache_size = 16777216
    compatible = 10.2.0.1.0
    db_file_multiblock_read_count= 16
    cluster_database = TRUE
    cluster_database_instances= 2
    thread = 2
    instance_number = 2
    undo_management = AUTO
    undo_tablespace = UNDOTBS2
    remote_login_passwordfile= EXCLUSIVE
    db_domain =
    dispatchers = (PROTOCOL=TCP) (SERVICE=kumaondbXDB)
    remote_listener = LISTENERS_KUMAONDB
    job_queue_processes = 10
    background_dump_dest = /u01/app/oracle/product/10.2.0/db_1/admin/kumaondb/bdump
    user_dump_dest = /u01/app/oracle/product/10.2.0/db_1/admin/kumaondb/udump
    core_dump_dest = /u01/app/oracle/product/10.2.0/db_1/admin/kumaondb/cdump
    audit_file_dest = /u01/app/oracle/product/10.2.0/db_1/admin/kumaondb/adump
    db_name = kumaondb
    open_cursors = 300
    pga_aggregate_target = 203423744
    Cluster communication is configured to use the following interface(s) for this instance
    80.0.0.2
    Sat Jun 17 00:15:46 2006
    cluster interconnect IPC version:Oracle UDP/IP
    IPC Vendor 1 proto 2
    PMON started with pid=2, OS id=10226
    DIAG started with pid=3, OS id=10230
    PSP0 started with pid=4, OS id=10253
    LMON started with pid=5, OS id=10259
    LMD0 started with pid=6, OS id=10261
    LMS0 started with pid=7, OS id=10263
    MMAN started with pid=8, OS id=10265
    DBW0 started with pid=9, OS id=10267
    LGWR started with pid=10, OS id=10269
    CKPT started with pid=11, OS id=10271
    SMON started with pid=12, OS id=10273
    RECO started with pid=13, OS id=10275
    CJQ0 started with pid=14, OS id=10277
    MMON started with pid=15, OS id=10279
    Sat Jun 17 00:15:48 2006
    starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
    MMNL started with pid=16, OS id=10281
    Sat Jun 17 00:15:48 2006
    starting up 1 shared server(s) ...
    Sat Jun 17 00:15:48 2006
    lmon registered with NM - instance id 2 (internal mem no 1)
    Sat Jun 17 00:15:49 2006
    Reconfiguration started (old inc 0, new inc 2)
    pseudo shared rm latch used
    List of nodes:
    1
    Global Resource Directory frozen
    * allocate domain 0, invalid = TRUE
    Communication channels reestablished
    Master broadcasted resource hash value bitmaps
    Non-local Process blocks cleaned out
    Sat Jun 17 00:15:49 2006
    LMS 0: 0 GCS shadows cancelled, 0 closed
    Set master node info
    Submitted all remote-enqueue requests
    Dwn-cvts replayed, VALBLKs dubious
    All grantable enqueues granted
    Post SMON to start 1st pass IR
    Sat Jun 17 00:15:49 2006
    LMS 0: 0 GCS shadows traversed, 0 replayed
    Sat Jun 17 00:15:49 2006
    Submitted all GCS remote-cache requests
    Post SMON to start 1st pass IR
    Fix write in gcs resources
    Reconfiguration complete
    LCK0 started with pid=19, OS id=10321
    Sat Jun 17 00:15:50 2006
    ALTER DATABASE MOUNT
    Sat Jun 17 00:15:50 2006
    This instance was first to mount
    Setting recovery target incarnation to 2
    Sat Jun 17 00:15:54 2006
    Successful mount of redo thread 2, with mount id 4106911414
    Sat Jun 17 00:15:54 2006
    Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)
    Completed: ALTER DATABASE MOUNT
    Sat Jun 17 00:15:54 2006
    ALTER DATABASE OPEN
    This instance was first to open
    Picked broadcast on commit scheme to generate SCNs
    Sat Jun 17 00:15:54 2006
    Thread 2 opened at log sequence 34
    Current log# 4 seq# 34 mem# 0: /dev/vg_rac/rkumaondb_redo2_2_raw
    Successful open of redo thread 2
    Sat Jun 17 00:15:54 2006
    MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
    Sat Jun 17 00:15:54 2006
    SMON: enabling cache recovery
    Sat Jun 17 00:15:55 2006
    Successfully onlined Undo Tablespace 5.
    Sat Jun 17 00:15:55 2006
    SMON: enabling tx recovery
    Sat Jun 17 00:15:55 2006
    Database Characterset is WE8ISO8859P1
    Sat Jun 17 00:15:55 2006
    Instance recovery: looking for dead threads
    Instance recovery: lock domain invalid but no dead threads
    Sat Jun 17 00:15:56 2006
    replication_dependency_tracking turned off (no async multimaster replication found)
    Starting background process QMNC
    QMNC started with pid=21, OS id=10513
    Sat Jun 17 00:15:59 2006
    Completed: ALTER DATABASE OPEN
    Sat Jun 17 00:34:53 2006
    Memory Notification: Library Cache Object loaded into SGA
    Heap size 3541K exceeds notification threshold (2048K)
    Details in trace file /u01/app/oracle/product/10.2.0/db_1/admin/kumaondb/udump/kumaondb2_ora_6539.trc
    KGL object name :XDB.XDbD/PLZ01TcHgNAgAIIegtw==
    Sat Jun 17 00:34:53 2006
    Memory Notification: Library Cache Object loaded into SGA
    Heap size 3669K exceeds notification threshold (2048K)
    Details in trace file /u01/app/oracle/product/10.2.0/db_1/admin/kumaondb/udump/kumaondb2_ora_6539.trc
    KGL object name :XDB.XD/HSUFelsTLjgNAAwbgNaAQ==
    Sat Jun 17 00:35:17 2006
    Memory Notification: Library Cache Object loaded into SGA
    Heap size 3099K exceeds notification threshold (2048K)
    Details in trace file /u01/app/oracle/product/10.2.0/db_1/admin/kumaondb/udump/kumaondb2_ora_6539.trc
    KGL object name :SELECT TOWNER, TNAME, NAME, LENGTH, PRECISION, SCALE, TYPE, ISNULL, CONNAME, COLID, INTCOLID, SEGCOLID, COMMENT$, DEFAULT$, DFLTLEN, ENABLED, DEFER, FLAGS, COLPROP, ADTNAME, ADTOWNER, CHARSETID, CHARSETFORM, FSPRECISION, LFPRECISION, CHARLEN, TFLAGS, TYPESYN FROM SYS.EXU9COO WHERE TOBJID = :1 ORDER BY INTCOLID
    Sat Jun 17 00:42:30 2006
    WARNING: inbound connection timed out (ORA-3136)
    Sat Jun 17 00:43:36 2006
    Memory Notification: Library Cache Object loaded into SGA
    Heap size 2249K exceeds notification threshold (2048K)
    Details in trace file /u01/app/oracle/product/10.2.0/db_1/admin/kumaondb/udump/kumaondb2_ora_22289.trc
    KGL object name :SELECT SYNNAM, DBMS_JAVA.LONGNAME(SYNNAM), DBMS_JAVA.LONGNAME(SYNTAB), TABOWN, TABNODE, PUBLIC$, SYNOWN, SYNOWNID, TABOWNID, SYNOBJNO FROM SYS.EXU9PTS WHERE SYNOBJNO IN ( SELECT SYNOBJNO FROM SYS.EXU9TYPTU WHERE TABOBJNO = :1 ) ORDER BY SYNTIME
    Sat Jun 17 00:43:36 2006
    Memory Notification: Library Cache Object loaded into SGA
    Heap size 2107K exceeds notification threshold (2048K)
    Details in trace file /u01/app/oracle/product/10.2.0/db_1/admin/kumaondb/udump/kumaondb2_ora_22289.trc
    KGL object name :SELECT SYNNAM, DBMS_JAVA.LONGNAME(SYNNAM), DBMS_JAVA.LONGNAME(SYNTAB), TABOWN, TABNODE, PUBLIC$, SYNOWN, SYNOWNID, TABOWNID, SYNOBJNO FROM SYS.EXU9UTSU WHERE SYNOBJNO IN ( SELECT SYNOBJNO FROM SYS.EXU9TYPTU2 WHERE TABOBJNO = :1 ) ORDER BY SYNTIME
    Sat Jun 17 00:59:32 2006
    Starting background process EMN0
    EMN0 started with pid=31, OS id=17538
    Sat Jun 17 00:59:32 2006
    Shutting down instance: further logons disabled
    Sat Jun 17 00:59:32 2006
    Stopping background process CJQ0
    Sat Jun 17 00:59:32 2006
    Stopping background process MMNL
    Sat Jun 17 00:59:33 2006
    Stopping background process MMON
    Sat Jun 17 00:59:33 2006
    Shutting down instance (immediate)
    License high water mark = 10
    Sat Jun 17 00:59:33 2006
    Stopping Job queue slave processes
    Sat Jun 17 00:59:33 2006
    Job queue slave processes stopped
    All dispatchers and shared servers shutdown
    Sat Jun 17 00:59:36 2006
    ALTER DATABASE CLOSE NORMAL
    Sat Jun 17 00:59:36 2006
    SMON: disabling tx recovery
    SMON: disabling cache recovery
    Sat Jun 17 00:59:36 2006
    Shutting down archive processes
    Archiving is disabled
    Archive process shutdown avoided: 0 active
    Thread 2 closed at log sequence 34
    Successful close of redo thread 2
    Sat Jun 17 00:59:37 2006
    Completed: ALTER DATABASE CLOSE NORMAL
    Sat Jun 17 00:59:37 2006
    ALTER DATABASE DISMOUNT
    Completed: ALTER DATABASE DISMOUNT
    ARCH: Archival disabled due to shutdown: 1089
    Shutting down archive processes
    Archiving is disabled
    Archive process shutdown avoided: 0 active
    ARCH: Archival disabled due to shutdown: 1089
    Shutting down archive processes
    Archiving is disabled
    Archive process shutdown avoided: 0 active
    Sat Jun 17 00:59:38 2006
    freeing rdom 0
    Sat Jun 17 01:10:44 2006
    Starting ORACLE instance (normal)
    LICENSE_MAX_SESSION = 0
    LICENSE_SESSIONS_WARNING = 0
    Interface type 1 lan0 80.0.0.0 configured from OCR for use as a cluster interconnect
    Interface type 1 lan900 198.100.0.0 configured from OCR for use as a public interface
    Picked latch-free SCN scheme 1
    Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/10.2.0/db_1/dbs/arch
    Autotune of undo retention is turned on.
    LICENSE_MAX_USERS = 0
    SYS auditing is disabled
    ksdpec: called for event 13740 prior to event group initialization
    Starting up ORACLE RDBMS Version: 10.2.0.1.0.
    System parameters with non-default values:
    processes = 150
    __shared_pool_size = 369098752
    __large_pool_size = 4194304
    __java_pool_size = 8388608
    __streams_pool_size = 0
    spfile = /dev/vg_rac/rkumaondb_spfilel_raw
    sga_target = 612368384
    control_files = /dev/vg_rac/rkumaondb_control1_raw, /dev/vg_rac/rkumaondb_control2_raw
    db_block_size = 8192
    __db_cache_size = 226492416
    compatible = 10.2.0.1.0
    db_file_multiblock_read_count= 16
    cluster_database = TRUE
    cluster_database_instances= 2
    thread = 2
    instance_number = 2
    undo_management = AUTO
    undo_tablespace = UNDOTBS2
    remote_login_passwordfile= EXCLUSIVE
    db_domain =
    dispatchers = (PROTOCOL=TCP) (SERVICE=kumaondbXDB)
    remote_listener = LISTENERS_KUMAONDB
    job_queue_processes = 10
    background_dump_dest = /u01/app/oracle/product/10.2.0/db_1/admin/kumaondb/bdump
    user_dump_dest = /u01/app/oracle/product/10.2.0/db_1/admin/kumaondb/udump
    core_dump_dest = /u01/app/oracle/product/10.2.0/db_1/admin/kumaondb/cdump
    audit_file_dest = /u01/app/oracle/product/10.2.0/db_1/admin/kumaondb/adump
    db_name = kumaondb
    open_cursors = 300
    pga_aggregate_target = 203423744
    Cluster communication is configured to use the following interface(s) for this instance
    80.0.0.2
    Sat Jun 17 01:10:45 2006
    cluster interconnect IPC version:Oracle UDP/IP
    IPC Vendor 1 proto 2
    PMON started with pid=2, OS id=4516
    DIAG started with pid=3, OS id=4541
    PSP0 started with pid=4, OS id=4547
    LMON started with pid=5, OS id=4549
    LMD0 started with pid=6, OS id=4551
    LMS0 started with pid=7, OS id=4553
    MMAN started with pid=8, OS id=4555
    DBW0 started with pid=9, OS id=4557
    LGWR started with pid=10, OS id=4559
    CKPT started with pid=11, OS id=4561
    SMON started with pid=12, OS id=4563
    RECO started with pid=13, OS id=4565
    CJQ0 started with pid=14, OS id=4567
    MMON started with pid=15, OS id=4592
    MMNL started with pid=16, OS id=4598
    Sat Jun 17 01:10:47 2006
    starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
    starting up 1 shared server(s) ...
    Sat Jun 17 01:10:47 2006
    lmon registered with NM - instance id 2 (internal mem no 1)
    Sat Jun 17 01:10:47 2006
    Reconfiguration started (old inc 0, new inc 2)
    pseudo shared rm latch used
    List of nodes:
    1
    Global Resource Directory frozen
    * allocate domain 0, invalid = TRUE
    Communication channels reestablished
    Master broadcasted resource hash value bitmaps
    Non-local Process blocks cleaned out
    Sat Jun 17 01:10:47 2006
    LMS 0: 0 GCS shadows cancelled, 0 closed
    Set master node info
    Submitted all remote-enqueue requests
    Dwn-cvts replayed, VALBLKs dubious
    All grantable enqueues granted
    Post SMON to start 1st pass IR
    Sat Jun 17 01:10:47 2006
    LMS 0: 0 GCS shadows traversed, 0 replayed
    Sat Jun 17 01:10:47 2006
    Submitted all GCS remote-cache requests
    Post SMON to start 1st pass IR
    Fix write in gcs resources
    Reconfiguration complete
    LCK0 started with pid=19, OS id=4604
    Sat Jun 17 01:10:48 2006
    ALTER DATABASE MOUNT
    Sat Jun 17 01:10:48 2006
    This instance was first to mount
    Setting recovery target incarnation to 2
    Sat Jun 17 01:10:52 2006
    Successful mount of redo thread 2, with mount id 4106921112
    Sat Jun 17 01:10:52 2006
    Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)
    Completed: ALTER DATABASE MOUNT
    Sat Jun 17 01:10:53 2006
    ALTER DATABASE OPEN
    This instance was first to open
    Picked broadcast on commit scheme to generate SCNs
    Sat Jun 17 01:10:53 2006
    Thread 2 opened at log sequence 34
    Current log# 4 seq# 34 mem# 0: /dev/vg_rac/rkumaondb_redo2_2_raw
    Successful open of redo thread 2
    Sat Jun 17 01:10:53 2006
    MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
    Sat Jun 17 01:10:53 2006
    SMON: enabling cache recovery
    Sat Jun 17 01:10:53 2006
    Successfully onlined Undo Tablespace 5.
    Sat Jun 17 01:10:53 2006
    SMON: enabling tx recovery
    Sat Jun 17 01:10:53 2006
    Database Characterset is WE8ISO8859P1
    Sat Jun 17 01:10:53 2006
    Instance recovery: looking for dead threads
    Instance recovery: lock domain invalid but no dead threads
    Sat Jun 17 01:10:54 2006
    replication_dependency_tracking turned off (no async multimaster replication found)
    Starting background process QMNC
    QMNC started with pid=21, OS id=4797
    Sat Jun 17 01:10:56 2006
    Completed: ALTER DATABASE OPEN
    Sat Jun 17 01:11:06 2006
    ALTER SYSTEM SET service_names='kumaondb','orcl' SCOPE=MEMORY SID='kumaondb2';
    Sat Jun 17 01:12:51 2006
    Starting ORACLE instance (normal)
    Sat Jun 17 01:12:57 2006
    Starting background process EMN0
    EMN0 started with pid=28, OS id=8001
    Sat Jun 17 01:12:57 2006
    Shutting down instance: further logons disabled
    Sat Jun 17 01:12:57 2006
    Stopping background process CJQ0
    Sat Jun 17 01:12:57 2006
    Stopping background process MMNL
    Sat Jun 17 01:12:58 2006
    Stopping background process MMON
    Sat Jun 17 01:12:59 2006
    ALTER SYSTEM SET service_names='kumaondb' SCOPE=MEMORY SID='kumaondb2';
    Sat Jun 17 01:12:59 2006
    Shutting down instance (normal)
    License high water mark = 5
    Sat Jun 17 01:12:59 2006
    Stopping Job queue slave processes
    Sat Jun 17 01:12:59 2006
    Job queue slave processes stopped
    All dispatchers and shared servers shutdown
    Sat Jun 17 01:13:06 2006
    ALTER DATABASE CLOSE NORMAL
    Sat Jun 17 01:13:06 2006
    SMON: disabling tx recovery
    SMON: disabling cache recovery
    Sat Jun 17 01:13:06 2006
    Shutting down archive processes
    Archiving is disabled
    Archive process shutdown avoided: 0 active
    Thread 2 closed at log sequence 34
    Successful close of redo thread 2
    Sat Jun 17 01:13:06 2006
    Completed: ALTER DATABASE CLOSE NORMAL
    Sat Jun 17 01:13:06 2006
    ALTER DATABASE DISMOUNT
    Completed: ALTER DATABASE DISMOUNT
    ARCH: Archival disabled due to shutdown: 1090
    Shutting down archive processes
    Archiving is disabled
    Archive process shutdown avoided: 0 active
    ARCH: Archival disabled due to shutdown: 1090
    Shutting down archive processes
    Archiving is disabled
    Archive process shutdown avoided: 0 active
    Sat Jun 17 01:13:07 2006
    freeing rdom 0
    Sat Jun 17 01:13:19 2006
    Starting ORACLE instance (normal)
    LICENSE_MAX_SESSION = 0
    LICENSE_SESSIONS_WARNING = 0
    Interface type 1 lan0 80.0.0.0 configured from OCR for use as a cluster interconnect
    Interface type 1 lan900 198.100.0.0 configured from OCR for use as a public interface
    Picked latch-free SCN scheme 1
    Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/10.2.0/db_1/dbs/arch
    Autotune of undo retention is turned on.
    LICENSE_MAX_USERS = 0
    SYS auditing is disabled
    ksdpec: called for event 13740 prior to event group initialization
    Starting up ORACLE RDBMS Version: 10.2.0.1.0.
    System parameters with non-default values:
    processes = 150
    __shared_pool_size = 373293056
    __large_pool_size = 4194304
    __java_pool_size = 8388608
    __streams_pool_size = 0
    spfile = /dev/vg_rac/rkumaondb_spfilel_raw
    sga_target = 612368384
    control_files = /dev/vg_rac/rkumaondb_control1_raw, /dev/vg_rac/rkumaondb_control2_raw
    db_block_size = 8192
    __db_cache_size = 222298112
    compatible = 10.2.0.1.0
    db_file_multiblock_read_count= 16
    cluster_database = TRUE
    cluster_database_instances= 2
    thread = 2
    instance_number = 2
    undo_management = AUTO
    undo_tablespace = UNDOTBS2
    remote_login_passwordfile= EXCLUSIVE
    db_domain =
    dispatchers = (PROTOCOL=TCP) (SERVICE=kumaondbXDB)
    remote_listener = LISTENERS_KUMAONDB
    job_queue_processes = 10
    background_dump_dest = /u01/app/oracle/product/10.2.0/db_1/admin/kumaondb/bdump
    user_dump_dest = /u01/app/oracle/product/10.2.0/db_1/admin/kumaondb/udump
    core_dump_dest = /u01/app/oracle/product/10.2.0/db_1/admin/kumaondb/cdump
    audit_file_dest = /u01/app/oracle/product/10.2.0/db_1/admin/kumaondb/adump
    db_name = kumaondb
    open_cursors = 300
    pga_aggregate_target = 203423744
    Cluster communication is configured to use the following interface(s) for this instance
    80.0.0.2
    Sat Jun 17 01:13:19 2006
    cluster interconnect IPC version:Oracle UDP/IP
    IPC Vendor 1 proto 2
    PMON started with pid=2, OS id=8584
    DIAG started with pid=3, OS id=8586
    PSP0 started with pid=4, OS id=8588
    LMON started with pid=5, OS id=8590
    LMD0 started with pid=6, OS id=8592
    LMS0 started with pid=7, OS id=8594
    MMAN started with pid=8, OS id=8596
    DBW0 started with pid=9, OS id=8598
    LGWR started with pid=10, OS id=8600
    CKPT started with pid=11, OS id=8603
    SMON started with pid=12, OS id=8627
    RECO started with pid=13, OS id=8633
    CJQ0 started with pid=14, OS id=8635
    MMON started with pid=15, OS id=8637
    MMNL started with pid=16, OS id=8639
    Sat Jun 17 01:13:21 2006
    starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
    starting up 1 shared server(s) ...
    Sat Jun 17 01:13:21 2006
    lmon registered with NM - instance id 2 (internal mem no 1)
    Sat Jun 17 01:13:22 2006
    Reconfiguration started (old inc 0, new inc 2)
    pseudo shared rm latch used
    List of nodes:
    1
    Global Resource Directory frozen
    * allocate domain 0, invalid = TRUE
    Communication channels reestablished
    Master broadcasted resource hash value bitmaps
    Non-local Process blocks cleaned out
    Sat Jun 17 01:13:22 2006
    LMS 0: 0 GCS shadows cancelled, 0 closed
    Set master node info
    Submitted all remote-enqueue requests
    Dwn-cvts replayed, VALBLKs dubious
    All grantable enqueues granted
    Post SMON to start 1st pass IR
    Sat Jun 17 01:13:22 2006
    LMS 0: 0 GCS shadows traversed, 0 replayed
    Sat Jun 17 01:13:22 2006
    Submitted all GCS remote-cache requests
    Post SMON to start 1st pass IR
    Fix write in gcs resources
    Reconfiguration complete
    LCK0 started with pid=19, OS id=8645
    Sat Jun 17 01:13:23 2006
    ALTER DATABASE MOUNT
    Sat Jun 17 01:13:23 2006
    This instance was first to mount
    Setting recovery target incarnation to 2
    Sat Jun 17 01:13:27 2006
    Successful mount of redo thread 2, with mount id 4106896947
    Sat Jun 17 01:13:27 2006
    Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)
    Completed: ALTER DATABASE MOUNT
    Sat Jun 17 01:13:27 2006
    ALTER DATABASE OPEN
    This instance was first to open
    Picked broadcast on commit scheme to generate SCNs
    Sat Jun 17 01:13:27 2006
    Thread 2 opened at log sequence 34
    Current log# 4 seq# 34 mem# 0: /dev/vg_rac/rkumaondb_redo2_2_raw
    Successful open of redo thread 2
    Sat Jun 17 01:13:27 2006
    MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
    Sat Jun 17 01:13:27 2006
    SMON: enabling cache recovery
    Sat Jun 17 01:13:29 2006
    Successfully onlined Undo Tablespace 5.
    Sat Jun 17 01:13:29 2006
    SMON: enabling tx recovery
    Sat Jun 17 01:13:29 2006
    Database Characterset is WE8ISO8859P1
    Sat Jun 17 01:13:29 2006
    Instance recovery: looking for dead threads
    Instance recovery: lock domain invalid but no dead threads
    Sat Jun 17 01:13:30 2006
    replication_dependency_tracking turned off (no async multimaster replication found)
    Starting background process QMNC
    QMNC started with pid=21, OS id=8903
    Sat Jun 17 01:13:32 2006
    Completed: ALTER DATABASE OPEN
    Sat Jun 17 02:36:45 2006
    WARNING: inbound connection timed out (ORA-3136)
    Sat Jun 17 02:41:19 2006
    Memory Notification: Library Cache Object loaded into SGA
    Heap size 2518K exceeds notification threshold (2048K)
    Details in trace file /u01/app/oracle/product/10.2.0/db_1/admin/kumaondb/udump/kumaondb2_ora_24314.trc
    KGL object name :Select uo.object_name, uo.object_type, uo.status, ut.status xstatus, decode( uo.object_type, 'TRIGGER', ut.status, null ) tStatus , uo.created, last_ddl_time from user_triggers ut, user_objects uo where ( ut.trigger_name(+) = uo.object_name ) and uo.object_type in ('JAVA CLASS', 'JAVA SOURCE', 'JAVA RESOURCE')
    Sat Jun 17 02:41:34 2006
    Memory Notification: Library Cache Object loaded into SGA
    Heap size 6223K exceeds notification threshold (2048K)
    Details in trace file /u01/app/oracle/product/10.2.0/db_1/admin/kumaondb/udump/kumaondb2_ora_24314.trc
    KGL object name :select
    round(sum(bytes)/1024/1024,2) Mb ,
    sum(decode(extent_id, 0, bytes, 0)) initial_ex,
    sum(decode(extent_id, 1, bytes, 0)) next_ex ,
    max(extent_id)+1 extents ,
    sum(bytes) ttlsize
    from
    user_extents
    Where segment_type='TABLE'
    and segment_name=:tblname
    Sun Jul 16 16:36:57 2006
    MMNL absent for 2553194 secs; Foregrounds taking over
    Mon Jul 17 04:37:53 2006
    MMNL absent for 43255 secs; Foregrounds taking over
    Sun Jul 16 12:46:00 2006
    Memory Notification: Library Cache Object loaded into SGA
    Heap size 2207K exceeds notification threshold (2048K)
    Details in trace file /u01/app/oracle/product/10.2.0/db_1/admin/kumaondb/udump/kumaondb2_ora_2084.trc
    KGL object name :select /*+ CHOOSE */
    CREATED, LAST_DDL_TIME,
    OBJECT_ID, STATUS, TIMESTAMP
    from ALL_OBJECTS
    where object_name = :objname
    and object_type = :objtype
    and owner = :ownere
    Sun Jul 16 13:01:27 2006
    Memory Notification: Library Cache Object loaded into SGA
    Heap size 2247K exceeds notification threshold (2048K)
    Details in trace file /u01/app/oracle/product/10.2.0/db_1/admin/kumaondb/udump/kumaondb2_ora_23577.trc
    KGL object name :select object_type from all_objects where owner = :OWNER and object_name =:Proc and object_type in ('PACKAGE', 'PROCEDURE', 'FUNCTION')
    Sun Jul 16 13:01:27 2006
    Memory Notification: Library Cache Object loaded into SGA
    Heap size 2901K exceeds notification threshold (2048K)
    Details in trace file /u01/app/oracle/product/10.2.0/db_1/admin/kumaondb/udump/kumaondb2_ora_23577.trc
    KGL object name : select * from all_arguments where object_id = ( select object_id from all_objects where owner = :OWNER and object_name =:Proc and object_type in ('PACKAGE', 'PROCEDURE', 'FUNCTION')) order by Object_Name, Overload, Sequence

  • RAC 10.2.0.5 asm redhat5 64位出现不停自己重启的现象

    手上一个rac的库,(版本10.2.0.5 64bit,操作系统是redhat5 64bit),9月份的时候down机了,查看了alert日志,在alert_asm.log中发现有io failed,在alert_orcl.log中发现有ORA-00204: error in reading (block 35, # blocks 1) of control file的报错.同事重启之后,能恢复过来,但不久之后在alert日志中又会发现,诸如:
    Reconfiguration started (old inc 16, new inc 17)
    List of nodes:
    0
    Global Resource Directory frozen
    Communication channels reestablished
    Master broadcasted resource hash value bitmaps
    Non-local Process blocks cleaned out
    之类的提示,经过观察实例会不停的自己down掉再重启..
    现在拿到手上想处理一下,但不知道该怎么下手,请各位指点,谢谢..
    2个节点的alert_asm.log如下:
    asm1:
    NOTE: ASMB process exiting due to lack of ASM file activity for 12 seconds
    Wed Sep 12 13:13:25 CST 2012
    WARNING: IO Failed. au:43 diskname:ORCL:VOL1
         rq:0x2ab86feb6f88 buffer:0x627ed000 au_offset(bytes):720896 iosz:4096 operation:1
         status:2
    NOTE: cache initiating offline of disk 0 group 1
    WARNING: process 6933 initiating offline of disk 0.3915955288 (VOL1) with mask 0x3 in group 1
    WARNING: Disk 0 in group 1 in mode: 0x7,state: 0x2 will be taken offline
    NOTE: PST update: grp = 1, dsk = 0, mode = 0x6
    Wed Sep 12 13:13:25 CST 2012
    ERROR: too many offline disks in PST (grp 1)
    Wed Sep 12 13:13:25 CST 2012
    ERROR: PST-initiated MANDATORY DISMOUNT of group DATA
    Wed Sep 12 13:13:25 CST 2012
    WARNING: Disk 0 in group 1 in mode: 0x7,state: 0x2 was taken offline
    Wed Sep 12 13:13:25 CST 2012
    NOTE: halting all I/Os to diskgroup DATA
    NOTE: active pin found: 0x0x65faf748
    NOTE: active pin found: 0x0x65faf8a8
    Wed Sep 12 13:13:26 CST 2012
    NOTE: cache dismounting group 1/0xB8984CA8 (DATA)
    Wed Sep 12 13:13:27 CST 2012
    kjbdomdet send to node 1
    detach from dom 1, sending detach message to node 1
    Wed Sep 12 13:13:27 CST 2012
    Dirty detach reconfiguration started (old inc 16, new inc 16)
    List of nodes:
    0 1
    Global Resource Directory partially frozen for dirty detach
    * dirty detach - domain 1 invalid = TRUE
    116 GCS resources traversed, 0 cancelled
    6104 GCS resources on freelist, 6124 on array, 6124 allocated
    Dirty Detach Reconfiguration complete
    Wed Sep 12 13:13:27 CST 2012
    WARNING: dirty detached from domain 1
    Wed Sep 12 13:13:27 CST 2012
    NOTE: PST enabling heartbeating (grp 1)
    Wed Sep 12 13:13:27 CST 2012
    SUCCESS: diskgroup DATA was dismounted
    Wed Sep 12 13:13:27 CST 2012
    WARNING: PST-initiated MANDATORY DISMOUNT of group DATA not performed - group not mounted
    Wed Sep 12 13:13:27 CST 2012
    Errors in file /u01/app/oracle/admin/+ASM/bdump/+asm1_b001_7494.trc:
    ORA-15001: diskgroup "DATA" does not exist or is not mounted
    Wed Sep 12 13:13:28 CST 2012
    freeing rdom 1
    Received dirty detach msg from node 1 for dom 1
    Wed Sep 12 14:53:41 CST 2012
    Reconfiguration started (old inc 16, new inc 17)
    List of nodes:
    0
    Global Resource Directory frozen
    Communication channels reestablished
    Master broadcasted resource hash value bitmaps
    Non-local Process blocks cleaned out
    Wed Sep 12 14:53:41 CST 2012
    LMS 0: 0 GCS shadows cancelled, 0 closed
    Set master node info
    Submitted all remote-enqueue requests
    Dwn-cvts replayed, VALBLKs dubious
    All grantable enqueues granted
    Post SMON to start 1st pass IR
    Wed Sep 12 14:53:41 CST 2012
    LMS 0: 0 GCS shadows traversed, 0 replayed
    Wed Sep 12 14:53:41 CST 2012
    Submitted all GCS remote-cache requests
    Fix write in gcs resources
    Reconfiguration complete
    Wed Sep 12 14:53:51 CST 2012
    Shutting down instance (abort)
    License high water mark = 4
    Instance terminated by USER, pid = 14969
    Wed Sep 12 14:56:38 CST 2012
    Starting ORACLE instance (normal)
    LICENSE_MAX_SESSION = 0
    LICENSE_SESSIONS_WARNING = 0
    Interface type 1 eth1 10.185.3.0 configured from OCR for use as a cluster interconnect
    Interface type 1 eth0 10.185.3.0 configured from OCR for use as a public interface
    Picked latch-free SCN scheme 3
    Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/db_1/dbs/arch
    Autotune of undo retention is turned off.
    LICENSE_MAX_USERS = 0
    SYS auditing is disabled
    ksdpec: called for event 13740 prior to event group initialization
    Starting up ORACLE RDBMS Version: 10.2.0.5.0.
    System parameters with non-default values:
    large_pool_size = 12582912
    instance_type = asm
    cluster_database = TRUE
    instance_number = 1
    remote_login_passwordfile= EXCLUSIVE
    background_dump_dest = /u01/app/oracle/admin/+ASM/bdump
    user_dump_dest = /u01/app/oracle/admin/+ASM/udump
    core_dump_dest = /u01/app/oracle/admin/+ASM/cdump
    asm_diskstring = ORCL:VOL*
    asm_diskgroups = DATA
    Cluster communication is configured to use the following interface(s) for this instance
    10.185.3.77
    Wed Sep 12 14:56:39 CST 2012
    cluster interconnect IPC version:Oracle UDP/IP (generic)
    IPC Vendor 1 proto 2
    PMON started with pid=2, OS id=7327
    LMON started with pid=5, OS id=7333
    PSP0 started with pid=4, OS id=7331
    DIAG started with pid=3, OS id=7329
    LMD0 started with pid=6, OS id=7335
    LMS0 started with pid=7, OS id=7337
    MMAN started with pid=8, OS id=7341
    DBW0 started with pid=9, OS id=7343
    LGWR started with pid=10, OS id=7345
    CKPT started with pid=11, OS id=7347
    SMON started with pid=12, OS id=7349
    RBAL started with pid=13, OS id=7351
    GMON started with pid=14, OS id=7353
    Wed Sep 12 14:56:40 CST 2012
    lmon registered with NM - instance id 1 (internal mem no 0)
    Wed Sep 12 14:56:40 CST 2012
    Reconfiguration started (old inc 0, new inc 2)
    ASM instance
    List of nodes:
    0 1
    Global Resource Directory frozen
    Communication channels reestablished
    * allocate domain 1, invalid = TRUE
    * domain 1 valid = 1 according to instance 1
    Wed Sep 12 14:56:40 CST 2012
    Master broadcasted resource hash value bitmaps
    Non-local Process blocks cleaned out
    Wed Sep 12 14:56:40 CST 2012
    LMS 0: 0 GCS shadows cancelled, 0 closed
    Set master node info
    Submitted all remote-enqueue requests
    Dwn-cvts replayed, VALBLKs dubious
    All grantable enqueues granted
    Wed Sep 12 14:56:40 CST 2012
    LMS 0: 0 GCS shadows traversed, 0 replayed
    Wed Sep 12 14:56:40 CST 2012
    Submitted all GCS remote-cache requests
    Post SMON to start 1st pass IR
    Fix write in gcs resources
    Reconfiguration complete
    LCK0 started with pid=15, OS id=7360
    Wed Sep 12 14:56:41 CST 2012
    SQL> ALTER DISKGROUP ALL MOUNT
    Wed Sep 12 14:56:41 CST 2012
    NOTE: cache registered group DATA number=1 incarn=0xdd6857ac
    Wed Sep 12 14:56:41 CST 2012
    Loaded ASM Library - Generic Linux, version 2.0.4 (KABI_V2) library for asmlib interface
    Wed Sep 12 14:56:41 CST 2012
    NOTE: Hbeat: instance not first (grp 1)
    NOTE: cache opening disk 0 of grp 1: VOL1 label:VOL1
    Wed Sep 12 14:56:41 CST 2012
    NOTE: F1X0 found on disk 0 fcn 0.0
    NOTE: cache mounting (not first) group 1/0xDD6857AC (DATA)
    Wed Sep 12 14:56:41 CST 2012
    kjbdomatt send to node 1
    Wed Sep 12 14:56:42 CST 2012
    NOTE: attached to recovery domain 1
    Wed Sep 12 14:56:42 CST 2012
    NOTE: LGWR attempting to mount thread 2 for disk group 1
    NOTE: LGWR mounted thread 2 for disk group 1
    NOTE: opening chunk 2 at fcn 0.131392 ABA
    NOTE: seq=41 blk=1148
    Wed Sep 12 14:56:42 CST 2012
    NOTE: cache mounting group 1/0xDD6857AC (DATA) succeeded
    SUCCESS: diskgroup DATA was mounted
    Wed Sep 12 14:56:43 CST 2012
    NOTE: recovering COD for group 1/0xdd6857ac (DATA)
    SUCCESS: completed COD recovery for group 1/0xdd6857ac (DATA)
    Wed Sep 12 14:56:45 CST 2012
    Starting background process ASMB
    ASMB started with pid=17, OS id=7454
    Wed Sep 12 14:56:55 CST 2012
    NOTE: ASMB process exiting due to lack of ASM file activity for 12 seconds
    asm2:
    NOTE: ASMB process exiting due to lack of ASM file activity for 12 seconds
    Received dirty detach msg from node 0 for dom 1
    Wed Sep 12 13:13:30 CST 2012
    Dirty detach reconfiguration started (old inc 16, new inc 16)
    List of nodes:
    0 1
    Global Resource Directory partially frozen for dirty detach
    * dirty detach - domain 1 invalid = TRUE
    14 GCS resources traversed, 0 cancelled
    6104 GCS resources on freelist, 6124 on array, 6124 allocated
    99 GCS shadows traversed, 0 replayed
    Dirty Detach Reconfiguration complete
    Wed Sep 12 13:13:30 CST 2012
    NOTE: SMON starting instance recovery for group 1 (mounted)
    Wed Sep 12 13:13:30 CST 2012
    WARNING: IO Failed. au:0 diskname:ORCL:VOL1
         rq:0x2b88463fb990 buffer:0x2b884670ca00 au_offset(bytes):0 iosz:4096 operation:0
         status:2
    WARNING: IO Failed. au:0 diskname:ORCL:VOL1
         rq:0x2b88463fb990 buffer:0x2b884670ca00 au_offset(bytes):0 iosz:4096 operation:0
         status:2
    WARNING: IO Failed. au:4 diskname:ORCL:VOL1
         rq:0xe4372e0 buffer:0x6045f000 au_offset(bytes):0 iosz:4096 operation:0
         status:2
    WARNING: cache failed to read gn 1 fn 3 blk 0 count 1 from disk 0
    ERROR: cache failed to read fn=3 blk=0 from disk(s): 0
    ORA-15081: failed to submit an I/O operation to a disk
    NOTE: cache initiating offline of disk 0 group 1
    WARNING: process 6999 initiating offline of disk 0.3915955111 (VOL1) with mask 0x3 in group 1
    NOTE: PST update: grp = 1, dsk = 0, mode = 0x6
    Wed Sep 12 13:13:30 CST 2012
    ERROR: too many offline disks in PST (grp 1)
    Wed Sep 12 13:13:30 CST 2012
    ERROR: PST-initiated MANDATORY DISMOUNT of group DATA
    Wed Sep 12 13:13:30 CST 2012
    WARNING: Disk 0 in group 1 in mode: 0x7,state: 0x2 was taken offline
    Wed Sep 12 13:13:30 CST 2012
    NOTE: halting all I/Os to diskgroup DATA
    NOTE: active pin found: 0x0x65faf748
    Wed Sep 12 13:13:30 CST 2012
    Abort recovery for domain 1
    Wed Sep 12 13:13:30 CST 2012
    NOTE: cache dismounting group 1/0xB8984B57 (DATA)
    Wed Sep 12 13:13:31 CST 2012
    kjbdomdet send to node 0
    detach from dom 1, sending detach message to node 0
    Wed Sep 12 13:13:31 CST 2012
    Dirty detach reconfiguration started (old inc 16, new inc 16)
    List of nodes:
    0 1
    Global Resource Directory partially frozen for dirty detach
    * dirty detach - domain 1 invalid = TRUE
    99 GCS resources traversed, 0 cancelled
    6104 GCS resources on freelist, 6124 on array, 6124 allocated
    Dirty Detach Reconfiguration complete
    Wed Sep 12 13:13:31 CST 2012
    freeing rdom 1
    Wed Sep 12 13:13:31 CST 2012
    WARNING: dirty detached from domain 1
    Wed Sep 12 13:13:31 CST 2012
    SUCCESS: diskgroup DATA was dismounted
    Wed Sep 12 13:13:31 CST 2012
    WARNING: PST-initiated MANDATORY DISMOUNT of group DATA not performed - group not mounted
    Wed Sep 12 13:13:31 CST 2012
    Errors in file /u01/app/oracle/admin/+ASM/bdump/+asm2_b001_17967.trc:
    ORA-15001: diskgroup "DATA" does not exist or is not mounted
    Wed Sep 12 14:53:43 CST 2012
    Shutting down instance (abort)
    License high water mark = 4
    Instance terminated by USER, pid = 25013
    Wed Sep 12 14:56:02 CST 2012
    Starting ORACLE instance (normal)
    LICENSE_MAX_SESSION = 0
    LICENSE_SESSIONS_WARNING = 0
    Interface type 1 eth1 10.185.3.0 configured from OCR for use as a cluster interconnect
    Interface type 1 eth0 10.185.3.0 configured from OCR for use as a public interface
    Picked latch-free SCN scheme 3
    Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/db_1/dbs/arch
    Autotune of undo retention is turned off.
    LICENSE_MAX_USERS = 0
    SYS auditing is disabled
    ksdpec: called for event 13740 prior to event group initialization
    Starting up ORACLE RDBMS Version: 10.2.0.5.0.
    System parameters with non-default values:
    large_pool_size = 12582912
    instance_type = asm
    cluster_database = TRUE
    instance_number = 2
    remote_login_passwordfile= EXCLUSIVE
    background_dump_dest = /u01/app/oracle/admin/+ASM/bdump
    user_dump_dest = /u01/app/oracle/admin/+ASM/udump
    core_dump_dest = /u01/app/oracle/admin/+ASM/cdump
    asm_diskstring = ORCL:VOL*
    asm_diskgroups = DATA
    Cluster communication is configured to use the following interface(s) for this instance
    10.185.3.79
    Wed Sep 12 14:56:03 CST 2012
    cluster interconnect IPC version:Oracle UDP/IP (generic)
    IPC Vendor 1 proto 2
    LMON started with pid=5, OS id=7171
    PSP0 started with pid=4, OS id=7169
    DIAG started with pid=3, OS id=7167
    PMON started with pid=2, OS id=7165
    LMD0 started with pid=6, OS id=7173
    LMS0 started with pid=7, OS id=7175
    MMAN started with pid=8, OS id=7179
    DBW0 started with pid=9, OS id=7181
    LGWR started with pid=10, OS id=7183
    CKPT started with pid=11, OS id=7185
    SMON started with pid=12, OS id=7187
    RBAL started with pid=13, OS id=7189
    GMON started with pid=14, OS id=7192
    Wed Sep 12 14:56:04 CST 2012
    lmon registered with NM - instance id 2 (internal mem no 1)
    Wed Sep 12 14:56:04 CST 2012
    Reconfiguration started (old inc 0, new inc 1)
    ASM instance
    List of nodes:
    1
    Global Resource Directory frozen
    Communication channels reestablished
    Master broadcasted resource hash value bitmaps
    Non-local Process blocks cleaned out
    Wed Sep 12 14:56:04 CST 2012
    LMS 0: 0 GCS shadows cancelled, 0 closed
    Set master node info
    Submitted all remote-enqueue requests
    Dwn-cvts replayed, VALBLKs dubious
    All grantable enqueues granted
    Post SMON to start 1st pass IR
    Wed Sep 12 14:56:04 CST 2012
    LMS 0: 0 GCS shadows traversed, 0 replayed
    Wed Sep 12 14:56:04 CST 2012
    Submitted all GCS remote-cache requests
    Fix write in gcs resources
    Reconfiguration complete
    LCK0 started with pid=15, OS id=7198
    Wed Sep 12 14:56:05 CST 2012
    SQL> ALTER DISKGROUP ALL MOUNT
    Wed Sep 12 14:56:05 CST 2012
    NOTE: cache registered group DATA number=1 incarn=0xdd684d18
    Wed Sep 12 14:56:05 CST 2012
    Loaded ASM Library - Generic Linux, version 2.0.4 (KABI_V2) library for asmlib interface
    Wed Sep 12 14:56:05 CST 2012
    NOTE: Hbeat: instance first (grp 1)
    Wed Sep 12 14:56:10 CST 2012
    NOTE: start heartbeating (grp 1)
    NOTE: cache opening disk 0 of grp 1: VOL1 label:VOL1
    Wed Sep 12 14:56:10 CST 2012
    NOTE: F1X0 found on disk 0 fcn 0.0
    NOTE: cache mounting (first) group 1/0xDD684D18 (DATA)
    * allocate domain 1, invalid = TRUE
    Wed Sep 12 14:56:10 CST 2012
    NOTE: attached to recovery domain 1
    Wed Sep 12 14:56:10 CST 2012
    NOTE: starting recovery of thread=1 ckpt=40.10147 group=1
    NOTE: starting recovery of thread=2 ckpt=40.1147 group=1
    NOTE: advancing ckpt for thread=2 ckpt=40.1147
    NOTE: advancing ckpt for thread=1 ckpt=40.10159
    NOTE: cache recovered group 1 to fcn 0.227416
    Wed Sep 12 14:56:10 CST 2012
    NOTE: LGWR attempting to mount thread 1 for disk group 1
    NOTE: LGWR mounted thread 1 for disk group 1
    NOTE: opening chunk 1 at fcn 0.227416 ABA
    NOTE: seq=41 blk=10160
    Wed Sep 12 14:56:10 CST 2012
    NOTE: cache mounting group 1/0xDD684D18 (DATA) succeeded
    SUCCESS: diskgroup DATA was mounted
    Wed Sep 12 14:56:13 CST 2012
    NOTE: recovering COD for group 1/0xdd684d18 (DATA)
    SUCCESS: completed COD recovery for group 1/0xdd684d18 (DATA)
    Wed Sep 12 14:56:16 CST 2012
    Starting background process ASMB
    ASMB started with pid=17, OS id=7358
    Wed Sep 12 14:56:26 CST 2012
    NOTE: ASMB process exiting due to lack of ASM file activity for 9 seconds
    Wed Sep 12 14:56:40 CST 2012
    Reconfiguration started (old inc 1, new inc 2)
    List of nodes:
    0 1
    Global Resource Directory frozen
    Communication channels reestablished
    * domain 1 valid = 1 according to instance 0
    Wed Sep 12 14:56:40 CST 2012
    Master broadcasted resource hash value bitmaps
    Non-local Process blocks cleaned out
    Wed Sep 12 14:56:40 CST 2012
    LMS 0: 0 GCS shadows cancelled, 0 closed
    Set master node info
    Submitted all remote-enqueue requests
    Dwn-cvts replayed, VALBLKs dubious
    All grantable enqueues granted
    Wed Sep 12 14:56:40 CST 2012
    LMS 0: 98 GCS shadows traversed, 0 replayed
    Wed Sep 12 14:56:40 CST 2012
    Submitted all GCS remote-cache requests
    Post SMON to start 1st pass IR
    Fix write in gcs resources
    Reconfiguration complete
    帖子经 suredandan编辑过

    2个节点的alert_orcl.log如下:
    实例1:
    Wed Sep 12 13:13:25 CST 2012
    NOTE:Waiting for all pending writes to complete before de-registering: grpnum 1
    Wed Sep 12 13:13:25 CST 2012
    Errors in file /u01/app/oracle/admin/orcl/bdump/orcl1_lmon_32575.trc:
    ORA-00202: control file: '+DATA/orcl/controlfile/current.260.789223249'
    ORA-15078: ASM diskgroup was forcibly dismounted
    Wed Sep 12 13:13:25 CST 2012
    Errors in file /u01/app/oracle/admin/orcl/bdump/orcl1_lmon_32575.trc:
    ORA-00204: error in reading (block 35, # blocks 1) of control file
    ORA-00202: control file: '+DATA/orcl/controlfile/current.260.789223249'
    ORA-15078: ASM diskgroup was forcibly dismounted
    Wed Sep 12 13:13:25 CST 2012
    LMON: terminating instance due to error 204
    Wed Sep 12 13:13:26 CST 2012
    System state dump is made for local instance
    System State dumped to trace file /u01/app/oracle/admin/orcl/bdump/orcl1_diag_32571.trc
    Wed Sep 12 13:13:26 CST 2012
    Shutting down instance (abort)
    License high water mark = 30
    Wed Sep 12 13:13:27 CST 2012
    Trace dumping is performing id=[cdmp_20120912131326]
    Wed Sep 12 13:13:27 CST 2012
    Instance terminated by LMON, pid = 32575
    Wed Sep 12 13:13:31 CST 2012
    Instance terminated by USER, pid = 7518
    Wed Sep 12 14:56:46 CST 2012
    Starting ORACLE instance (normal)
    LICENSE_MAX_SESSION = 0
    LICENSE_SESSIONS_WARNING = 0
    Interface type 1 eth1 10.185.3.0 configured from OCR for use as a cluster interconnect
    Interface type 1 eth0 10.185.3.0 configured from OCR for use as a public interface
    Picked latch-free SCN scheme 3
    Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/db_1/dbs/arch
    LICENSE_MAX_USERS = 0
    SYS auditing is disabled
    ksdpec: called for event 13740 prior to event group initialization
    Starting up ORACLE RDBMS Version: 10.2.0.5.0.
    System parameters with non-default values:
    processes = 500
    sessions = 555
    sga_max_size = 10485760000
    __shared_pool_size = 1811939328
    __large_pool_size = 16777216
    __java_pool_size = 33554432
    __streams_pool_size = 0
    spfile = +DATA/orcl/spfileorcl.ora
    sga_target = 10485760000
    control_files = +DATA/orcl/controlfile/current.260.789223249
    db_block_size = 8192
    __db_cache_size = 8489271296
    db_keep_cache_size = 117440512
    compatible = 10.2.0.5.0
    db_file_multiblock_read_count= 16
    cluster_database = TRUE
    cluster_database_instances= 2
    db_create_file_dest = +DATA
    thread = 1
    instance_number = 1
    undo_management = AUTO
    undo_tablespace = UNDOTBS1
    remote_login_passwordfile= EXCLUSIVE
    db_domain =
    dispatchers =
    local_listener = (ADDRESS = (PROTOCOL = TCP)(HOST = 10.185.3.80)(PORT = 1521))
    remote_listener = LISTENERS_ORCL
    job_queue_processes = 10
    background_dump_dest = /u01/app/oracle/admin/orcl/bdump
    user_dump_dest = /u01/app/oracle/admin/orcl/udump
    core_dump_dest = /u01/app/oracle/admin/orcl/cdump
    audit_file_dest = /u01/app/oracle/admin/orcl/adump
    db_name = orcl
    open_cursors = 300
    pga_aggregate_target = 770703360
    Cluster communication is configured to use the following interface(s) for this instance
    10.185.3.77
    Wed Sep 12 14:56:46 CST 2012
    cluster interconnect IPC version:Oracle UDP/IP (generic)
    IPC Vendor 1 proto 2
    PMON started with pid=2, OS id=7468
    DIAG started with pid=3, OS id=7470
    PSP0 started with pid=4, OS id=7479
    LMON started with pid=5, OS id=7481
    LMD0 started with pid=6, OS id=7484
    LMS0 started with pid=7, OS id=7486
    LMS1 started with pid=8, OS id=7490
    MMAN started with pid=9, OS id=7494
    DBW0 started with pid=10, OS id=7496
    LGWR started with pid=11, OS id=7498
    CKPT started with pid=12, OS id=7500
    SMON started with pid=13, OS id=7502
    RECO started with pid=14, OS id=7504
    CJQ0 started with pid=15, OS id=7506
    MMON started with pid=16, OS id=7508
    MMNL started with pid=17, OS id=7510
    Wed Sep 12 14:56:48 CST 2012
    lmon registered with NM - instance id 1 (internal mem no 0)
    Wed Sep 12 14:56:49 CST 2012
    Reconfiguration started (old inc 0, new inc 4)
    List of nodes:
    0 1
    Global Resource Directory frozen
    * allocate domain 0, invalid = TRUE
    Communication channels reestablished
    * domain 0 valid according to instance 1
    * domain 0 valid = 1 according to instance 1
    Wed Sep 12 14:56:49 CST 2012
    Master broadcasted resource hash value bitmaps
    Non-local Process blocks cleaned out
    Wed Sep 12 14:56:49 CST 2012
    LMS 0: 0 GCS shadows cancelled, 0 closed
    Wed Sep 12 14:56:49 CST 2012
    LMS 1: 0 GCS shadows cancelled, 0 closed
    Set master node info
    Submitted all remote-enqueue requests
    Dwn-cvts replayed, VALBLKs dubious
    All grantable enqueues granted
    Wed Sep 12 14:56:49 CST 2012
    LMS 0: 0 GCS shadows traversed, 0 replayed
    Wed Sep 12 14:56:49 CST 2012
    LMS 1: 0 GCS shadows traversed, 0 replayed
    Wed Sep 12 14:56:49 CST 2012
    Submitted all GCS remote-cache requests
    Post SMON to start 1st pass IR
    Fix write in gcs resources
    Reconfiguration complete
    LCK0 started with pid=18, OS id=7546
    Wed Sep 12 14:56:50 CST 2012
    ALTER DATABASE MOUNT
    Wed Sep 12 14:56:50 CST 2012
    Starting background process ASMB
    ASMB started with pid=20, OS id=7552
    Starting background process RBAL
    RBAL started with pid=21, OS id=7556
    Loaded ASM Library - Generic Linux, version 2.0.4 (KABI_V2) library for asmlib interface
    Wed Sep 12 14:56:53 CST 2012
    SUCCESS: diskgroup DATA was mounted
    Wed Sep 12 14:56:57 CST 2012
    Setting recovery target incarnation to 2
    Wed Sep 12 14:56:57 CST 2012
    Successful mount of redo thread 1, with mount id 1321558804
    Wed Sep 12 14:56:57 CST 2012
    Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)
    Completed: ALTER DATABASE MOUNT
    Wed Sep 12 14:56:58 CST 2012
    ALTER DATABASE OPEN
    Picked broadcast on commit scheme to generate SCNs
    Wed Sep 12 14:56:58 CST 2012
    Thread 1 opened at log sequence 354
    Current log# 2 seq# 354 mem# 0: +DATA/orcl/onlinelog/group_2.262.789223251
    Successful open of redo thread 1
    Wed Sep 12 14:56:58 CST 2012
    MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
    Wed Sep 12 14:56:58 CST 2012
    SMON: enabling cache recovery
    Wed Sep 12 14:56:59 CST 2012
    Successfully onlined Undo Tablespace 1.
    Wed Sep 12 14:56:59 CST 2012
    SMON: enabling tx recovery
    Wed Sep 12 14:56:59 CST 2012
    Database Characterset is AL32UTF8
    Opening with internal Resource Manager plan
    replication_dependency_tracking turned off (no async multimaster replication found)
    Starting background process QMNC
    QMNC started with pid=25, OS id=7618
    Wed Sep 12 14:57:00 CST 2012
    Completed: ALTER DATABASE OPEN
    实例2:
    Wed Sep 12 13:13:30 CST 2012
    SUCCESS: diskgroup DATA was dismounted
    SUCCESS: diskgroup DATA was dismounted
    Wed Sep 12 13:13:30 CST 2012
    Errors in file /u01/app/oracle/admin/orcl/bdump/orcl2_lmon_2673.trc:
    ORA-00202: control file: '+DATA/orcl/controlfile/current.260.789223249'
    ORA-15078: ASM diskgroup was forcibly dismounted
    Wed Sep 12 13:13:30 CST 2012
    Errors in file /u01/app/oracle/admin/orcl/bdump/orcl2_ckpt_2691.trc:
    ORA-00206: error in writing (block 4, # blocks 1) of control file
    ORA-00202: control file: '+DATA/orcl/controlfile/current.260.789223249'
    ORA-15078: ASM diskgroup was forcibly dismounted
    Wed Sep 12 13:13:30 CST 2012
    Errors in file /u01/app/oracle/admin/orcl/bdump/orcl2_ckpt_2691.trc:
    ORA-00221: error on write to control file
    ORA-00206: error in writing (block 4, # blocks 1) of control file
    ORA-00202: control file: '+DATA/orcl/controlfile/current.260.789223249'
    ORA-15078: ASM diskgroup was forcibly dismounted
    Wed Sep 12 13:13:30 CST 2012
    CKPT: terminating instance due to error 221
    Wed Sep 12 13:13:30 CST 2012
    Errors in file /u01/app/oracle/admin/orcl/bdump/orcl2_lmon_2673.trc:
    ORA-00204: error in reading (block 35, # blocks 1) of control file
    ORA-00202: control file: '+DATA/orcl/controlfile/current.260.789223249'
    ORA-15078: ASM diskgroup was forcibly dismounted
    Wed Sep 12 13:13:31 CST 2012
    Shutting down instance (abort)
    License high water mark = 29
    Wed Sep 12 13:13:32 CST 2012
    System state dump is made for local instance
    System State dumped to trace file /u01/app/oracle/admin/orcl/bdump/orcl2_diag_2669.trc
    Wed Sep 12 13:13:33 CST 2012
    Trace dumping is performing id=[cdmp_20120912131330]
    Wed Sep 12 13:13:36 CST 2012
    Instance terminated by CKPT, pid = 2691
    Wed Sep 12 13:13:41 CST 2012
    Instance terminated by USER, pid = 18002
    Wed Sep 12 14:56:16 CST 2012
    Starting ORACLE instance (normal)
    LICENSE_MAX_SESSION = 0
    LICENSE_SESSIONS_WARNING = 0
    Interface type 1 eth1 10.185.3.0 configured from OCR for use as a cluster interconnect
    Interface type 1 eth0 10.185.3.0 configured from OCR for use as a public interface
    Picked latch-free SCN scheme 3
    Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/db_1/dbs/arch
    LICENSE_MAX_USERS = 0
    SYS auditing is disabled
    ksdpec: called for event 13740 prior to event group initialization
    Starting up ORACLE RDBMS Version: 10.2.0.5.0.
    System parameters with non-default values:
    processes = 500
    sessions = 555
    sga_max_size = 10485760000
    __shared_pool_size = 1962934272
    __large_pool_size = 16777216
    __java_pool_size = 16777216
    __streams_pool_size = 0
    spfile = +DATA/orcl/spfileorcl.ora
    sga_target = 10485760000
    control_files = +DATA/orcl/controlfile/current.260.789223249
    db_block_size = 8192
    __db_cache_size = 8355053568
    db_keep_cache_size = 117440512
    compatible = 10.2.0.5.0
    db_file_multiblock_read_count= 16
    cluster_database = TRUE
    cluster_database_instances= 2
    db_create_file_dest = +DATA
    thread = 2
    instance_number = 2
    undo_management = AUTO
    undo_tablespace = UNDOTBS2
    remote_login_passwordfile= EXCLUSIVE
    db_domain =
    dispatchers =
    local_listener = (ADDRESS = (PROTOCOL = TCP)(HOST = 10.185.3.81)(PORT = 1521))
    remote_listener = LISTENERS_ORCL
    job_queue_processes = 10
    background_dump_dest = /u01/app/oracle/admin/orcl/bdump
    user_dump_dest = /u01/app/oracle/admin/orcl/udump
    core_dump_dest = /u01/app/oracle/admin/orcl/cdump
    audit_file_dest = /u01/app/oracle/admin/orcl/adump
    db_name = orcl
    open_cursors = 300
    pga_aggregate_target = 770703360
    Cluster communication is configured to use the following interface(s) for this instance
    10.185.3.79
    Wed Sep 12 14:56:16 CST 2012
    cluster interconnect IPC version:Oracle UDP/IP (generic)
    IPC Vendor 1 proto 2
    PMON started with pid=2, OS id=7366
    DIAG started with pid=3, OS id=7368
    PSP0 started with pid=4, OS id=7370
    LMON started with pid=5, OS id=7372
    LMD0 started with pid=6, OS id=7374
    LMS0 started with pid=7, OS id=7376
    LMS1 started with pid=8, OS id=7380
    MMAN started with pid=9, OS id=7384
    DBW0 started with pid=10, OS id=7386
    LGWR started with pid=11, OS id=7388
    CKPT started with pid=12, OS id=7390
    SMON started with pid=13, OS id=7392
    RECO started with pid=14, OS id=7394
    CJQ0 started with pid=15, OS id=7396
    MMON started with pid=16, OS id=7398
    MMNL started with pid=17, OS id=7401
    Wed Sep 12 14:56:19 CST 2012
    lmon registered with NM - instance id 2 (internal mem no 1)
    Wed Sep 12 14:56:19 CST 2012
    Reconfiguration started (old inc 0, new inc 2)
    List of nodes:
    1
    Global Resource Directory frozen
    * allocate domain 0, invalid = TRUE
    Communication channels reestablished
    Master broadcasted resource hash value bitmaps
    Non-local Process blocks cleaned out
    Wed Sep 12 14:56:19 CST 2012
    LMS 0: 0 GCS shadows cancelled, 0 closed
    Wed Sep 12 14:56:19 CST 2012
    LMS 1: 0 GCS shadows cancelled, 0 closed
    Set master node info
    Submitted all remote-enqueue requests
    Dwn-cvts replayed, VALBLKs dubious
    All grantable enqueues granted
    Post SMON to start 1st pass IR
    Wed Sep 12 14:56:19 CST 2012
    LMS 0: 0 GCS shadows traversed, 0 replayed
    Wed Sep 12 14:56:19 CST 2012
    LMS 1: 0 GCS shadows traversed, 0 replayed
    Wed Sep 12 14:56:19 CST 2012
    Submitted all GCS remote-cache requests
    Fix write in gcs resources
    Reconfiguration complete
    LCK0 started with pid=18, OS id=7418
    Wed Sep 12 14:56:20 CST 2012
    ALTER DATABASE MOUNT
    Wed Sep 12 14:56:20 CST 2012
    This instance was first to mount
    Wed Sep 12 14:56:20 CST 2012
    Starting background process ASMB
    ASMB started with pid=20, OS id=7424
    Starting background process RBAL
    RBAL started with pid=21, OS id=7428
    Loaded ASM Library - Generic Linux, version 2.0.4 (KABI_V2) library for asmlib interface
    Wed Sep 12 14:56:23 CST 2012
    SUCCESS: diskgroup DATA was mounted
    Wed Sep 12 14:56:27 CST 2012
    Setting recovery target incarnation to 2
    Wed Sep 12 14:56:27 CST 2012
    Successful mount of redo thread 2, with mount id 1321558804
    Wed Sep 12 14:56:27 CST 2012
    Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)
    Completed: ALTER DATABASE MOUNT
    Wed Sep 12 14:56:27 CST 2012
    ALTER DATABASE OPEN
    This instance was first to open
    Wed Sep 12 14:56:27 CST 2012
    Beginning crash recovery of 2 threads
    parallel recovery started with 7 processes
    Wed Sep 12 14:56:28 CST 2012
    Started redo scan
    Wed Sep 12 14:56:28 CST 2012
    Completed redo scan
    717 redo blocks read, 25 data blocks need recovery
    Wed Sep 12 14:56:28 CST 2012
    Started redo application at
    Thread 1: logseq 353, block 70791
    Thread 2: logseq 401, block 88381
    Wed Sep 12 14:56:28 CST 2012
    Recovery of Online Redo Log: Thread 1 Group 1 Seq 353 Reading mem 0
    Mem# 0: +DATA/orcl/onlinelog/group_1.261.789223251
    Wed Sep 12 14:56:28 CST 2012
    Recovery of Online Redo Log: Thread 2 Group 3 Seq 401 Reading mem 0
    Mem# 0: +DATA/orcl/onlinelog/group_3.265.789223275
    Wed Sep 12 14:56:28 CST 2012
    Completed redo application
    Wed Sep 12 14:56:28 CST 2012
    Completed crash recovery at
    Thread 1: logseq 353, block 71419, scn 18394186
    Thread 2: logseq 401, block 88470, scn 18386240
    25 data blocks read, 25 data blocks written, 717 redo blocks read
    Wed Sep 12 14:56:29 CST 2012
    Thread 1 advanced to log sequence 354 (thread recovery)
    Picked broadcast on commit scheme to generate SCNs
    Wed Sep 12 14:56:29 CST 2012
    Thread 2 advanced to log sequence 402 (thread open)
    Thread 2 opened at log sequence 402
    Current log# 4 seq# 402 mem# 0: +DATA/orcl/onlinelog/group_4.266.789223275
    Successful open of redo thread 2
    Wed Sep 12 14:56:29 CST 2012
    MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
    Wed Sep 12 14:56:29 CST 2012
    SMON: enabling cache recovery
    Wed Sep 12 14:56:30 CST 2012
    Successfully onlined Undo Tablespace 5.
    Wed Sep 12 14:56:30 CST 2012
    SMON: enabling tx recovery
    Wed Sep 12 14:56:30 CST 2012
    Database Characterset is AL32UTF8
    Opening with internal Resource Manager plan
    replication_dependency_tracking turned off (no async multimaster replication found)
    Starting background process QMNC
    QMNC started with pid=30, OS id=7508
    Wed Sep 12 14:56:32 CST 2012
    Completed: ALTER DATABASE OPEN
    Wed Sep 12 14:56:49 CST 2012
    Reconfiguration started (old inc 2, new inc 4)
    List of nodes:
    0 1
    Global Resource Directory frozen
    Communication channels reestablished
    * domain 0 valid = 1 according to instance 0
    Wed Sep 12 14:56:49 CST 2012
    Master broadcasted resource hash value bitmaps
    Non-local Process blocks cleaned out
    Wed Sep 12 14:56:49 CST 2012
    LMS 0: 0 GCS shadows cancelled, 0 closed
    Wed Sep 12 14:56:49 CST 2012
    LMS 1: 0 GCS shadows cancelled, 0 closed
    Set master node info
    Submitted all remote-enqueue requests
    Dwn-cvts replayed, VALBLKs dubious
    All grantable enqueues granted
    Wed Sep 12 14:56:49 CST 2012
    LMS 1: 2192 GCS shadows traversed, 1025 replayed
    Wed Sep 12 14:56:49 CST 2012
    LMS 0: 2135 GCS shadows traversed, 1082 replayed
    Wed Sep 12 14:56:49 CST 2012
    Submitted all GCS remote-cache requests
    Fix write in gcs resources
    Reconfiguration complete

  • 오라클 10g 장애 관련 alert log 파일좀 올려봅니다.

    Sun Dec 10 03:11:05 2006
    Process J000 died, see its trace file
    Sun Dec 10 03:11:05 2006
    kkjcre1p: unable to spawn jobq slave process
    Sun Dec 10 03:11:05 2006
    Errors in file c:\oracle\product\10.2.0\admin\golf\bdump\golf1_cjq0_4704.trc:
    Process J000 died, see its trace file
    Sun Dec 10 03:11:11 2006
    kkjcre1p: unable to spawn jobq slave process
    Sun Dec 10 03:11:11 2006
    Errors in file c:\oracle\product\10.2.0\admin\golf\bdump\golf1_cjq0_4704.trc:
    Sun Dec 10 03:12:12 2006
    Process J000 died, see its trace file
    Sun Dec 10 03:12:12 2006
    kkjcre1p: unable to spawn jobq slave process
    Sun Dec 10 03:12:12 2006
    Errors in file c:\oracle\product\10.2.0\admin\golf\bdump\golf1_cjq0_4704.trc:
    Sun Dec 10 03:13:18 2006
    Process J000 died, see its trace file
    Sun Dec 10 03:13:18 2006
    kkjcre1p: unable to spawn jobq slave process
    Sun Dec 10 03:13:18 2006
    Errors in file c:\oracle\product\10.2.0\admin\golf\bdump\golf1_cjq0_4704.trc:
    Sun Dec 10 03:20:58 2006
    Process PZ99 died, see its trace file
    Process PZ99 died, see its trace file
    Sun Dec 10 03:31:48 2006
    Process PZ99 died, see its trace file
    Process PZ99 died, see its trace file
    Sun Dec 10 03:42:38 2006
    Process PZ99 died, see its trace file
    Process PZ99 died, see its trace file
    Sun Dec 10 03:48:04 2006
    Process J000 died, see its trace file
    Sun Dec 10 03:48:04 2006
    kkjcre1p: unable to spawn jobq slave process
    Sun Dec 10 03:48:04 2006
    Errors in file c:\oracle\product\10.2.0\admin\golf\bdump\golf1_cjq0_4704.trc:
    Sun Dec 10 03:48:19 2006
    Process J000 died, see its trace file
    Sun Dec 10 03:48:19 2006
    kkjcre1p: unable to spawn jobq slave process
    Sun Dec 10 03:48:19 2006
    Errors in file c:\oracle\product\10.2.0\admin\golf\bdump\golf1_cjq0_4704.trc:
    Sun Dec 10 06:52:34 2006
    Reconfiguration started (old inc 4, new inc 6)
    List of nodes:
    0
    Sun Dec 10 06:52:35 2006
    Process q002 died, see its trace file
    Sun Dec 10 06:52:35 2006
    ksvcreate: Process(q002) creation failed
    Dump file c:\oracle\product\10.2.0/admin/golf/bdump\alert_golf1.log
    Sun Dec 10 06:59:01 2006
    ORACLE V10.2.0.1.0 - Production vsnsta=0
    vsnsql=14 vsnxtr=3
    Windows Server 2003 Version V5.2 Service Pack 1
    CPU : 4 - type 586, 2 Physical Cores
    Process Affinity : 0x00000000
    Memory (Avail/Total): Ph:2877M/3583M, Ph+PgF:4816M/5475M, VA:1937M/2047M
    Sun Dec 10 06:59:01 2006
    Starting ORACLE instance (normal)
    LICENSE_MAX_SESSION = 0
    LICENSE_SESSIONS_WARNING = 0
    Picked latch-free SCN scheme 2
    Autotune of undo retention is turned on.
    LICENSE_MAX_USERS = 0
    SYS auditing is disabled
    ksdpec: called for event 13740 prior to event group initialization
    Starting up ORACLE RDBMS Version: 10.2.0.1.0.
    System parameters with non-default values:
    processes = 150
    __shared_pool_size = 184549376
    __large_pool_size = 4194304
    __java_pool_size = 8388608
    __streams_pool_size = 0
    spfile = +DATA/golf/spfilegolf.ora
    nls_language = KOREAN
    nls_territory = KOREA
    cluster_interconnects = 10.10.10.1:192.168.200.3
    sga_target = 1073741824
    control_files = DATA/golf/controlfile/current.260.573740937, BACKUP/golf/controlfile/current.256.573740937
    db_block_size = 8192
    __db_cache_size = 868220928
    compatible = 10.2.0.1.0
    log_archive_dest_1 = LOCATION=+BACKUP/GOLF/
    log_archive_format = ARC%S_%R.%T
    db_file_multiblock_read_count= 16
    cluster_database = TRUE
    cluster_database_instances= 2
    db_create_file_dest = +DATA
    db_recovery_file_dest = +BACKUP
    db_recovery_file_dest_size= 128849018880
    thread = 1
    instance_number = 1
    undo_management = AUTO
    undo_tablespace = UNDOTBS1
    remote_login_passwordfile= EXCLUSIVE
    db_domain =
    dispatchers = (PROTOCOL=TCP) (SERVICE=GOLFXDB)
    local_listener = GOLF1
    remote_listener =
    job_queue_processes = 10
    audit_file_dest = C:\ORACLE\PRODUCT\10.2.0\ADMIN\GOLF\ADUMP
    background_dump_dest = C:\ORACLE\PRODUCT\10.2.0\ADMIN\GOLF\BDUMP
    user_dump_dest = C:\ORACLE\PRODUCT\10.2.0\ADMIN\GOLF\UDUMP
    core_dump_dest = C:\ORACLE\PRODUCT\10.2.0\ADMIN\GOLF\CDUMP
    db_name = GOLF
    open_cursors = 300
    pga_aggregate_target = 524288000
    Cluster communication is configured to use the following interface(s) for this instance
    10.10.10.1
    Sun Dec 10 06:59:01 2006
    cluster interconnect IPC version:Oracle 9i Winsock2 TCP/IP IPC
    IPC Vendor 0 proto 0
    Version 0.0
    PMON started with pid=2, OS id=4100
    DIAG started with pid=3, OS id=4112
    PSP0 started with pid=4, OS id=4120
    LMON started with pid=5, OS id=4144
    LMD0 started with pid=6, OS id=4184
    LMS0 started with pid=7, OS id=4192
    LMS1 started with pid=8, OS id=4212
    MMAN started with pid=9, OS id=4216
    DBW0 started with pid=10, OS id=4224
    LGWR started with pid=11, OS id=4208
    CKPT started with pid=12, OS id=4236
    SMON started with pid=13, OS id=4244
    RECO started with pid=14, OS id=4252
    CJQ0 started with pid=15, OS id=4264
    MMON started with pid=16, OS id=4280
    MMNL started with pid=17, OS id=4288
    Sun Dec 10 06:59:01 2006
    starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
    starting up 1 shared server(s) ...
    Sun Dec 10 06:59:01 2006
    lmon registered with NM - instance id 1 (internal mem no 0)
    Sun Dec 10 06:59:02 2006
    Reconfiguration started (old inc 0, new inc 4)
    List of nodes:
    0 1
    Global Resource Directory frozen
    * allocate domain 0, invalid = TRUE
    Communication channels reestablished
    * domain 0 valid according to instance 1
    * domain 0 valid = 1 according to instance 1
    Sun Dec 10 06:59:03 2006
    Master broadcasted resource hash value bitmaps
    Non-local Process blocks cleaned out
    Sun Dec 10 06:59:03 2006
    LMS 0: 0 GCS shadows cancelled, 0 closed
    Sun Dec 10 06:59:04 2006
    LMS 1: 0 GCS shadows cancelled, 0 closed
    Set master node info
    Submitted all remote-enqueue requests
    Dwn-cvts replayed, VALBLKs dubious
    All grantable enqueues granted
    Sun Dec 10 06:59:05 2006
    LMS 0: 0 GCS shadows traversed, 0 replayed
    Sun Dec 10 06:59:06 2006
    LMS 1: 0 GCS shadows traversed, 0 replayed
    Sun Dec 10 06:59:06 2006
    Submitted all GCS remote-cache requests
    Post SMON to start 1st pass IR
    Fix write in gcs resources
    Reconfiguration complete
    LCK0 started with pid=20, OS id=5164
    Sun Dec 10 06:59:06 2006
    ALTER DATABASE MOUNT
    Sun Dec 10 06:59:07 2006
    Starting background process ASMB
    ASMB started with pid=22, OS id=4920
    Starting background process RBAL
    RBAL started with pid=23, OS id=5172
    Sun Dec 10 06:59:15 2006
    SUCCESS: diskgroup DATA was mounted
    SUCCESS: diskgroup BACKUP was mounted
    Sun Dec 10 06:59:19 2006
    Setting recovery target incarnation to 2
    Sun Dec 10 06:59:19 2006
    Successful mount of redo thread 1, with mount id 2679476622
    Sun Dec 10 06:59:19 2006
    Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)
    Completed: ALTER DATABASE MOUNT
    Sun Dec 10 06:59:19 2006
    ALTER DATABASE OPEN
    Picked broadcast on commit scheme to generate SCNs
    Sun Dec 10 07:00:21 2006
    LGWR: STARTING ARCH PROCESSES
    ARC0 started with pid=27, OS id=5668
    Sun Dec 10 07:00:21 2006
    ARC0: Archival started
    ARC1 started with pid=28, OS id=5964
    Sun Dec 10 07:00:21 2006
    ARC1: Archival started
    LGWR: STARTING ARCH PROCESSES COMPLETE
    Thread 1 opened at log sequence 1855
    Current log# 2 seq# 1855 mem# 0: +DATA/golf/onlinelog/group_2.262.573740941
    Current log# 2 seq# 1855 mem# 1: +BACKUP/golf/onlinelog/group_2.258.573740941
    Sun Dec 10 07:00:22 2006
    ARC0: STARTING ARCH PROCESSES
    Sun Dec 10 07:00:22 2006
    Successful open of redo thread 1
    Sun Dec 10 07:00:22 2006
    MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
    Sun Dec 10 07:00:23 2006
    ARC1: Becoming the 'no FAL' ARCH
    Sun Dec 10 07:00:23 2006
    ARC1: Becoming the 'no SRL' ARCH
    Sun Dec 10 07:00:23 2006
    ARC2: Archival started
    ARC2 started with pid=29, OS id=5908
    Sun Dec 10 07:00:23 2006
    ARC0: STARTING ARCH PROCESSES COMPLETE
    ARC0: Becoming the heartbeat ARCH
    Sun Dec 10 07:00:24 2006
    SMON: enabling cache recovery
    Sun Dec 10 07:00:24 2006
    Successfully onlined Undo Tablespace 1.
    Sun Dec 10 07:00:24 2006
    SMON: enabling tx recovery
    Sun Dec 10 07:00:24 2006
    Database Characterset is KO16KSC5601
    replication_dependency_tracking turned off (no async multimaster replication found)
    Starting background process QMNC
    QMNC started with pid=30, OS id=6076
    Sun Dec 10 07:00:28 2006
    Completed: ALTER DATABASE OPEN
    Sun Dec 10 07:06:21 2006
    Shutting down archive processes
    Sun Dec 10 07:06:26 2006
    ARCH shutting down
    ARC2: Archival stopped
    db 접속이(client) "tns 리스너 소켓을 찾을수..." 이런 메세지만 나오며
    db 가 설치되어있는 서버에서 sql-plus 접속이 위와같은 메세지로 접속이 되지 않았습니다.
    status는 모든 인스턴스가 정상적으로 작동중이었구요..
    그래서 서버 그냥 off 시켰다가 on 시키니깐 정상으로 돌아왔는데
    불안하네요...이런문제...
    혹시 보시고 답변좀 부탁합니다.

    ALERT 내용만 보면 아래 두개의 증상과 비슷하네요.
    첫번째는 alter system set undo_retention=1800 sid='*' 명령문을 node1으로 지정해서 실행하면 모든 instance 가 행업이 걸리는 현상이 있네요. 명령문을 없애주거나 아니면 모든 sid 별로 다 실행을 해주라고 나와있네요.
    Symptoms
    3 nodes RAC system on 10.1.0.4. The command "alter system set undo_retention=1800 sid='*';" was issued on node1, immeidately all instances hang. User can not logon to instance 2 and 3. Logon to instance 1 is still OK but query hangs.Alert log on node 2 and 3 report:Timed out trying to start process J000.Timed out trying to start process q000.kkjcre1p: unable to spawn jobq slave processErrors in file /opt/oracle/admin/RAC/bdump/rac3_cjq0_4229.trc
    Cause
    This is caused by unpublished Bug.3023661 RAC INSTANCE HANGS WHEN MODIFYING UNDO_RETENTION PARAMETER(or published Bug 4220405 ALTER SYSTEM SET UNDO_RETENTION=<N> HANGS RAC INSTANCES which has been closed as base bug 3023661)Bug 4220405 ALTER SYSTEM SET UNDO_RETENTION=<N> HANGS RAC INSTANCES which has been closed as base bug 3023661)The problem is caused by deadlock between CKPT and PZ99 slave. The internal algorithm to query thecurrent value of undo_retention of each instance and modify it has problem. It does unnecessary gv$ query andlob$ update when spfile is used. It is coding problem. The bug has been fixed in 10.2 and it is not backportable to 10.1.0.x due to code structure change.
    Solution
    Workaround is to modify one instance at a time, specify the detailed SID name for alter system command, eg:alter system set undo_retention=1800 sid='RAC1';alter system set undo_retention=1800 sid='RAC2';alter system set undo_retention=1800 sid='RAC3';......until all instances are modified.

Maybe you are looking for