Second node cannot bring up clusterware stack

Hi forum,
I have a two-node RAC in VirtualBox VMs. After booting the first node (regardless which of the two), the complete Clusterware stack is started successfully. When booting the other node, the Clusterware stack does not come up successfully. When I try to start components manually, there seems to be a circular dependency between CRS/CSSD and ASM.
Configuration:
OS: OEL 5.8, UEK
Grid: 11.2
Database: 11.2.0.1
Any help is appreciated!
Thanks,
Andreas

Hi,
What is the error you have in the OHASD.log and CRSD.log. There are dependencies of the component from 11gR2.
http://levipereira.wordpress.com/2012/01/11/explaining-how-to-store-ocr-voting-disks-and-asm-spfile-on-asm-diskgroup-rac-or-rac-extended/
Anand

Similar Messages

  • 0x800713CF: The owner node cannot run this resource (SQL)

    I have SQL 2012 SP2 service on 2 FC nodes of 2012 R2 servers. Everything were working fine about a year, but now my first Node got mixed up with updates and it is offline now. Problem is, that second Node cannot start SQL services By its own.
    I checked/tried:
    - I checked, that other services related to SQL are online.
    - I chenged Node´s ownership to SQL service
    - Cluster name has full Access rights to OU where all Computer record of this service are in place
    The error I get: 
    0x800713cf:The cluster resource cannot be brought online. The owner node cannot run this resource.

    Thank you both. I´m not sure, how to to check the workload. In FC graphical GUI, both nodes are memebers in owner group, same Place where the owner order can be changed or owner can be disabled. I gone through entire FC management snap-in
    and didn´t find any other places or settings to go through. The article kb313882 Umesh refers, is for Server 2008, and I have 2012 R2. Command
    cluster res <var> name of resource</var> /listowners is not supported (there is no such command). I got first node up and running, and then SQL services were ON again, but if I shutdown first Node, SQL services will go down
    and won´t come up. Disk storage are online and SP level matches.
    Please could you guide me step By step, how to check ownership in 2012 R2?

  • 11gR2 clusterware installation problem on root.sh script on second node

    Hi all,
    I wanna install the *11gR2 RAC* on ORA-Linux 5.5 (x86_64) using VMware server but on the second node i get two "*failed*" at the end of root.sh script.
    After that i try to install DB but ı can see only one node.What is the problem...
    I will send the output, ı need your help.
    Thank you all for helping..
    Hosts file:(we have no ping problem )
    [root@rac2 ~]# cat /etc/hosts
    # Do not remove the following line, or various programs
    # that require network functionality will fail.
    127.0.0.1       localhost.localdomain   localhost
    # Public
    192.168.2.101   rac1.localdomain        rac1
    192.168.2.102   rac2.localdomain        rac2
    # Private
    192.168.0.101   rac1-priv.localdomain   rac1-priv
    192.168.0.102   rac2-priv.localdomain   rac2-priv
    # Virtual
    192.168.2.111   rac1-vip.localdomain    rac1-vip
    192.168.2.112   rac2-vip.localdomain    rac2-vip
    # SCAN
    192.168.2.201   rac-scan.localdomain rac-scan
    [root@rac2 ~]#
    FIRST NODE root.sh script output...
    [root@rac2 ~]# /u01/app/11.2.0/db_1/root.sh
    Running Oracle 11g root.sh script...
    The following environment variables are set as:
        ORACLE_OWNER= oracle
        ORACLE_HOME=  /u01/app/11.2.0/db_1
    Enter the full pathname of the local bin directory: [/usr/local/bin]:
       Copying dbhome to /usr/local/bin ...
       Copying oraenv to /usr/local/bin ...
       Copying coraenv to /usr/local/bin ...
    Creating /etc/oratab file...
    Entries will be added to the /etc/oratab file as needed by
    Database Configuration Assistant when a database is created
    Finished running generic part of root.sh script.
    Now product-specific root actions will be performed.
    2010-12-06 14:45:06: Parsing the host name
    2010-12-06 14:45:06: Checking for super user privileges
    2010-12-06 14:45:06: User has super user privileges
    Using configuration parameter file: /u01/app/11.2.0/db_1/crs/install/crsconfig_params
    Creating trace directory
    LOCAL ADD MODE
    Creating OCR keys for user 'root', privgrp 'root'..
    Operation successful.
      root wallet
      root wallet cert
      root cert export
      peer wallet
      profile reader wallet
      pa wallet
      peer wallet keys
      pa wallet keys
      peer cert request
      pa cert request
      peer cert
      pa cert
      peer root cert TP
      profile reader root cert TP
      pa root cert TP
      peer pa cert TP
      pa peer cert TP
      profile reader pa cert TP
      profile reader peer cert TP
      peer user cert
      pa user cert
    Adding daemon to inittab
    CRS-4123: Oracle High Availability Services has been started.
    ohasd is starting
    CRS-2672: Attempting to start 'ora.gipcd' on 'rac2'
    CRS-2672: Attempting to start 'ora.mdnsd' on 'rac2'
    CRS-2676: Start of 'ora.gipcd' on 'rac2' succeeded
    CRS-2676: Start of 'ora.mdnsd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.gpnpd' on 'rac2'
    CRS-2676: Start of 'ora.gpnpd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac2'
    CRS-2676: Start of 'ora.cssdmonitor' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.cssd' on 'rac2'
    CRS-2672: Attempting to start 'ora.diskmon' on 'rac2'
    CRS-2676: Start of 'ora.diskmon' on 'rac2' succeeded
    CRS-2676: Start of 'ora.cssd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.ctssd' on 'rac2'
    CRS-2676: Start of 'ora.ctssd' on 'rac2' succeeded
    ASM created and started successfully.
    DiskGroup DATA created successfully.
    clscfg: -install mode specified
    Successfully accumulated necessary OCR keys.
    Creating OCR keys for user 'root', privgrp 'root'..
    Operation successful.
    CRS-2672: Attempting to start 'ora.crsd' on 'rac2'
    CRS-2676: Start of 'ora.crsd' on 'rac2' succeeded
    CRS-4256: Updating the profile
    Successful addition of voting disk 587cc69413ce4fd3bf0c2c2548fb9017.
    Successfully replaced voting disk group with +DATA.
    CRS-4256: Updating the profile
    CRS-4266: Voting file(s) successfully replaced
    ##  STATE    File Universal Id                File Name Disk group
    1. ONLINE   587cc69413ce4fd3bf0c2c2548fb9017 (/dev/oracleasm/disks/DISK1) [DATA]
    Located 1 voting disk(s).
    CRS-2673: Attempting to stop 'ora.crsd' on 'rac2'
    CRS-2677: Stop of 'ora.crsd' on 'rac2' succeeded
    CRS-2673: Attempting to stop 'ora.asm' on 'rac2'
    CRS-2677: Stop of 'ora.asm' on 'rac2' succeeded
    CRS-2673: Attempting to stop 'ora.ctssd' on 'rac2'
    CRS-2677: Stop of 'ora.ctssd' on 'rac2' succeeded
    CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'rac2'
    CRS-2677: Stop of 'ora.cssdmonitor' on 'rac2' succeeded
    CRS-2673: Attempting to stop 'ora.cssd' on 'rac2'
    CRS-2677: Stop of 'ora.cssd' on 'rac2' succeeded
    CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac2'
    CRS-2677: Stop of 'ora.gpnpd' on 'rac2' succeeded
    CRS-2673: Attempting to stop 'ora.gipcd' on 'rac2'
    CRS-2677: Stop of 'ora.gipcd' on 'rac2' succeeded
    CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac2'
    CRS-2677: Stop of 'ora.mdnsd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.mdnsd' on 'rac2'
    CRS-2676: Start of 'ora.mdnsd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.gipcd' on 'rac2'
    CRS-2676: Start of 'ora.gipcd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.gpnpd' on 'rac2'
    CRS-2676: Start of 'ora.gpnpd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac2'
    CRS-2676: Start of 'ora.cssdmonitor' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.cssd' on 'rac2'
    CRS-2672: Attempting to start 'ora.diskmon' on 'rac2'
    CRS-2676: Start of 'ora.diskmon' on 'rac2' succeeded
    CRS-2676: Start of 'ora.cssd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.ctssd' on 'rac2'
    CRS-2676: Start of 'ora.ctssd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.asm' on 'rac2'
    CRS-2676: Start of 'ora.asm' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.crsd' on 'rac2'
    CRS-2676: Start of 'ora.crsd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.evmd' on 'rac2'
    CRS-2676: Start of 'ora.evmd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.asm' on 'rac2'
    CRS-2676: Start of 'ora.asm' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.DATA.dg' on 'rac2'
    CRS-2676: Start of 'ora.DATA.dg' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.registry.acfs' on 'rac2'
    CRS-2676: Start of 'ora.registry.acfs' on 'rac2' succeeded
    rac2     2010/12/06 14:52:06     /u01/app/11.2.0/db_1/cdata/rac2/backup_20101206_145206.olr
    Configure Oracle Grid Infrastructure for a Cluster ... succeeded
    Updating inventory properties for clusterware
    Starting Oracle Universal Installer...
    Checking swap space: must be greater than 500 MB.   Actual 6847 MB    Passed
    The inventory pointer is located at /etc/oraInst.loc
    The inventory is located at /u01/app/oraInventory
    'UpdateNodeList' was successful.
    [root@rac2 ~]#
    SECOND NODE root.sh script output
    [root@rac1 db_1]# ./root.sh
    Running Oracle 11g root.sh script...
    The following environment variables are set as:
        ORACLE_OWNER= oracle
        ORACLE_HOME=  /u01/app/11.2.0/db_1
    Enter the full pathname of the local bin directory: [/usr/local/bin]:
       Copying dbhome to /usr/local/bin ...
       Copying oraenv to /usr/local/bin ...
       Copying coraenv to /usr/local/bin ...
    Creating /etc/oratab file...
    Entries will be added to the /etc/oratab file as needed by
    Database Configuration Assistant when a database is created
    Finished running generic part of root.sh script.
    Now product-specific root actions will be performed.
    2010-12-06 14:54:11: Parsing the host name
    2010-12-06 14:54:11: Checking for super user privileges
    2010-12-06 14:54:11: User has super user privileges
    Using configuration parameter file: /u01/app/11.2.0/db_1/crs/install/crsconfig_params
    Creating trace directory
    LOCAL ADD MODE
    Creating OCR keys for user 'root', privgrp 'root'..
    Operation successful.
    Adding daemon to inittab
    CRS-4123: Oracle High Availability Services has been started.
    ohasd is starting
    CRS-2672: Attempting to start 'ora.gipcd' on 'rac1'
    CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1'
    CRS-2676: Start of 'ora.gipcd' on 'rac1' succeeded
    CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeeded
    CRS-2672: Attempting to start 'ora.gpnpd' on 'rac1'
    CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeeded
    CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'
    CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded
    CRS-2672: Attempting to start 'ora.cssd' on 'rac1'
    CRS-2672: Attempting to start 'ora.diskmon' on 'rac1'
    CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded
    CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded
    CRS-2672: Attempting to start 'ora.ctssd' on 'rac1'
    CRS-2676: Start of 'ora.ctssd' on 'rac1' succeeded
    ASM created and started successfully.
    DiskGroup DATA created successfully.
    clscfg: -install mode specified
    Successfully accumulated necessary OCR keys.
    Creating OCR keys for user 'root', privgrp 'root'..
    Operation successful.
    CRS-2672: Attempting to start 'ora.crsd' on 'rac1'
    CRS-2676: Start of 'ora.crsd' on 'rac1' succeeded
    Successful addition of voting disk 2761ce8d47b44fbabf73462151e3ba1d.
    Successfully replaced voting disk group with +DATA.
    CRS-4266: Voting file(s) successfully replaced
    ##  STATE    File Universal Id                File Name Disk group
    1. ONLINE   2761ce8d47b44fbabf73462151e3ba1d (/dev/oracleasm/disks/DISK1) [DATA]
    Located 1 voting disk(s).
    CRS-2673: Attempting to stop 'ora.crsd' on 'rac1'
    CRS-2677: Stop of 'ora.crsd' on 'rac1' succeeded
    CRS-2673: Attempting to stop 'ora.asm' on 'rac1'
    CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded
    CRS-2673: Attempting to stop 'ora.ctssd' on 'rac1'
    CRS-2677: Stop of 'ora.ctssd' on 'rac1' succeeded
    CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'rac1'
    CRS-2677: Stop of 'ora.cssdmonitor' on 'rac1' succeeded
    CRS-2673: Attempting to stop 'ora.cssd' on 'rac1'
    CRS-2677: Stop of 'ora.cssd' on 'rac1' succeeded
    CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac1'
    CRS-2677: Stop of 'ora.gpnpd' on 'rac1' succeeded
    CRS-2673: Attempting to stop 'ora.gipcd' on 'rac1'
    CRS-2677: Stop of 'ora.gipcd' on 'rac1' succeeded
    CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac1'
    CRS-2677: Stop of 'ora.mdnsd' on 'rac1' succeeded
    CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1'
    CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeeded
    CRS-2672: Attempting to start 'ora.gipcd' on 'rac1'
    CRS-2676: Start of 'ora.gipcd' on 'rac1' succeeded
    CRS-2672: Attempting to start 'ora.gpnpd' on 'rac1'
    CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeeded
    CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'
    CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded
    CRS-2672: Attempting to start 'ora.cssd' on 'rac1'
    CRS-2672: Attempting to start 'ora.diskmon' on 'rac1'
    CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded
    CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded
    CRS-2672: Attempting to start 'ora.ctssd' on 'rac1'
    CRS-2676: Start of 'ora.ctssd' on 'rac1' succeeded
    CRS-2672: Attempting to start 'ora.asm' on 'rac1'
    CRS-2676: Start of 'ora.asm' on 'rac1' succeeded
    CRS-2672: Attempting to start 'ora.crsd' on 'rac1'
    CRS-2676: Start of 'ora.crsd' on 'rac1' succeeded
    CRS-2672: Attempting to start 'ora.evmd' on 'rac1'
    CRS-2676: Start of 'ora.evmd' on 'rac1' succeeded
    CRS-2672: Attempting to start 'ora.asm' on 'rac1'
    CRS-2676: Start of 'ora.asm' on 'rac1' succeeded
    CRS-2672: Attempting to start 'ora.DATA.dg' on 'rac1'
    CRS-2676: Start of 'ora.DATA.dg' on 'rac1' succeeded
    PRCR-1079 : *Failed* to start resource ora.scan1.vip
    CRS-5005: IP Address: 192.168.2.201 is already in use in the network
    CRS-2674: Start of 'ora.scan1.vip' on 'rac1' *failed*
    CRS-2632: There are no more servers to try to place resource 'ora.scan1.vip' on that would satisfy its placement policy
    start scan ... *failed*
    Configure Oracle Grid Infrastructure for a Cluster ... *failed*
    Updating inventory properties for clusterware
    Starting Oracle Universal Installer...
    Checking swap space: must be greater than 500 MB.   Actual 6847 MB    Passed
    The inventory pointer is located at /etc/oraInst.loc
    The inventory is located at /u01/app/oraInventory
    'UpdateNodeList' was successful.
    [root@rac1 db_1]# * "./runcluvfy.sh stage -pre -crsinst -n rac1,rac2 " outputs are same each node....*
    [oracle@rac2 grid]$ ./runcluvfy.sh stage -pre crsinst -n rac1,rac2
    Performing pre-checks for cluster services setup
    Checking node reachability...
    Node reachability check passed from node "rac2"
    Checking user equivalence...
    User equivalence check passed for user "oracle"
    Checking node connectivity...
    Checking hosts config file...
    Verification of the hosts config file successful
    Node connectivity passed for subnet "192.168.2.0" with node(s) rac2,rac1
    TCP connectivity check passed for subnet "192.168.2.0"
    Node connectivity passed for subnet "192.168.122.0" with node(s) rac2,rac1
    TCP connectivity check failed for subnet "192.168.122.0"
    Node connectivity passed for subnet "192.168.0.0" with node(s) rac2,rac1
    TCP connectivity check passed for subnet "192.168.0.0"
    Interfaces found on subnet "192.168.2.0" that are likely candidates for VIP are:
    rac2 eth0:192.168.2.102 eth0:192.168.2.112 eth0:192.168.2.201
    rac1 eth0:192.168.2.101 eth0:192.168.2.111
    Interfaces found on subnet "192.168.122.0" that are likely candidates for a private interconnect are:
    rac2 virbr0:192.168.122.1
    rac1 virbr0:192.168.122.1
    Interfaces found on subnet "192.168.0.0" that are likely candidates for a private interconnect are:
    rac2 eth1:192.168.0.102
    rac1 eth1:192.168.0.101
    Node connectivity check passed
    Total memory check passed
    Available memory check passed
    Swap space check passed
    Free disk space check passed for "rac2:/tmp"
    Free disk space check passed for "rac1:/tmp"
    User existence check passed for "oracle"
    Group existence check passed for "oinstall"
    Group existence check passed for "dba"
    Membership check for user "oracle" in group "oinstall" [as Primary] passed
    Membership check for user "oracle" in group "dba" passed
    Run level check passed
    Hard limits check passed for "maximum open file descriptors"
    Soft limits check passed for "maximum open file descriptors"
    Hard limits check passed for "maximum user processes"
    Soft limits check passed for "maximum user processes"
    System architecture check passed
    Kernel version check passed
    Kernel parameter check passed for "semmsl"
    Kernel parameter check passed for "semmns"
    Kernel parameter check passed for "semopm"
    Kernel parameter check passed for "semmni"
    Kernel parameter check passed for "shmmax"
    Kernel parameter check passed for "shmmni"
    Kernel parameter check passed for "shmall"
    Kernel parameter check passed for "file-max"
    Kernel parameter check passed for "ip_local_port_range"
    Kernel parameter check passed for "rmem_default"
    Kernel parameter check passed for "rmem_max"
    Kernel parameter check passed for "wmem_default"
    Kernel parameter check passed for "wmem_max"
    Kernel parameter check passed for "aio-max-nr"
    Package existence check passed for "make-3.81"
    Package existence check passed for "binutils-2.17.50.0.6"
    Package existence check passed for "gcc-4.1.2"
    Package existence check passed for "libaio-0.3.106 (i386)"
    Package existence check passed for "libaio-0.3.106 (x86_64)"
    Package existence check passed for "glibc-2.5-24 (i686)"
    Package existence check passed for "glibc-2.5-24 (x86_64)"
    Package existence check passed for "compat-libstdc++-33-3.2.3 (i386)"
    Package existence check passed for "compat-libstdc++-33-3.2.3 (x86_64)"
    Package existence check passed for "elfutils-libelf-0.125 (x86_64)"
    Package existence check passed for "elfutils-libelf-devel-0.125"
    Package existence check passed for "glibc-common-2.5"
    Package existence check passed for "glibc-devel-2.5 (i386)"
    Package existence check passed for "glibc-devel-2.5 (x86_64)"
    Package existence check passed for "glibc-headers-2.5"
    Package existence check passed for "gcc-c++-4.1.2"
    Package existence check passed for "libaio-devel-0.3.106 (i386)"
    Package existence check passed for "libaio-devel-0.3.106 (x86_64)"
    Package existence check passed for "libgcc-4.1.2 (i386)"
    Package existence check passed for "libgcc-4.1.2 (x86_64)"
    Package existence check passed for "libstdc++-4.1.2 (i386)"
    Package existence check passed for "libstdc++-4.1.2 (x86_64)"
    Package existence check passed for "libstdc++-devel-4.1.2 (x86_64)"
    Package existence check passed for "sysstat-7.0.2"
    Package existence check passed for "unixODBC-2.2.11 (i386)"
    Package existence check passed for "unixODBC-2.2.11 (x86_64)"
    Package existence check passed for "unixODBC-devel-2.2.11 (i386)"
    Package existence check passed for "unixODBC-devel-2.2.11 (x86_64)"
    Package existence check passed for "ksh-20060214"
    Check for multiple users with UID value 0 passed
    Current group ID check passed
    Core file name pattern consistency check passed.
    User "oracle" is not part of "root" group. Check passed
    Default user file creation mask check passed
    Starting Clock synchronization checks using Network Time Protocol(NTP)...
    NTP Configuration file check started...
    NTP Configuration file check passed
    Checking daemon liveness...
    Liveness check passed for "ntpd"
    NTP daemon slewing option check passed
    NTP daemon's boot time configuration check for slewing option passed
    NTP common Time Server Check started...
    Check of common NTP Time Server passed
    Clock time offset check from NTP Time Server started...
    Clock time offset check passed
    Clock synchronization check using Network Time Protocol(NTP) passed
    Pre-check for cluster services setup was successful.
    [oracle@rac2 grid]$ I'm confused :)
    Edited by: Eren GULERYUZ on 06.Ara.2010 05:57

    Hi,
    it looks like, that your "shared device" you are using is not really shared.
    The second node does "create an ASM diskgroup" and create OCR and Voting disks. If this indeed would be a shared device, he should have recognized, that your disk is shared.
    So as a result your VMware configuration must be wrong, and the disk you presented as shared disk is not really shared.
    Which VMWare version did you use? It will not work correctly with the workstation or player edition, since shared disks are only really working with the server version.
    If you indeed using the server, could you paste your vm configurations?
    Furthermore I recommend using Virtual Box. There is a nice how-to:
    http://www.oracle-base.com/articles/11g/OracleDB11gR2RACInstallationOnOEL5UsingVirtualBox.php
    Sebastian

  • Timed out waiting for the CRS stack to start on Second node.

    Hi,
    I am trying to setup 2 node 11gR2 RAC in vmware. I face the Timed out waiting for the CRS stack to start error on second node while running root.sh.I checked the cluster log files located in  /u01/app/11.2.0/grid/log/node1/alertnode2.log it shows as mentioned below.But when i logged into ASM instance and checked the diskgroup and it is in mount state on both nodes,So i am really confused why i ended up with error in 2nd node on running root.sh. Can anyone tell me how to correct this error or any other checking needs to be done?
    /u01/app/11.2.0/grid/log/node1/alertnode2.log
    [ohasd(23261)]CRS-2765:Resource 'ora.crsd' has failed on server 'node2'.
    2014-01-21 21:40:10.314
    [crsd(25309)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/node2/crsd/crsd.log.
    2014-01-21 21:40:11.217
    [ohasd(23261)]CRS-2765:Resource 'ora.crsd' has failed on server 'node2'.
    2014-01-21 21:40:12.422
    [crsd(25335)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/node2/crsd/crsd.log.
    2014-01-21 21:40:13.327
    [ohasd(23261)]CRS-2765:Resource 'ora.crsd' has failed on server 'node2'.
    2014-01-21 21:40:14.514
    [crsd(25356)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/node2/crsd/crsd.log.
    2014-01-21 21:40:15.439
    [ohasd(23261)]CRS-2765:Resource 'ora.crsd' has failed on server 'node2'.
    2014-01-21 21:40:15.440
    [ohasd(23261)]CRS-2771:Maximum restart attempts reached for resource 'ora.crsd'; will not restart.
    root.sh output
    [root@node2 ~]# /u01/app/11.2.0/grid/root.sh
    Running Oracle 11g root.sh script...
    The following environment variables are set as:
        ORACLE_OWNER= oracle
        ORACLE_HOME=  /u01/app/11.2.0/grid
    Enter the full pathname of the local bin directory: [/usr/local/bin]:
    The file "dbhome" already exists in /usr/local/bin.  Overwrite it? (y/n)
    [n]: y
       Copying dbhome to /usr/local/bin ...
    The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
    [n]: y
       Copying oraenv to /usr/local/bin ...
    The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
    [n]: y
       Copying coraenv to /usr/local/bin ...
    Entries will be added to the /etc/oratab file as needed by
    Database Configuration Assistant when a database is created
    Finished running generic part of root.sh script.
    Now product-specific root actions will be performed.
    2014-01-21 21:37:55: Parsing the host name
    2014-01-21 21:37:55: Checking for super user privileges
    2014-01-21 21:37:55: User has super user privileges
    Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_p                   
    arams
    LOCAL ADD MODE
    Creating OCR keys for user 'root', privgrp 'root'..
    Operation successful.
    Adding daemon to inittab
    CRS-4123: Oracle High Availability Services has been started.
    ohasd is starting
    CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node 11gdb, number 1, and is terminating
    An active cluster was found during exclusive startup, restarting to join the cluster
    CRS-2672: Attempting to start 'ora.mdnsd' on 'node2'
    CRS-2676: Start of 'ora.mdnsd' on 'node2' succeeded
    CRS-2672: Attempting to start 'ora.gipcd' on 'node2'
    CRS-2676: Start of 'ora.gipcd' on 'node2' succeeded
    CRS-2672: Attempting to start 'ora.gpnpd' on 'node2'
    CRS-2676: Start of 'ora.gpnpd' on 'node2' succeeded
    CRS-2672: Attempting to start 'ora.cssdmonitor' on 'node2'
    CRS-2676: Start of 'ora.cssdmonitor' on 'node2' succeeded
    CRS-2672: Attempting to start 'ora.cssd' on 'node2'
    CRS-2672: Attempting to start 'ora.diskmon' on 'node2'
    CRS-2676: Start of 'ora.diskmon' on 'node2' succeeded
    CRS-2676: Start of 'ora.cssd' on 'node2' succeeded
    CRS-2672: Attempting to start 'ora.ctssd' on 'node2'
    CRS-2676: Start of 'ora.ctssd' on 'node2' succeeded
    CRS-2672: Attempting to start 'ora.drivers.acfs' on 'node2'
    CRS-2676: Start of 'ora.drivers.acfs' on 'node2' succeeded
    CRS-2672: Attempting to start 'ora.asm' on 'node2'
    CRS-2676: Start of 'ora.asm' on 'node2' succeeded
    CRS-2672: Attempting to start 'ora.crsd' on 'node2'
    CRS-2676: Start of 'ora.crsd' on 'node2' succeeded
    CRS-2672: Attempting to start 'ora.evmd' on 'node2'
    CRS-2676: Start of 'ora.evmd' on 'node2' succeeded
    Timed out waiting for the CRS stack to start.
    Regards,
    007

    Hi,
    Are you trying to install on Vmware Workstation or Vmware Server?
    If you are using vmware server add below line in your .vmx file for both node
    scsi1.sharedBus = "VIRTUAL"
    First check which scsi serial number are you using as above I have selected scsi1 serial during disk addition, 

  • The Script root.sh problem - ora.asm and ASM and Clusterware Stack failed

    Folks,
    Hello. I am installing Oracle 11gR2 RAC using 2 VMs (rac1 and rac2) whose OS are Oracle Linux 5.6 in VMPlayer according to the website http://appsdbaworkshop.blogspot.com/2011/10/11gr2-rac-on-linux-56-using-vmware.html
    I am installing Grid infrastructure. On step 9 of 10 - execute script /u01/app/grid/root.sh for 2 VMs rac1 and rac2.
    After run root.sh in rac1 successfully. I run root.sh in rac2 and get an error as below:
    [root@rac2 grid]# ./root.sh
    Running Oracle 11g root.sh script...
    The following environment variables are set as:
    ORACLE_OWNER= ora11g
    ORACLE_HOME= /u01/app/grid
    Enter the full pathname of the local bin directory: [usr/local/bin]: /usr/local/bin
    Copying dbhome to /usr/local/bin ...
    Copying oraenv to /usr/local/bin ...
    Copying coraenv to /usr/local/bin ...
    Creating /etc/oratab file...
    Entries will be added to the /etc/oratab file as needed by
    Database Configuration Assistant when a database is created
    Finished running generic part of root.sh script.
    Now product-specific root actions will be performed.
    2012-03-05 16:32:52: Parsing the host name
    2012-03-05 16:32:52: Checking for super user privileges
    2012-03-05 16:32:52: User has super user privileges
    Using configuration parameter file: /u01/app/grid/crs/install/crsconfig_params
    Creating trace directory
    LOCAL ADD MODE
    Creating OCR keys for user 'root', privgrp 'root'..
    Operation successful.
    Adding daemon to inittab
    CRS-4123: Oracle High Availability Services has been started.
    ohasd is starting
    CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node rac1, number 1, and is terminating
    An active cluster was found during exclusive startup, restarting to join the cluster
    CRS-2672: Attempting to start 'ora.mdnsd' on 'rac2'
    CRS-2676: Start of 'ora.mdnsd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.gipcd' on 'rac2'
    CRS-2676: Start of 'ora.gipcd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.gpnpd' on 'rac2'
    CRS-2676: Start of 'ora.gpnpd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac2'
    CRS-2676: Start of 'ora.cssdmonitor' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.cssd' on 'rac2'
    CRS-2672: Attempting to start 'ora.diskmon' on 'rac2'
    CRS-2676: Start of 'ora.diskmon' on 'rac2' succeeded
    CRS-2676: Start of 'ora.cssd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.ctssd' on 'rac2'
    Start action for octssd aborted
    CRS-2676: Start of 'ora.ctssd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.drivers.acfs' on 'rac2'
    CRS-2672: Attempting to start 'ora.asm' on 'rac2'
    CRS-2676: Start of 'ora.drivers.acfs' on 'rac2' succeeded
    CRS-2676: Start of 'ora.asm' on 'rac2' succeeded
    CRS-2664: Resource 'ora.ctssd' is already running on 'rac2'
    CRS-4000: Command Start failed, or completed with errors.
    Command return code of 1 (256) from command: /u01/app/grid/bin/crsctl start resource ora.asm -init
    Start of resource "ora.asm -init" failed
    Failed to start ASM
    Failed to start Oracle Clusterware stack
    [root@rac2 grid]#
    As we see the output above, at the end of the output
    1) Start of resource ora.asm -init failed
    2) Failed to start ASM
    3) Failed to start Oracle Clusterware stack
    The runInstaller is in the first VM rac1. My question is:
    Do any folk understand how to solve the script root.sh in rac2 problem ( 3 fails of ora.asm, ASM and Clusterware stack as above) ?
    Thanks.

    Please check there is no firewall exist:
    try this like:
    root.sh fails on second node
    MOS note:
    11gR2 Grid: root.sh Fails to Start the Clusterware on the Second Node Due to Firewall on Private Network [ID 981357.1]
    Grid Infrastructure 11.2.0.2 Installation or Upgrade may fail due to Multicasting Requirement [ID 1212703.1] (Most probabily this issue)

  • Error after running root.sh on second node

    hi,
    i have installed clusterware on a 2 node system running on RHEL 5
    i followed the prereqs , and solved all the errors i encountered
    after clusterware installation , it asks to run root.sh on all the nodes
    when i ran root.sh on second node
    it gave this error
    Running vipca(silent) for configuring nodeapps
    /home/oracle/crs/oracle/product/10/crs/jdk/jre//bin/java: error while loading
    shared libraries: libpthread.so.0: cannot open shared object file:
    No such file or directory
    so i follwed metalink note 414163.1
    after that i called it a day
    in the morning , wheni started both nodes
    and started vipca on second node
    it gave this error
    PRKH:1010 unable to communicate with crs services
    the i ran ps-ef | grep crs
    root 3201 1 0 15:37 ? 00:00:00 /bin/sh /etc/init.d/init.crsd run
    crsctl check crs gave
    failure 1 contacting css daemon
    cannot communicate with crs
    cannot communicate with evm
    what should i do to start these services?

    crsd and cssd logs were empty and there was no relevant info in crs alert
    i am just reinstalling clusterware now
    one thing i wanted to ask
    why does owner ship of raw files change back to root (after node restart)
    even though i chnged them to oracle

  • Error CLSRSC-507 during the execution of root.sh on second node

    Hi all.
       OS.......: Red-Hat 6.5
       RDBMS: Oracle 12.1.0.2.0
       During the installation of a 2-node RAC in a RHEL 6.5, during the execution of the root.sh script in the second node, i get the following error:
    [root@oraprd02 grid]# ./root.sh
    Performing root user operation.
    The following environment variables are set as:
        ORACLE_OWNER= grid
        ORACLE_HOME=  /u01/app/12.1.0/grid
    Enter the full pathname of the local bin directory: [/usr/local/bin]:
       Copying dbhome to /usr/local/bin ...
       Copying oraenv to /usr/local/bin ...
       Copying coraenv to /usr/local/bin ...
    Creating /etc/oratab file...
    Entries will be added to the /etc/oratab file as needed by
    Database Configuration Assistant when a database is created
    Finished running generic part of root script.
    Now product-specific root actions will be performed.
    Using configuration parameter file: /u01/app/12.1.0/grid/crs/install/crsconfig_params
    2015/05/04 22:47:16 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
    2015/05/04 22:47:59 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
    2015/05/04 22:48:00 CLSRSC-363: User ignored prerequisites during installation
    OLR initialization - successful
    2015/05/04 22:48:46 CLSRSC-507: The root script cannot proceed on this node oraprd02 because either the first-node operations have not completed on node oraprd01 or there was an error in obtaining the status of the first-node operations.
    Died at /u01/app/12.1.0/grid/crs/install/crsutils.pm line 3681.
    The command '/u01/app/12.1.0/grid/perl/bin/perl -I/u01/app/12.1.0/grid/perl/lib -I/u01/app/12.1.0/grid/crs/install /u01/app/12.1.0/grid/crs/install/rootcrs.pl ' execution failed
    The root.sh on the first node completed successfully. I get the succeeded message from the script in the first node.
    Have anyone faced this problem? Any assistance will be most helpfull.
    Thanks in advance.

    crsd and cssd logs were empty and there was no relevant info in crs alert
    i am just reinstalling clusterware now
    one thing i wanted to ask
    why does owner ship of raw files change back to root (after node restart)
    even though i chnged them to oracle

  • Root.sh failed on second node while installing CRS 10g on centos 5.5

    root.sh failed on second node while installing CRS 10g
    Hi all,
    I am able to install Oracle 10g RAC clusterware on first node of the cluster. However, when I run the root.sh script as root
    user on second node of the cluster, it fails with following error message:
    NO KEYS WERE WRITTEN. Supply -force parameter to override.
    -force is destructive and will destroy any previous cluster
    configuration.
    Oracle Cluster Registry for cluster has already been initialized
    Startup will be queued to init within 90 seconds.
    Adding daemons to inittab
    Expecting the CRS daemons to be up within 600 seconds.
    Failure at final check of Oracle CRS stack.
    10
    and run cluvfy stage -post hwos -n all -verbose,it show message:
    ERROR:
    Could not find a suitable set of interfaces for VIPs.
    Result: Node connectivity check failed.
    Checking shared storage accessibility...
    Disk Sharing Nodes (2 in count)
    /dev/sda db2 db1
    and run cluvfy stage -pre crsinst -n all -verbose,it show message:
    ERROR:
    Could not find a suitable set of interfaces for VIPs.
    Result: Node connectivity check failed.
    Checking system requirements for 'crs'...
    No checks registered for this product.
    and run cluvfy stage -post crsinst -n all -verbose,it show message:
    Result: Node reachability check passed from node "DB2".
    Result: User equivalence check passed for user "oracle".
    Node Name CRS daemon CSS daemon EVM daemon
    db2 no no no
    db1 yes yes yes
    Check: Health of CRS
    Node Name CRS OK?
    db1 unknown
    Result: CRS health check failed.
    check crsd.log and show message:
    clsc_connect: (0x143ca610) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_db2_crs))
    clsssInitNative: connect failed, rc 9
    Any help would be greatly appreciated.
    Edited by: 868121 on 2011-6-24 上午12:31

    Hello, it took a little searching, but I found this in a note in the GRID installation guide for Linux/UNIX:
    Public IP addresses and virtual IP addresses must be in the same subnet.
    In your case, you are using two different subnets for the VIPs.

  • Root.sh failed at second node OUL 6.3 Oracle GRID 11.2.0.3

    Hi, im installing a two node cluster mounted on Oracle Linux 6.3 with Oracle DB 11.2.0.3, the installation went smooth up until the execution of the root.sh script on the second node.
    THe script return this final lines:
    CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node nodo1, number 1, and is terminating
    An active cluster was found during exclusive startup, restarting to join the cluster
    Start of resource "ora.crsd" failed
    CRS-2800: Cannot start resource 'ora.asm' as it is already in the INTERMEDIATE state on server 'nodo2'
    CRS-4000: Command Start failed, or completed with errors.
    Failed to start Oracle Grid Infrastructure stack
    Failed to start Cluster Ready Services at /u01/app/11.2.0/grid/crs/install/crsconfig_lib.pm line 1286.
    /u01/app/11.2.0/grid/perl/bin/perl -I/u01/app/11.2.0/grid/perl/lib -I/u01/app/11.2.0/grid/crs/install /u01/app/11.2.0/grid/crs/install/rootcrs.pl execution failed
    In $GRID_HOME/log/node2/alertnode.log It appears to be a Cluster Time Synchronization Service issue, (i didn't synchronyze the nodes..) however the CTSS is running in observer mode, wich i believe it shouldn't affect the installation process. After that i lost it...there's an entry CRS-5018 indicating that an unused HAIP route was removed... and then, out of the blue: CRS-5818:Aborted command 'start' for resource 'ora.asm'. Some clarification will be deeply apreciated.
    Here's the complete log:
    2013-04-01 13:39:35.358
    [client(12163)]CRS-2101:The OLR was formatted using version 3.
    2013-04-01 19:40:19.597
    [ohasd(12338)]CRS-2112:The OLR service started on node nodo2.
    2013-04-01 19:40:19.657
    [ohasd(12338)]CRS-1301:Oracle High Availability Service started on node nodo2.
    [client(12526)]CRS-10001:01-Apr-13 13:41 ACFS-9459: ADVM/ACFS is not supported on this OS version: '2.6.39-400.17.2.el6uek.i686'
    [client(12528)]CRS-10001:01-Apr-13 13:41 ACFS-9201: Not Supported
    [client(12603)]CRS-10001:01-Apr-13 13:41 ACFS-9459: ADVM/ACFS is not supported on this OS version: '2.6.39-400.17.2.el6uek.i686'
    2013-04-01 19:41:17.509
    [ohasd(12338)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
    2013-04-01 19:41:17.618
    [gpnpd(12695)]CRS-2328:GPNPD started on node nodo2.
    2013-04-01 19:41:21.363
    [cssd(12755)]CRS-1713:CSSD daemon is started in exclusive mode
    2013-04-01 19:41:23.194
    [ohasd(12338)]CRS-2767:Resource state recovery not attempted for 'ora.diskmon' as its target state is OFFLINE
    2013-04-01 19:41:56.144
    [cssd(12755)]CRS-1707:Lease acquisition for node nodo2 number 2 completed
    2013-04-01 19:41:57.545
    [cssd(12755)]CRS-1605:CSSD voting file is online: /dev/oracleasm/disks/ASM_DISK_1; details in /u01/app/11.2.0/grid/log/nodo2/cssd/ocssd.log.
    [cssd(12755)]CRS-1636:The CSS daemon was started in exclusive mode but found an active CSS daemon on node nodo1 and is terminating; details at (:CSSNM00006:) in /u01/app/11.2.0/grid/log/nodo2/cssd/ocssd.log
    2013-04-01 19:41:58.549
    [ohasd(12338)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server 'nodo2'.
    2013-04-01 19:42:10.025
    [gpnpd(12695)]CRS-2329:GPNPD on node nodo2 shutdown.
    2013-04-01 19:42:11.407
    [mdnsd(12685)]CRS-5602:mDNS service stopping by request.
    2013-04-01 19:42:29.642
    [gpnpd(12947)]CRS-2328:GPNPD started on node nodo2.
    2013-04-01 19:42:33.241
    [cssd(13012)]CRS-1713:CSSD daemon is started in clustered mode
    2013-04-01 19:42:35.104
    [ohasd(12338)]CRS-2767:Resource state recovery not attempted for 'ora.diskmon' as its target state is OFFLINE
    2013-04-01 19:42:44.065
    [cssd(13012)]CRS-1707:Lease acquisition for node nodo2 number 2 completed
    2013-04-01 19:42:45.484
    [cssd(13012)]CRS-1605:CSSD voting file is online: /dev/oracleasm/disks/ASM_DISK_1; details in /u01/app/11.2.0/grid/log/nodo2/cssd/ocssd.log.
    2013-04-01 19:42:52.138
    [cssd(13012)]CRS-1601:CSSD Reconfiguration complete. Active nodes are nodo1 nodo2 .
    2013-04-01 19:42:55.081
    [ctssd(13076)]CRS-2403:The Cluster Time Synchronization Service on host nodo2 is in observer mode.
    2013-04-01 19:42:55.581
    [ctssd(13076)]CRS-2401:The Cluster Time Synchronization Service started on host nodo2.
    2013-04-01 19:42:55.581
    [ctssd(13076)]CRS-2407:The new Cluster Time Synchronization Service reference node is host nodo1.
    2013-04-01 19:43:08.875
    [ctssd(13076)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/app/11.2.0/grid/log/nodo2/ctssd/octssd.log.
    2013-04-01 19:43:08.876
    [ctssd(13076)]CRS-2409:The clock on host nodo2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
    2013-04-01 19:43:13.565
    [u01/app/11.2.0/grid/bin/orarootagent.bin(13064)]CRS-5018:(:CLSN00037:) Removed unused HAIP route: 169.254.0.0 / 255.255.0.0 / 0.0.0.0 / eth0
    2013-04-01 19:53:09.800
    [u01/app/11.2.0/grid/bin/oraagent.bin(12922)]CRS-5818:Aborted command 'start' for resource 'ora.asm'. Details at (:CRSAGF00113:) {0:0:223} in /u01/app/11.2.0/grid/log/nodo2/agent/ohasd/oraagent_oracle/oraagent_oracle.log.
    2013-04-01 19:53:11.827
    [ohasd(12338)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.asm'. Details at (:CRSPE00111:) {0:0:223} in /u01/app/11.2.0/grid/log/nodo2/ohasd/ohasd.log.
    2013-04-01 19:53:12.779
    [u01/app/11.2.0/grid/bin/oraagent.bin(12922)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/11.2.0/grid/log/nodo2/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
    2013-04-01 19:53:13.892
    [u01/app/11.2.0/grid/bin/oraagent.bin(12922)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/11.2.0/grid/log/nodo2/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
    2013-04-01 19:53:43.877
    [u01/app/11.2.0/grid/bin/oraagent.bin(12922)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/11.2.0/grid/log/nodo2/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
    2013-04-01 19:54:13.891
    [u01/app/11.2.0/grid/bin/oraagent.bin(12922)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/11.2.0/grid/log/nodo2/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
    2013-04-01 19:54:43.906
    [u01/app/11.2.0/grid/bin/oraagent.bin(12922)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/11.2.0/grid/log/nodo2/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
    2013-04-01 19:55:13.914
    [u01/app/11.2.0/grid/bin/oraagent.bin(12922)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/11.2.0/grid/log/nodo2/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
    2013-04-01 19:55:43.918
    [u01/app/11.2.0/grid/bin/oraagent.bin(12922)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/11.2.0/grid/log/nodo2/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
    2013-04-01 19:56:13.922
    [u01/app/11.2.0/grid/bin/oraagent.bin(12922)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/11.2.0/grid/log/nodo2/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
    2013-04-01 19:56:53.209
    [crsd(13741)]CRS-1012:The OCR service started on node nodo2.
    2013-04-01 20:07:01.128
    [crsd(13741)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
    2013-04-01 20:07:01.278
    [ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
    2013-04-01 20:07:08.689
    [crsd(15248)]CRS-1012:The OCR service started on node nodo2.
    2013-04-01 20:13:10.138
    [ctssd(13076)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/app/11.2.0/grid/log/nodo2/ctssd/octssd.log.
    2013-04-01 20:17:13.024
    [crsd(15248)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
    2013-04-01 20:17:13.171
    [ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
    2013-04-01 20:17:20.826
    [crsd(16746)]CRS-1012:The OCR service started on node nodo2.
    2013-04-01 20:27:25.020
    [crsd(16746)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
    2013-04-01 20:27:25.176
    [ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
    2013-04-01 20:27:31.591
    [crsd(18266)]CRS-1012:The OCR service started on node nodo2.
    2013-04-01 20:37:35.668
    [crsd(18266)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
    2013-04-01 20:37:35.808
    [ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
    2013-04-01 20:37:43.209
    [crsd(19762)]CRS-1012:The OCR service started on node nodo2.
    2013-04-01 20:43:11.160
    [ctssd(13076)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/app/11.2.0/grid/log/nodo2/ctssd/octssd.log.
    2013-04-01 20:47:47.487
    [crsd(19762)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
    2013-04-01 20:47:47.637
    [ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
    2013-04-01 20:47:55.086
    [crsd(21242)]CRS-1012:The OCR service started on node nodo2.
    2013-04-01 20:57:59.343
    [crsd(21242)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
    2013-04-01 20:57:59.492
    [ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
    2013-04-01 20:58:06.996
    [crsd(22744)]CRS-1012:The OCR service started on node nodo2.
    2013-04-01 21:08:11.046
    [crsd(22744)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
    2013-04-01 21:08:11.192
    [ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
    2013-04-01 21:08:18.726
    [crsd(24260)]CRS-1012:The OCR service started on node nodo2.
    2013-04-01 21:13:12.000
    [ctssd(13076)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/app/11.2.0/grid/log/nodo2/ctssd/octssd.log.
    2013-04-01 21:18:22.262
    [crsd(24260)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
    2013-04-01 21:18:22.411
    [ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
    2013-04-01 21:18:29.927
    [crsd(25759)]CRS-1012:The OCR service started on node nodo2.
    2013-04-01 21:28:34.467
    [crsd(25759)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
    2013-04-01 21:28:34.616
    [ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
    2013-04-01 21:28:41.990
    [crsd(27291)]CRS-1012:The OCR service started on node nodo2.
    2013-04-01 21:38:45.012
    [crsd(27291)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
    2013-04-01 21:38:45.160
    [ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
    2013-04-01 21:38:52.790
    [crsd(28784)]CRS-1012:The OCR service started on node nodo2.
    2013-04-01 21:43:12.378
    [ctssd(13076)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/app/11.2.0/grid/log/nodo2/ctssd/octssd.log.
    2013-04-01 21:48:56.285
    [crsd(28784)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
    2013-04-01 21:48:56.435
    [ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
    2013-04-01 21:49:04.421
    [crsd(30272)]CRS-1012:The OCR service started on node nodo2.
    2013-04-01 21:59:08.183
    [crsd(30272)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
    2013-04-01 21:59:08.318
    [ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
    2013-04-01 21:59:15.860
    [crsd(31772)]CRS-1012:The OCR service started on node nodo2.

    Hi santysharma, thanks for the reply, i have two ethernet interfaces: eth0 (public network 192.168.1.0) and eth1 (private network 10.5.3.0), there is no device using that ip range, here's the output of route command:
    (Sorry for the alignment, i tried to tab it but the editor trims it again)
    Kernel IP routing table
    Destination Gateway Genmask Flags Metric Ref Use Iface
    default 192.168.1.1 0.0.0.0 UG 0 0 0 eth0
    private * 255.255.255.0 U 0 0 0 eth1
    link-local * 255.255.0.0 U 1002 0 0 eth0
    link-local * 255.255.0.0 U 1003 0 0 eth1
    public * 255.255.255.0 U 0 0 0 eth0
    And the /etc/hosts file
    127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
    ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
    10.5.3.1 nodo1.cluster nodo1
    10.5.3.2 nodo2.cluster nodo2
    192.168.1.13 cluster-scan
    192.168.1.14 nodo1-vip
    192.168.1.15 nodo2-vip
    And the ifconfig -a
    eth0 Link encap:Ethernet HWaddr C8:3A:35:D9:C6:2B
    inet addr:192.168.1.12 Bcast:192.168.1.255 Mask:255.255.255.0
    inet6 addr: fe80::ca3a:35ff:fed9:c62b/64 Scope:Link
    UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
    RX packets:34708 errors:0 dropped:18 overruns:0 frame:0
    TX packets:24693 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000
    RX bytes:48545969 (46.2 MiB) TX bytes:1994381 (1.9 MiB)
    eth1 Link encap:Ethernet HWaddr 00:0D:87:D0:A3:8E
    inet addr:10.5.3.2 Bcast:10.5.3.255 Mask:255.255.255.0
    inet6 addr: fe80::20d:87ff:fed0:a38e/64 Scope:Link
    UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
    RX packets:0 errors:0 dropped:0 overruns:0 frame:0
    TX packets:44 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000
    RX bytes:0 (0.0 b) TX bytes:5344 (5.2 KiB)
    Interrupt:23 Base address:0x6000
    lo Link encap:Local Loopback
    inet addr:127.0.0.1 Mask:255.0.0.0
    inet6 addr: ::1/128 Scope:Host
    UP LOOPBACK RUNNING MTU:16436 Metric:1
    RX packets:20 errors:0 dropped:0 overruns:0 frame:0
    TX packets:20 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:0
    RX bytes:1320 (1.2 KiB) TX bytes:1320 (1.2 KiB)
    Now that i'm thinking i've read somewhere that ipv6 was no supported...yet there's no relation with the 169.254.x.x ip range.

  • Root.sh fails on second node

    I already posted this issue on database installation forum, and was suggested to post it on this forum.
    Here are the details.
    I am running Linux 64bit on ESx clients. Installing Oracle 11gR2.
    It passed all the per-requisite. Run root.sh on first node. It finished with no errorrs.
    On second node I got the following:
    Running Oracle 11g root.sh script...
    The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME= /u01/app/11.2.0/grid
    Enter the full pathname of the local bin directory: [usr/local/bin]:
    The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n)
    [n]:
    The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
    [n]:
    The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
    [n]:
    Entries will be added to the /etc/oratab file as needed by
    Database Configuration Assistant when a database is created
    Finished running generic part of root.sh script.
    Now product-specific root actions will be performed.
    2010-07-13 12:51:28: Parsing the host name
    2010-07-13 12:51:28: Checking for super user privileges
    2010-07-13 12:51:28: User has super user privileges
    Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
    Creating trace directory
    LOCAL ADD MODE
    Creating OCR keys for user 'root', privgrp 'root'..
    Operation successful.
    Adding daemon to inittab
    CRS-4123: Oracle High Availability Services has been started.
    ohasd is starting
    CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node fred0224, number 1, and is terminating
    An active cluster was found during exclusive startup, restarting to join the cluster
    CRS-2672: Attempting to start 'ora.mdnsd' on 'fred0225'
    CRS-2676: Start of 'ora.mdnsd' on 'fred0225' succeeded
    CRS-2672: Attempting to start 'ora.gipcd' on 'fred0225'
    CRS-2676: Start of 'ora.gipcd' on 'fred0225' succeeded
    CRS-2672: Attempting to start 'ora.gpnpd' on 'fred0225'
    CRS-2676: Start of 'ora.gpnpd' on 'fred0225' succeeded
    CRS-2672: Attempting to start 'ora.cssdmonitor' on 'fred0225'
    CRS-2676: Start of 'ora.cssdmonitor' on 'fred0225' succeeded
    CRS-2672: Attempting to start 'ora.cssd' on 'fred0225'
    CRS-2672: Attempting to start 'ora.diskmon' on 'fred0225'
    CRS-2676: Start of 'ora.diskmon' on 'fred0225' succeeded
    CRS-2676: Start of 'ora.cssd' on 'fred0225' succeeded
    CRS-2672: Attempting to start 'ora.ctssd' on 'fred0225'
    Start action for octssd aborted
    CRS-2676: Start of 'ora.ctssd' on 'fred0225' succeeded
    CRS-2672: Attempting to start 'ora.drivers.acfs' on 'fred0225'
    CRS-2672: Attempting to start 'ora.asm' on 'fred0225'
    CRS-2676: Start of 'ora.drivers.acfs' on 'fred0225' succeeded
    CRS-2676: Start of 'ora.asm' on 'fred0225' succeeded
    CRS-2664: Resource 'ora.ctssd' is already running on 'fred0225'
    CRS-4000: Command Start failed, or completed with errors.
    Command return code of 1 (256) from command: /u01/app/11.2.0/grid/bin/crsctl start resource ora.asm -init
    Start of resource "ora.asm -init" failed
    Failed to start ASM
    Failed to start Oracle Clusterware stack
    In the ocssd.log I found
    [ CSSD][3559689984]clssnmvDHBValidateNCopy: node 1, fred0224, has a disk HB, but no network HB, DHB has rcfg 174483948, wrtcnt, 232, LATS 521702664, lastSeqNo 232, uniqueness 1279039649, timestamp 1279039959/521874274
    In oraagent_oracle.log I found
    [ clsdmc][1212365120]Fail to connect (ADDRESS=(PROTOCOL=ipc)(KEY=fred0225DBG_GPNPD)) with status 9
    2010-07-13 12:54:07.234: [ora.gpnpd][1212365120] [check] Error = error 9 encountered when connecting to GPNPD
    2010-07-13 12:54:07.238: [ora.gpnpd][1212365120] [check] Calling PID check for daemon
    2010-07-13 12:54:07.238: [ora.gpnpd][1212365120] [check] Trying to check PID = 20584
    2010-07-13 12:54:07.432: [ COMMCRS][1285794112]clsc_connect: (0x1304d850) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=fred0225DBG_GPNPD))
    [ clsdmc][1222854976]Fail to connect (ADDRESS=(PROTOCOL=ipc)(KEY=fred0225DBG_MDNSD)) with status 9
    2010-07-13 12:54:08.649: [ora.mdnsd][1222854976] [check] Error = error 9 encountered when connecting to MDNSD
    2010-07-13 12:54:08.649: [ora.mdnsd][1222854976] [check] Calling PID check for daemon
    2010-07-13 12:54:08.649: [ora.mdnsd][1222854976] [check] Trying to check PID = 20571
    2010-07-13 12:54:08.841: [ COMMCRS][1201875264]clsc_connect: (0x12f3b1d0) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=fred0225DBG_MDNSD))
    [ clsdmc][1159915840]Fail to connect (ADDRESS=(PROTOCOL=ipc)(KEY=fred0225DBG_GIPCD)) with status 9
    2010-07-13 12:54:10.051: [ora.gipcd][1159915840] [check] Error = error 9 encountered when connecting to GIPCD
    2010-07-13 12:54:10.051: [ora.gipcd][1159915840] [check] Calling PID check for daemon
    2010-07-13 12:54:10.051: [ora.gipcd][1159915840] [check] Trying to check PID = 20566
    2010-07-13 12:54:10.242: [ COMMCRS][1254324544]clsc_connect: (0x12f35630) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=fred0225DBG_GIPCD))
    In oracssdagent_root.log I found
    2010-07-13 12:52:28.698: [ CSSCLNT][1102481728]clssscConnect: gipc request failed with 29 (0x16)
    2010-07-13 12:52:28.698: [ CSSCLNT][1102481728]clsssInitNative: connect failed, rc 29
    2010-07-13 12:53:55.222: [ CSSCLNT][1102481728]clssnsqlnum: RPC failed rc 3
    2010-07-13 12:53:55.222: [ USRTHRD][1102481728] clsnomon_cssini: failed 3 to fetch node number
    2010-07-13 12:53:55.222: [ USRTHRD][1102481728] clsnomon_init: css init done, nodenum -1.
    2010-07-13 12:53:55.222: [ CSSCLNT][1102481728]clsssRecvMsg: got a disconnect from the server while waiting for message type 43
    2010-07-13 12:53:55.222: [ CSSCLNT][1102481728]clsssGetNLSData: Failure receiving a msg, rc 3
    If you need more info, let me know.

    Well, the error clearly indicates that a communication problem exists on the private interconnect.
    Could this be a setting in ESX, which prevents some communication between the clients on the second network card? Any routing table in ESX not configured correctly?
    Sebastian

  • Root.sh on second node fails

    I am running Linux 64bit. Installing Oracle 11gR2.
    It passed all the per-requisite. Run root.sh on first node. It finished with no errorrs.
    On second node I got the following:
    Running Oracle 11g root.sh script...
    The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME= /u01/app/11.2.0/grid
    Enter the full pathname of the local bin directory: [usr/local/bin]:
    The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n)
    [n]:
    The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
    [n]:
    The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
    [n]:
    Entries will be added to the /etc/oratab file as needed by
    Database Configuration Assistant when a database is created
    Finished running generic part of root.sh script.
    Now product-specific root actions will be performed.
    2010-07-13 12:51:28: Parsing the host name
    2010-07-13 12:51:28: Checking for super user privileges
    2010-07-13 12:51:28: User has super user privileges
    Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
    Creating trace directory
    LOCAL ADD MODE
    Creating OCR keys for user 'root', privgrp 'root'..
    Operation successful.
    Adding daemon to inittab
    CRS-4123: Oracle High Availability Services has been started.
    ohasd is starting
    CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node fred0224, number 1, and is terminating
    An active cluster was found during exclusive startup, restarting to join the cluster
    CRS-2672: Attempting to start 'ora.mdnsd' on 'fred0225'
    CRS-2676: Start of 'ora.mdnsd' on 'fred0225' succeeded
    CRS-2672: Attempting to start 'ora.gipcd' on 'fred0225'
    CRS-2676: Start of 'ora.gipcd' on 'fred0225' succeeded
    CRS-2672: Attempting to start 'ora.gpnpd' on 'fred0225'
    CRS-2676: Start of 'ora.gpnpd' on 'fred0225' succeeded
    CRS-2672: Attempting to start 'ora.cssdmonitor' on 'fred0225'
    CRS-2676: Start of 'ora.cssdmonitor' on 'fred0225' succeeded
    CRS-2672: Attempting to start 'ora.cssd' on 'fred0225'
    CRS-2672: Attempting to start 'ora.diskmon' on 'fred0225'
    CRS-2676: Start of 'ora.diskmon' on 'fred0225' succeeded
    CRS-2676: Start of 'ora.cssd' on 'fred0225' succeeded
    CRS-2672: Attempting to start 'ora.ctssd' on 'fred0225'
    Start action for octssd aborted
    CRS-2676: Start of 'ora.ctssd' on 'fred0225' succeeded
    CRS-2672: Attempting to start 'ora.drivers.acfs' on 'fred0225'
    CRS-2672: Attempting to start 'ora.asm' on 'fred0225'
    CRS-2676: Start of 'ora.drivers.acfs' on 'fred0225' succeeded
    CRS-2676: Start of 'ora.asm' on 'fred0225' succeeded
    CRS-2664: Resource 'ora.ctssd' is already running on 'fred0225'
    CRS-4000: Command Start failed, or completed with errors.
    Command return code of 1 (256) from command: /u01/app/11.2.0/grid/bin/crsctl start resource ora.asm -init
    Start of resource "ora.asm -init" failed
    Failed to start ASM
    Failed to start Oracle Clusterware stack
    In the ocssd.log I found
    [    CSSD][3559689984]clssnmvDHBValidateNCopy: node 1, fred0224, has a disk HB, but no network HB, DHB has rcfg 174483948, wrtcnt, 232, LATS 521702664, lastSeqNo 232, uniqueness 1279039649, timestamp 1279039959/521874274
    In oraagent_oracle.log I found
    [  clsdmc][1212365120]Fail to connect (ADDRESS=(PROTOCOL=ipc)(KEY=fred0225DBG_GPNPD)) with status 9
    2010-07-13 12:54:07.234: [ora.gpnpd][1212365120] [check] Error = error 9 encountered when connecting to GPNPD
    2010-07-13 12:54:07.238: [ora.gpnpd][1212365120] [check] Calling PID check for daemon
    2010-07-13 12:54:07.238: [ora.gpnpd][1212365120] [check] Trying to check PID = 20584
    2010-07-13 12:54:07.432: [ COMMCRS][1285794112]clsc_connect: (0x1304d850) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=fred0225DBG_GPNPD))
    [  clsdmc][1222854976]Fail to connect (ADDRESS=(PROTOCOL=ipc)(KEY=fred0225DBG_MDNSD)) with status 9
    2010-07-13 12:54:08.649: [ora.mdnsd][1222854976] [check] Error = error 9 encountered when connecting to MDNSD
    2010-07-13 12:54:08.649: [ora.mdnsd][1222854976] [check] Calling PID check for daemon
    2010-07-13 12:54:08.649: [ora.mdnsd][1222854976] [check] Trying to check PID = 20571
    2010-07-13 12:54:08.841: [ COMMCRS][1201875264]clsc_connect: (0x12f3b1d0) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=fred0225DBG_MDNSD))
    [  clsdmc][1159915840]Fail to connect (ADDRESS=(PROTOCOL=ipc)(KEY=fred0225DBG_GIPCD)) with status 9
    2010-07-13 12:54:10.051: [ora.gipcd][1159915840] [check] Error = error 9 encountered when connecting to GIPCD
    2010-07-13 12:54:10.051: [ora.gipcd][1159915840] [check] Calling PID check for daemon
    2010-07-13 12:54:10.051: [ora.gipcd][1159915840] [check] Trying to check PID = 20566
    2010-07-13 12:54:10.242: [ COMMCRS][1254324544]clsc_connect: (0x12f35630) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=fred0225DBG_GIPCD))
    In oracssdagent_root.log I found
    2010-07-13 12:52:28.698: [ CSSCLNT][1102481728]clssscConnect: gipc request failed with 29 (0x16)
    2010-07-13 12:52:28.698: [ CSSCLNT][1102481728]clsssInitNative: connect failed, rc 29
    2010-07-13 12:53:55.222: [ CSSCLNT][1102481728]clssnsqlnum: RPC failed rc 3
    2010-07-13 12:53:55.222: [ USRTHRD][1102481728] clsnomon_cssini: failed 3 to fetch node number
    2010-07-13 12:53:55.222: [ USRTHRD][1102481728] clsnomon_init: css init done, nodenum -1.
    2010-07-13 12:53:55.222: [ CSSCLNT][1102481728]clsssRecvMsg: got a disconnect from the server while waiting for message type 43
    2010-07-13 12:53:55.222: [ CSSCLNT][1102481728]clsssGetNLSData: Failure receiving a msg, rc 3
    If anyone needs more info please let me know.

    On all nodes,
    1. Modify the /etc/sysconfig/oracleasm with:
    ORACLEASM_SCANORDER="dm"
    ORACLEASM_SCANEXCLUDE="sd"
    2. restart the asmlib by :
    # /etc/init.d/oracleasm restart
    3. Run root.sh on the 2nd node
    hope this helps you

  • 11gr2 crsd core dump during failover or start attempt on second node

    Hi,
    I installed 11gr2 with ASM on one node (solaris SPARC). Then I added another node to this cluster (via addNode.sh script).
    Than got strange error: If my first node is up, second node is started fine and run well. If I shutdown first node - crsd on second node dump to core and fails to restart. I get the same error if I try to start second node when the first one is down.
    In the crsd.log I see the following:
    [  clsdmt][2]Listening to (ADDRESS=(PROTOCOL=ipc)(KEY=mskbkp2DBG_CRSD))
    2010-03-03 17:31:35.330: [  clsdmt][2]PID for the Process [18669], connkey 1
    2010-03-03 17:31:35.331: [  clsdmt][2]Creating PID [18669] file for home /u01/grid/11.2.0 host mskbkp2 bin crs to /u01/grid/11
    .2.0/crs/init/
    2010-03-03 17:31:35.331: [  clsdmt][2]Writing PID [18669] to the file [u01/grid/11.2.0/crs/init/mskbkp2.pid]
    2010-03-03 17:31:35.925: [ default][1] CRS Daemon Starting
    2010-03-03 17:31:35.933: [ default][1] ENV Logging level for Module: AGENT 1
    2010-03-03 17:31:35.934: [ default][1] ENV Logging level for Module: AGFW 0
    2010-03-03 17:31:35.934: [ default][1] ENV Logging level for Module: CLSFRAME 0
    2010-03-03 17:31:35.934: [ default][1] ENV Logging level for Module: CLSVER 0
    2010-03-03 17:31:35.934: [ default][1] ENV Logging level for Module: CLUCLS 0
    2010-03-03 17:31:35.934: [ default][1] ENV Logging level for Module: COMMCRS 0
    2010-03-03 17:31:35.934: [ default][1] ENV Logging level for Module: COMMNS 0
    2010-03-03 17:31:35.936: [ default][1] ENV Logging level for Module: CRSAPP 0
    2010-03-03 17:31:35.936: [ default][1] ENV Logging level for Module: CRSCCL 0
    2010-03-03 17:31:35.936: [ default][1] ENV Logging level for Module: CRSCEVT 0
    2010-03-03 17:31:35.936: [ default][1] ENV Logging level for Module: CRSCOMM 1
    2010-03-03 17:31:35.936: [    CRSD][1] ENV Debug Level(CRSD): 50
    2010-03-03 17:31:35.936: [    CRSD][1] ENV Logging level for Module: CRSD 50
    2010-03-03 17:31:35.937: [    CRSD][1] ENV Debug Level(CRSEVT): 0
    2010-03-03 17:31:35.937: [    CRSD][1] ENV Logging level for Module: CRSEVT 0
    2010-03-03 17:31:35.937: [    CRSD][1] ENV Debug Level(CRSMAIN): 1
    2010-03-03 17:31:35.937: [    CRSD][1] ENV Logging level for Module: CRSMAIN 1
    2010-03-03 17:31:35.937: [    CRSD][1] ENV Debug Level(CRSOCR): 0
    2010-03-03 17:31:35.937: [    CRSD][1] ENV Logging level for Module: CRSOCR 0
    2010-03-03 17:31:35.939: [    CRSD][1] ENV Debug Level(CRSPE): 0
    2010-03-03 17:31:35.939: [    CRSD][1] ENV Logging level for Module: CRSPE 0
    2010-03-03 17:31:35.939: [    CRSD][1] ENV Debug Level(CRSPLACE): 0
    2010-03-03 17:31:35.939: [    CRSD][1] ENV Logging level for Module: CRSPLACE 0
    2010-03-03 17:31:35.940: [    CRSD][1] ENV Debug Level(CRSRES): 0
    2010-03-03 17:31:35.940: [    CRSD][1] ENV Logging level for Module: CRSRES 0
    2010-03-03 17:31:35.940: [    CRSD][1] ENV Debug Level(CRSRPT): 0
    2010-03-03 17:31:35.940: [    CRSD][1] ENV Logging level for Module: CRSRPT 0
    2010-03-03 17:31:35.940: [    CRSD][1] ENV Debug Level(CRSRTI): 0
    2010-03-03 17:31:35.940: [    CRSD][1] ENV Logging level for Module: CRSRTI 0
    2010-03-03 17:31:35.940: [    CRSD][1] ENV Debug Level(CRSSE): 0
    2010-03-03 17:31:35.941: [    CRSD][1] ENV Logging level for Module: CRSSE 0
    2010-03-03 17:31:35.941: [    CRSD][1] ENV Debug Level(CRSSEC): 0
    2010-03-03 17:31:35.941: [    CRSD][1] ENV Logging level for Module: CRSSEC 0
    2010-03-03 17:31:35.941: [    CRSD][1] ENV Debug Level(CRSSHARED): 0
    2010-03-03 17:31:35.941: [    CRSD][1] ENV Logging level for Module: CRSSHARED 0
    2010-03-03 17:31:35.942: [    CRSD][1] ENV Debug Level(CRSTIMER): 0
    2010-03-03 17:31:35.942: [    CRSD][1] ENV Logging level for Module: CRSTIMER 0
    2010-03-03 17:31:35.942: [    CRSD][1] ENV Debug Level(CRSUI): 0
    2010-03-03 17:31:35.942: [    CRSD][1] ENV Logging level for Module: CRSUI 0
    2010-03-03 17:31:35.942: [    CRSD][1] ENV Debug Level(CSSCLNT): 0
    2010-03-03 17:31:35.942: [    CRSD][1] ENV Logging level for Module: CSSCLNT 0
    2010-03-03 17:31:35.942: [    CRSD][1] ENV Debug Level(OCRAPI): 1
    2010-03-03 17:31:35.942: [    CRSD][1] ENV Logging level for Module: OCRAPI 1
    2010-03-03 17:31:35.942: [    CRSD][1] ENV Debug Level(OCRASM): 1
    2010-03-03 17:31:35.943: [    CRSD][1] ENV Logging level for Module: OCRASM 1
    2010-03-03 17:31:35.943: [    CRSD][1] ENV Debug Level(OCRCAC): 1
    2010-03-03 17:31:35.943: [    CRSD][1] ENV Logging level for Module: OCRCAC 1
    2010-03-03 17:31:35.943: [    CRSD][1] ENV Debug Level(OCRCLI): 1
    2010-03-03 17:31:35.943: [    CRSD][1] ENV Logging level for Module: OCRCLI 1
    2010-03-03 17:31:35.943: [    CRSD][1] ENV Debug Level(OCRMAS): 1
    2010-03-03 17:31:35.943: [    CRSD][1] ENV Logging level for Module: OCRMAS 1
    2010-03-03 17:31:35.944: [    CRSD][1] ENV Debug Level(OCRMSG): 1
    2010-03-03 17:31:35.944: [    CRSD][1] ENV Logging level for Module: OCRMSG 1
    2010-03-03 17:31:35.945: [    CRSD][1] ENV Debug Level(OCROSD): 1
    2010-03-03 17:31:35.945: [    CRSD][1] ENV Logging level for Module: OCROSD 1
    2010-03-03 17:31:35.945: [    CRSD][1] ENV Debug Level(OCRRAW): 1
    2010-03-03 17:31:35.945: [    CRSD][1] ENV Logging level for Module: OCRRAW 1
    2010-03-03 17:31:35.945: [    CRSD][1] ENV Debug Level(OCRSRV): 1
    2010-03-03 17:31:35.945: [    CRSD][1] ENV Logging level for Module: OCRSRV 1
    2010-03-03 17:31:35.945: [    CRSD][1] ENV Debug Level(OCRUTL): 1
    2010-03-03 17:31:35.945: [    CRSD][1] ENV Logging level for Module: OCRUTL 1
    2010-03-03 17:31:35.945: [    CRSD][1] ENV Debug Level(SuiteTes): 1
    2010-03-03 17:31:35.946: [    CRSD][1] ENV Logging level for Module: SuiteTes 1
    2010-03-03 17:31:35.946: [    CRSD][1] ENV Debug Level(UiServer): 0
    2010-03-03 17:31:35.946: [    CRSD][1] ENV Logging level for Module: UiServer 0
    2010-03-03 17:31:35.946: [ CRSMAIN][1] Checking the OCR device
    2010-03-03 17:31:35.948: [ CRSMAIN][1] Connecting to the CSS Daemon
    2010-03-03 17:31:35.976: [ CRSMAIN][1] Initializing OCR
    2010-03-03 17:31:35.981: [  OCRAPI][1]clsu_get_private_ip_addr: Calling clsu_get_private_ip_addresses to get first private ip
    2010-03-03 17:31:35.981: [  OCRAPI][1]Check namebufs
    2010-03-03 17:31:35.981: [  OCRAPI][1]Finished checking namebufs
    2010-03-03 17:31:35.982: [    GIPC][1] gipcCheckInitialization: possible incompatible non-threaded init from [clsinet.c : 3232
    ], original from [clsss.c : 5026]
    2010-03-03 17:31:36.036: [    GPnP][1]clsgpnp_Init: [at clsgpnp0.c:405] gpnp tracelevel 3, component tracelevel 0
    2010-03-03 17:31:36.037: [    GPnP][1]clsgpnp_Init: [at clsgpnp0.c:535] '/u01/grid/11.2.0' in effect as GPnP home base.
    2010-03-03 17:31:36.059: [    GIPC][1] gipcCheckInitialization: possible incompatible non-threaded init from [clsgpnp0.c : 680
    ], original from [clsss.c : 5026]
    2010-03-03 17:31:36.067: [    GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3867] Init gpnp local security key providers (2)
    fatal if both fail
    2010-03-03 17:31:36.068: [    GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3870] Init gpnp local security key proveders 1 o
    f 2: file wallet (LSKP-FSW)
    2010-03-03 17:31:36.068: [    GPnP][1]clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:398] Using FS Wallet Location : /u01/grid/11.2.0/
    gpnp/mskbkp2/wallets/peer/
    2010-03-03 17:31:36.069: [    GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3892] Init gpnp local security key provider 1 of
    2: file wallet (LSKP-FSW) OK
    2010-03-03 17:31:36.069: [    GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3898] Init gpnp local security key proveders 2 o
    f 2: OLR wallet (LSKP-CLSW-OLR)
    [   CLWAL][1]clsw_Initialize: OLR initlevel [30000]
    2010-03-03 17:31:36.080: [    GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3921] Init gpnp local security key provider 2 of
    2: OLR wallet (LSKP-CLSW-OLR) OK
    2010-03-03 17:31:36.081: [    GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1952] <Get gpnp security keys (wallet) for id:1,typ;7. (2
    providers - fatal if all fail)
    2010-03-03 17:31:36.081: [    GPnP][1]clsgpnpkwf_getWalletPath: [at clsgpnpkwf.c:501] req_id=1 ck_prov_id=1 wallet path: /u01/
    grid/11.2.0/gpnp/mskbkp2/wallets/peer/
    2010-03-03 17:31:36.152: [    GPnP][1]clsgpnpwu_walletfopen: [at clsgpnpwu.c:496] Opened SSO wallet: '/u01/grid/11.2.0/gpnp/ms
    kbkp2/wallets/peer/cwallet.sso'
    2010-03-03 17:31:36.152: [    GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1968] Result: (0) CLSGPNP_OK. Get gpnp wallet - provider 1
    of 2 (LSKP-FSW(1))
    2010-03-03 17:31:36.152: [    GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1982] Got gpnp security keys (wallet).>
    2010-03-03 17:31:36.172: [    GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1952] <Get gpnp security keys (wallet) for id:1,typ;4. (2
    providers - fatal if all fail)
    2010-03-03 17:31:36.172: [    GPnP][1]clsgpnpkwf_getWalletPath: [at clsgpnpkwf.c:501] req_id=1 ck_prov_id=1 wallet path: /u01/
    grid/11.2.0/gpnp/mskbkp2/wallets/peer/
    2010-03-03 17:31:36.238: [    GPnP][1]clsgpnpwu_walletfopen: [at clsgpnpwu.c:496] Opened SSO wallet: '/u01/grid/11.2.0/gpnp/ms
    kbkp2/wallets/peer/cwallet.sso'
    2010-03-03 17:31:36.239: [    GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1968] Result: (0) CLSGPNP_OK. Get gpnp wallet - provider 1
    of 2 (LSKP-FSW(1))
    2010-03-03 17:31:36.239: [    GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1982] Got gpnp security keys (wallet).>
    2010-03-03 17:31:36.239: [    GPnP][1]clsgpnp_Init: [at clsgpnp0.c:840] GPnP client pid=18669, tl=3, f=0
    2010-03-03 17:31:36.541: [GIPCXCPT][1] gipcShutdownF: skipping shutdown, count 2, from [ clsinet.c : 1735], ret gipcretSuccess
    (0)
    2010-03-03 17:31:36.552: [GIPCXCPT][1] gipcShutdownF: skipping shutdown, count 1, from [ clsgpnp0.c : 1021], ret gipcretSucces
    s (0)
    2010-03-03 17:31:36.771: [  OCRRAW][1]proprioo: for disk 0 (+DR2_BIN), id match (1), total id sets, (1) need recover (0), my v
    otes (0), total votes (0), commit_lsn (9), lsn (9)
    2010-03-03 17:31:36.771: [  OCRRAW][1]proprioo: my id set: (833490748, 1028247821, 0, 0, 0)
    2010-03-03 17:31:36.772: [  OCRRAW][1]proprioo: 1st set: (833490748, 1028247821, 0, 0, 0)
    2010-03-03 17:31:36.772: [  OCRRAW][1]proprioo: 2nd set: (0, 0, 0, 0, 0)
    2010-03-03 17:31:36.830: [  OCRSRV][1]th_init: Successfully retrieved CSS misscount [31].
    2010-03-03 17:31:36.830: [  OCRSRV][1]th_init: Successfully query CLSS mode [3].
    [  OCRMAS][20]th_calc_av:5': Rturn persisted AV [186646784] [11.2.0.1.0]
    2010-03-03 17:31:36.920: [  OCRSRV][20]th_not_master_change: Master change callback not registered
    2010-03-03 17:31:36.920: [  OCRMAS][20]th_master:12: I AM THE NEW OCR MASTER at incar 1. Node Number 2
    2010-03-03 17:31:37.134: [  OCRASM][20]proprasmo: ASM cache size is [5MB]
    2010-03-03 17:31:37.142: [  OCRASM][20]proprasmo: ASM cache [5MB] enabled for disk group [DR2_BIN].
    2010-03-03 17:31:37.155: [  OCRRAW][20]proprioo: for disk 0 (+DR2_BIN), id match (1), total id sets, (1) need recover (0), my
    votes (0), total votes (0), commit_lsn (9), lsn (9)
    2010-03-03 17:31:37.155: [  OCRRAW][20]proprioo: my id set: (833490748, 1028247821, 0, 0, 0)
    2010-03-03 17:31:37.155: [  OCRRAW][20]proprioo: 1st set: (833490748, 1028247821, 0, 0, 0)
    2010-03-03 17:31:37.155: [  OCRRAW][20]proprioo: 2nd set: (0, 0, 0, 0, 0)
    2010-03-03 17:31:37.214: [  OCRMAS][20]proath_master:18: Spawned connection mgr thread
    2010-03-03 17:31:37.214: [  OCRMAS][20]proath_master:20: Spawned upgrade thread
    2010-03-03 17:31:37.214: [  OCRMAS][20]th_master:19.1: Wake up upgrade thread
    2010-03-03 17:31:37.216: [  OCRSRV][1]th_snap_local_spawn: Inside snap local spawn. host is [mskbkp2]
    2010-03-03 17:31:37.219: [ CRSMAIN][1] Running as user: root
    2010-03-03 17:31:37.219: [ CRSMAIN][1] CRSD running as the Privileged user
    2010-03-03 17:31:37.219: [  CLSVER][1] Static Version 11.2.0.1.0
    2010-03-03 17:31:37.226: [  OCRMAS][20]th_master:1': Recvd pubdata event from node [2]
    2010-03-03 17:31:37.227: [  OCRMAS][20]th_master:2': Recvd pubdata event for self. Do nothing.
    2010-03-03 17:31:37.227: [  CLSVER][1] Daemon version: 11.2.0.1.0 Software version: 11.2.0.1.0
    2010-03-03 17:31:37.231: [  CLSVER][1] Active Version from OCR:11.2.0.1.0
    2010-03-03 17:31:37.232: [  CLSVER][1] Active Version and Software Version are same
    2010-03-03 17:31:37.232: [  CLSVER][1] Active Version changed to 11.2.0.1.0
    2010-03-03 17:31:37.232: [  OCRSRV][1]th_reg_master_change: Master change callback registered
    2010-03-03 17:31:37.232: [  OCRAPI][1]a_reg_master_change: Registered master change callback
    2010-03-03 17:31:37.232: [  OCRSRV][1]th_not_master_change: Invoking master change callback. Master [2] Inc [1]
    2010-03-03 17:31:37.232: [  OCRAPI][1]a_reg_master_change: Notified master change
    2010-03-03 17:31:37.232: [ CRSMAIN][1] CAA Node Group Pri Data size: 128
    2010-03-03 17:31:37.233: [ CRSMAIN][1] CAA Node Group Pub Data size: 128
    2010-03-03 17:31:37.247: [ CRSMAIN][1] Getting private data of booted nodes
    2010-03-03 17:31:37.247: [ CRSMAIN][1] Checking for booted param on nodenum: 2
    2010-03-03 17:31:37.306: [    CLSE][1]clse_get_auth_loc: Returning default authloc: /u01/grid/11.2.0/auth/crs/mskbkp2
    2010-03-03 17:31:37.306: [ CRSMAIN][1] Using Authorizer location: /u01/grid/11.2.0/auth/crs/mskbkp2
    2010-03-03 17:31:37.314: [  OCRSRV][23]th_upgrade: Starting upgrade calculation
    2010-03-03 17:31:37.364: [  CLSCLU][1]clsclu_init: rc 0
    2010-03-03 17:31:37.381: [  OCRSRV][23]th_upgrade:10.1 AV [186646784]. State [11]. Already upgraded.Updated global data to the
    crs version group. Return [0]
    2010-03-03 17:31:37.385: [ CRSMAIN][1] Initializing RTI
    2010-03-03 17:31:37.433: [ CRSMAIN][1] Initializing ResouceStateListener
    2010-03-03 17:31:37.433: [CRSTIMER][37] Timer Thread Starting.
    2010-03-03 17:31:37.433: [ CRSMAIN][1] Initializing EVMMgr
    2010-03-03 17:31:37.446: [ CRSMAIN][1] Initializing ResourceMap Map
    2010-03-03 17:31:37.461: [ CRSMAIN][1] Subscribing to EVM events for apps
    2010-03-03 17:31:37.504: [ CRSMAIN][1] CRSD locked during state recovery, please wait.
    2010-03-03 17:31:37.516: [ CRSMAIN][1] CRSD recovered, unlocked.
    2010-03-03 17:31:37.525: [ default][1]clsu_get_private_ip_addr: Calling clsu_get_private_ip_addresses to get first private ip
    2010-03-03 17:31:37.525: [ default][1]Check namebufs
    2010-03-03 17:31:37.525: [ default][1]Finished checking namebufs
    2010-03-03 17:31:37.526: [    GIPC][1] gipcCheckInitialization: possible incompatible non-threaded init from [clsinet.c : 3232
    ], original from [clsss.c : 5026]
    2010-03-03 17:31:37.569: [    GPnP][1]clsgpnp_Init: [at clsgpnp0.c:405] gpnp tracelevel 3, component tracelevel 0
    2010-03-03 17:31:37.569: [    GPnP][1]clsgpnp_Init: [at clsgpnp0.c:535] '/u01/grid/11.2.0' in effect as GPnP home base.
    2010-03-03 17:31:37.587: [    GIPC][1] gipcCheckInitialization: possible incompatible non-threaded init from [clsgpnp0.c : 680
    ], original from [clsss.c : 5026]
    2010-03-03 17:31:37.595: [    GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3867] Init gpnp local security key providers (2)
    fatal if both fail
    2010-03-03 17:31:37.595: [    GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3870] Init gpnp local security key proveders 1 o
    f 2: file wallet (LSKP-FSW)
    2010-03-03 17:31:37.596: [    GPnP][1]clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:398] Using FS Wallet Location : /u01/grid/11.2.0/
    gpnp/mskbkp2/wallets/peer/
    2010-03-03 17:31:37.596: [    GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3892] Init gpnp local security key provider 1 of
    2: file wallet (LSKP-FSW) OK
    2010-03-03 17:31:37.596: [    GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3898] Init gpnp local security key proveders 2 o
    f 2: OLR wallet (LSKP-CLSW-OLR)
    [   CLWAL][1]clsw_Initialize: OLR initlevel [30000]
    2010-03-03 17:31:37.607: [    GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3921] Init gpnp local security key provider 2 of
    2: OLR wallet (LSKP-CLSW-OLR) OK
    2010-03-03 17:31:37.607: [    GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1952] <Get gpnp security keys (wallet) for id:1,typ;7. (2
    providers - fatal if all fail)
    2010-03-03 17:31:37.607: [    GPnP][1]clsgpnpkwf_getWalletPath: [at clsgpnpkwf.c:501] req_id=1 ck_prov_id=1 wallet path: /u01/
    grid/11.2.0/gpnp/mskbkp2/wallets/peer/
    2010-03-03 17:31:37.673: [    GPnP][1]clsgpnpwu_walletfopen: [at clsgpnpwu.c:496] Opened SSO wallet: '/u01/grid/11.2.0/gpnp/ms
    kbkp2/wallets/peer/cwallet.sso'
    2010-03-03 17:31:37.673: [    GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1968] Result: (0) CLSGPNP_OK. Get gpnp wallet - provider 1
    of 2 (LSKP-FSW(1))
    2010-03-03 17:31:37.673: [    GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1982] Got gpnp security keys (wallet).>
    2010-03-03 17:31:37.690: [    GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1952] <Get gpnp security keys (wallet) for id:1,typ;4. (2
    providers - fatal if all fail)
    2010-03-03 17:31:37.690: [    GPnP][1]clsgpnpkwf_getWalletPath: [at clsgpnpkwf.c:501] req_id=1 ck_prov_id=1 wallet path: /u01/
    grid/11.2.0/gpnp/mskbkp2/wallets/peer/
    2010-03-03 17:31:37.754: [    GPnP][1]clsgpnpwu_walletfopen: [at clsgpnpwu.c:496] Opened SSO wallet: '/u01/grid/11.2.0/gpnp/ms
    kbkp2/wallets/peer/cwallet.sso'
    2010-03-03 17:31:37.754: [    GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1968] Result: (0) CLSGPNP_OK. Get gpnp wallet - provider 1
    of 2 (LSKP-FSW(1))
    2010-03-03 17:31:37.754: [    GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1982] Got gpnp security keys (wallet).>
    2010-03-03 17:31:37.755: [    GPnP][1]clsgpnp_Init: [at clsgpnp0.c:840] GPnP client pid=18669, tl=3, f=0
    2010-03-03 17:31:37.806: [GIPCXCPT][1] gipcShutdownF: skipping shutdown, count 2, from [ clsinet.c : 1735], ret gipcretSuccess
    (0)
    2010-03-03 17:31:37.817: [GIPCXCPT][1] gipcShutdownF: skipping shutdown, count 1, from [ clsgpnp0.c : 1021], ret gipcretSucces
    s (0)
    2010-03-03 17:31:37.822: [ CRSMAIN][1] CRSD listening on 10 style E2E port (ADDRESS=(PROTOCOL=tcp)(HOST=172.31.25.112)(PORT=38
    983))
    2010-03-03 17:31:37.835: [ CRSMAIN][1] Starting Threads
    2010-03-03 17:31:37.858: [    CLSE][1]clse_get_auth_loc: Returning default authloc: /u01/grid/11.2.0/auth/crs/mskbkp2
    2010-03-03 17:31:37.858: [    CRSD][1] AuthLoc /u01/grid/11.2.0/auth/crs/mskbkp2
    2010-03-03 17:31:37.859: [    CRSD][1] PE active version: 11.2.0.1.0
    2010-03-03 17:31:37.859: [    CRSD][1] PE Engine: NEW
    2010-03-03 17:31:37.859: [    CRSD][1] Using OCR batch ops : ENABLED
    2010-03-03 17:31:37.860: [ CRSMAIN][1] Initializing Node Down Monitor
    2010-03-03 17:31:37.860: [ CRSMAIN][1] CRS Daemon Started.
    2010-03-03 17:31:37.860: [    CRSD][1] Connecting to the CSS Daemon
    2010-03-03 17:31:37.861: [    CRSD][1] Local CSS Node Number is: 2
    2010-03-03 17:31:37.863: [    CRSD][1] Local Css Node Name is: mskbkp2
    2010-03-03 17:31:37.863: [    CRSD][1] CRSDPersonality initialized
    2010-03-03 17:31:37.864: [ CRSMAIN][1] Process member data: CRSD:mskbkp2
    2010-03-03 17:31:37.864: [    CRSD][1][F-ALGO] getIpcPath returning (ADDRESS=(PROTOCOL=IPC)(KEY=CRSD_IPC_SOCKET_11))
    2010-03-03 17:31:37.865: [CLSFRAME][1] Inited lsf context 102b3f670
    2010-03-03 17:31:37.865: [CLSFRAME][1] Initing CLS Framework messaging
    2010-03-03 17:31:37.869: [    CRSD][1][F-ALGO] getIpcPath returning (ADDRESS=(PROTOCOL=IPC)(KEY=CRSD_IPC_SOCKET_11))
    2010-03-03 17:31:37.873: [UiServer][1] UI Comms initalize() 1
    2010-03-03 17:31:37.873: [CLSFRAME][1] New Framework state: 2
    2010-03-03 17:31:37.873: [CLSFRAME][1] M2M is starting...
    2010-03-03 17:31:37.873: [  CRSCCL][1]clsCclInit called by process: 18669
    2010-03-03 17:31:37.885: [  CRSCCL][1]USING CLSC ============
    2010-03-03 17:31:37.895: [ default][1]clsu_get_private_ip_addr: Calling clsu_get_private_ip_addresses to get first private ip
    2010-03-03 17:31:37.895: [ default][1]Check namebufs
    2010-03-03 17:31:37.895: [ default][1]Finished checking namebufs
    2010-03-03 17:31:37.950: [    GPnP][1]clsgpnp_Init: [at clsgpnp0.c:405] gpnp tracelevel 3, component tracelevel 0
    2010-03-03 17:31:37.951: [    GPnP][1]clsgpnp_Init: [at clsgpnp0.c:535] '/u01/grid/11.2.0' in effect as GPnP home base.
    2010-03-03 17:31:37.970: [    GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3867] Init gpnp local security key providers (2)
    fatal if both fail
    2010-03-03 17:31:37.970: [    GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3870] Init gpnp local security key proveders 1 o
    f 2: file wallet (LSKP-FSW)
    2010-03-03 17:31:37.970: [    GPnP][1]clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:398] Using FS Wallet Location : /u01/grid/11.2.0/
    gpnp/mskbkp2/wallets/peer/
    2010-03-03 17:31:37.970: [    GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3892] Init gpnp local security key provider 1 of
    2: file wallet (LSKP-FSW) OK
    2010-03-03 17:31:37.970: [    GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3898] Init gpnp local security key proveders 2 o
    f 2: OLR wallet (LSKP-CLSW-OLR)
    [   CLWAL][1]clsw_Initialize: OLR initlevel [70000]
    2010-03-03 17:31:37.980: [    GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3921] Init gpnp local security key provider 2 of
    2: OLR wallet (LSKP-CLSW-OLR) OK
    2010-03-03 17:31:37.980: [    GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1952] <Get gpnp security keys (wallet) for id:1,typ;7. (2
    providers - fatal if all fail)
    2010-03-03 17:31:37.980: [    GPnP][1]clsgpnpkwf_getWalletPath: [at clsgpnpkwf.c:501] req_id=1 ck_prov_id=1 wallet path: /u01/
    grid/11.2.0/gpnp/mskbkp2/wallets/peer/
    2010-03-03 17:31:38.049: [    GPnP][1]clsgpnpwu_walletfopen: [at clsgpnpwu.c:496] Opened SSO wallet: '/u01/grid/11.2.0/gpnp/ms
    kbkp2/wallets/peer/cwallet.sso'
    2010-03-03 17:31:38.049: [    GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1968] Result: (0) CLSGPNP_OK. Get gpnp wallet - provider 1
    of 2 (LSKP-FSW(1))
    2010-03-03 17:31:38.049: [    GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1982] Got gpnp security keys (wallet).>
    2010-03-03 17:31:38.068: [    GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1952] <Get gpnp security keys (wallet) for id:1,typ;4. (2
    providers - fatal if all fail)
    2010-03-03 17:31:38.068: [    GPnP][1]clsgpnpkwf_getWalletPath: [at clsgpnpkwf.c:501] req_id=1 ck_prov_id=1 wallet path: /u01/
    grid/11.2.0/gpnp/mskbkp2/wallets/peer/
    2010-03-03 17:31:38.134: [    GPnP][1]clsgpnpwu_walletfopen: [at clsgpnpwu.c:496] Opened SSO wallet: '/u01/grid/11.2.0/gpnp/ms
    kbkp2/wallets/peer/cwallet.sso'
    2010-03-03 17:31:38.135: [    GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1968] Result: (0) CLSGPNP_OK. Get gpnp wallet - provider 1
    of 2 (LSKP-FSW(1))
    2010-03-03 17:31:38.135: [    GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1982] Got gpnp security keys (wallet).>
    2010-03-03 17:31:38.135: [    GPnP][1]clsgpnp_Init: [at clsgpnp0.c:840] GPnP client pid=18669, tl=3, f=3
    2010-03-03 17:31:38.184: [GIPCXCPT][1] gipcShutdownF: skipping shutdown, count 2, from [ clsinet.c : 1735], ret gipcretSuccess
    (0)
    2010-03-03 17:31:38.194: [GIPCXCPT][1] gipcShutdownF: skipping shutdown, count 1, from [ clsgpnp0.c : 1021], ret gipcretSucces
    s (0)
    2010-03-03 17:31:38.200: [  CRSCCL][1]Listening endpoint created sucessfully @ (ADDRESS=(PROTOCOL=tcp)(DEV=54)(HOST=172.31.25.
    112)(PORT=38984)).con = 10359a0d0
    2010-03-03 17:31:38.209: [  CRSCCL][48]CSS Group Registration complete.
    2010-03-03 17:31:38.213: [  CRSCCL][48]cclGetMemberData called
    2010-03-03 17:31:38.215: [  CRSCCL][48]Obtained first membership map.
    2010-03-03 17:31:38.215: [  CRSCCL][48]Dumping member data ------------------
    2010-03-03 17:31:38.215: [  CRSCCL][48]Member (2, 603412550) on node port=.
    2010-03-03 17:31:38.216: [  CRSCCL][48]Done ------------------
    2010-03-03 17:31:38.216: [  CRSCCL][48]Waiting for reconfigs
    2010-03-03 17:31:38.216: [  CRSCCL][49]cclCommunicationHandler started.
    2010-03-03 17:31:38.220: [ CRSCOMM][1] Ipc: m_pClscCtx=1020c4850m_pUgblm=1035b2a50
    2010-03-03 17:31:38.220: [ CRSCOMM][1] Ipc: Starting send thread
    2010-03-03 17:31:38.220: [ CRSCOMM][1] IpcL: Listener instantiated for: (ADDRESS=(PROTOCOL=IPC)(KEY=CRSD_IPC_SOCKET_11))
    2010-03-03 17:31:38.221: [ CRSCOMM][52] Ipc: sendWork thread started.
    2010-03-03 17:31:38.222: [ CRSCOMM][1] IpcL: Listener started listening.
    2010-03-03 17:31:38.223: [ CRSCOMM][53] IpcL: thread started listening
    2010-03-03 17:31:38.223: [CLSFRAME][1] Starting thread model named: AgfwProxySrvTM
    2010-03-03 17:31:38.224: [CLSFRAME][1] Starting thread model named: OcrModuleTM
    2010-03-03 17:31:38.225: [CLSFRAME][1] Starting thread model named: PolicyEngineTM
    2010-03-03 17:31:38.225: [CLSFRAME][1] Starting thread model named: SharedThreadTM
    2010-03-03 17:31:38.225: [CLSFRAME][1] Starting thread model named: UiServerTM
    2010-03-03 17:31:38.225: [CLSFRAME][1] New Framework state: 3
    2010-03-03 17:31:38.227: [  CRSRPT][62] Enabled
    2010-03-03 17:31:38.228: [   CRSPE][61] PE Role|State Update: old role [INVALID] new [INVALID]; old state [Not yet initialized
    ] new [Enabling: waiting for role]
    2010-03-03 17:31:38.229: [   CRSSE][62] Master Change Event; New Master Node ID:2 This Node's ID:2
    2010-03-03 17:31:38.230: [   CRSPE][61] PE Role|State Update: old role [INVALID] new [MASTER]; old state [Enabling: waiting fo
    r role] new [Configuring]
    2010-03-03 17:31:38.230: [   CRSPE][61] PE MASTER NAME: mskbkp2
    2010-03-03 17:31:38.230: [   CRSPE][61] Starting to read configuration
    2010-03-03 17:31:38.260: [   CRSPE][61] Reading (2) servers
    2010-03-03 17:31:38.459: [   CRSPE][61] DM: set global config version to: 150
    2010-03-03 17:31:38.459: [   CRSPE][61] DM: set pool freeze timeout to: 60000
    2010-03-03 17:31:38.459: [   CRSPE][61] DM: Set event seq number to: 13900000
    2010-03-03 17:31:38.459: [   CRSPE][61] DM: Set threshold event seq number to: 13980000
    2010-03-03 17:31:38.460: [   CRSPE][61] Sent request to write event sequence number 14000000 to repository
    2010-03-03 17:31:38.483: [   CRSPE][61] Wrote new event sequence to repository
    2010-03-03 17:31:38.568: [   CRSPE][61] Reading (15) types
    2010-03-03 17:31:38.593: [   CRSPE][61] Reading (3) server pools
    2010-03-03 17:31:38.624: [   CRSPE][61] Reading (21) resources
    2010-03-03 17:31:39.987: [   CRSPE][61] Finished reading configuration. Parsing...
    2010-03-03 17:31:39.988: [   CRSPE][61] Parsing resource types...
    2010-03-03 17:31:40.030: [    CRSD][61] Initializing the config version for type ora.asm.type to: 1
    2010-03-03 17:31:40.035: [    CRSD][61] Initializing the config version for type ora.cluster_resource.type to: 1
    2010-03-03 17:31:40.040: [    CRSD][61] Initializing the config version for type ora.cluster_vip.type to: 1
    2010-03-03 17:31:40.044: [    CRSD][61] Initializing the config version for type ora.cluster_vip_net1.type to: 1
    2010-03-03 17:31:40.048: [    CRSD][61] Dump State Starting ...
    2010-03-03 17:31:40.048: [    CRSD][61] State Dump for RTILock
    2010-03-03 17:31:40.048: [    CRSD][61] Lock State List is busy, skipping ..
    2010-03-03 17:31:40.048: [    CRSD][61] State Dump for Timer
    2010-03-03 17:31:40.049: [    CRSD][61] Timer map size=0
    2010-03-03 17:31:40.049: [   CRSPE][61] Dumping PE Data Model...:DM has [0 resources][0 types][0 servers][0 spools]
    ------------- RESOURCES:
    ------------- TYPES:
    ------------- SERVERS:
    ------------- SERVER POOLS:
    2010-03-03 17:31:40.049: [   CRSPE][61] Dumping ICE contents...:ICE operation count: 0
    2010-03-03 17:31:40.049: [    CRSD][61] Dump State Done.
    I guess that there is some thing wrong in configuration, but cannot find out what.
    Any help would be appreciated.
    Thanks

    Hi,
    Please check your disk attributes and permission of OCR/Voting and other ASM devices. The disk attribute should be changed to be shared among all nodes of cluster. It happened with us in 10.2.0.4 where disk was not shared and we were able to start crs from only one node at a time so please check disk attributes. Please see blog keyurmakwanacrs.blogspot.com for AIX which we faced. Not surle whether you've similar problem or not. We had 10.2.0.4 clusterware.
    thanks,
    Keyur

  • Why root.sh fails in the second node?

    Hi
    After successful install of oracle11 grid in 2 nodes and running root.sh on
    node1, root.sh on node2 fails:
    [root@vmorarac2 dev]#
    /u01/app/product/11.2.0/oracle/root.sh
    Running Oracle 11g root.sh script...
    The following environment variables are set as:
        ORACLE_OWNER= oracle
        ORACLE_HOME=  /u01/app/product/11.2.0/oracle
    Enter the full pathname of the local bin directory:
    [/usr/local/bin]:
    The file "dbhome" already exists in /usr/local/bin. 
    Overwrite it? (y/n)
    [n]: y
       Copying dbhome to /usr/local/bin ...
    The file "oraenv" already exists in /usr/local/bin. 
    Overwrite it? (y/n)
    [n]: y
       Copying oraenv to /usr/local/bin ...
    The file "coraenv" already exists in /usr/local/bin. 
    Overwrite it? (y/n)
    [n]: y
       Copying coraenv to /usr/local/bin ...
    Creating /etc/oratab file...
    Entries will be added to the /etc/oratab file as needed
    by
    Database Configuration Assistant when a database is
    created
    Finished running generic part of root.sh script.
    Now product-specific root actions will be performed.
    2013-07-17 08:37:10: Parsing the host name
    2013-07-17 08:37:10: Checking for super user
    privileges
    2013-07-17 08:37:10: User has super user privileges
    Using configuration parameter file:
    /u01/app/product/11.2.0/oracle/crs/install/crsconfig_params
    Creating trace directory
    LOCAL ADD MODE
    Creating OCR keys for user 'root', privgrp 'root'..
    Operation successful.
    Adding daemon to inittab
    CRS-4123: Oracle High Availability Services has been
    started.
    ohasd is starting
    CRS-4402: The CSS daemon was started in exclusive mode but
    found an active CSS daemon on node vmorarac1, number 1, and is terminating
    CRS-2673: Attempting to stop 'ora.cssdmonitor' on
    'vmorarac2'
    CRS-2677: Stop of 'ora.cssdmonitor' on 'vmorarac2'
    succeeded
    An active cluster was found during exclusive startup,
    restarting to join the cluster
    CRS-2672: Attempting to start 'ora.mdnsd' on
    'vmorarac2'
    CRS-2676: Start of 'ora.mdnsd' on 'vmorarac2'
    succeeded
    CRS-2672: Attempting to start 'ora.gipcd' on
    'vmorarac2'
    CRS-2676: Start of 'ora.gipcd' on 'vmorarac2'
    succeeded
    CRS-2672: Attempting to start 'ora.gpnpd' on
    'vmorarac2'
    CRS-2676: Start of 'ora.gpnpd' on 'vmorarac2'
    succeeded
    CRS-2672: Attempting to start 'ora.cssdmonitor' on
    'vmorarac2'
    CRS-2676: Start of 'ora.cssdmonitor' on 'vmorarac2'
    succeeded
    CRS-2672: Attempting to start 'ora.cssd' on
    'vmorarac2'
    CRS-2672: Attempting to start 'ora.diskmon' on
    'vmorarac2'
    CRS-2676: Start of 'ora.diskmon' on 'vmorarac2'
    succeeded
    CRS-2674: Start of 'ora.cssd' on 'vmorarac2' failed
    CRS-2679: Attempting to clean 'ora.cssd' on
    'vmorarac2'
    CRS-2681: Clean of 'ora.cssd' on 'vmorarac2' succeeded
    CRS-2673: Attempting to stop 'ora.diskmon' on
    'vmorarac2'
    CRS-2677: Stop of 'ora.diskmon' on 'vmorarac2'
    succeeded
    CRS-4000: Command Start failed, or completed with
    errors.
    CRS-2672: Attempting to start 'ora.cssd' on
    'vmorarac2'
    CRS-2672: Attempting to start 'ora.diskmon' on
    'vmorarac2'
    CRS-2674: Start of 'ora.diskmon' on 'vmorarac2' failed
    CRS-2679: Attempting to clean 'ora.diskmon' on
    'vmorarac2'
    CRS-5016: Process
    "/u01/app/product/11.2.0/oracle/bin/diskmon" spawned by agent
    "/u01/app/product/11.2.0/oracle/bin/orarootagent.bin" for action "clean" failed:
    details at "(:CLSN00010:)" in
    "/u01/app/product/11.2.0/oracle/log/vmorarac2/agent/ohasd/orarootagent_root/orarootagent_root.log"
    CRS-2681: Clean of 'ora.diskmon' on 'vmorarac2'
    succeeded
    CRS-2674: Start of 'ora.cssd' on 'vmorarac2' failed
    CRS-2679: Attempting to clean 'ora.cssd' on
    'vmorarac2'
    CRS-2681: Clean of 'ora.cssd' on 'vmorarac2' succeeded
    CRS-4000: Command Start failed, or completed with
    errors.
    Command return code of 1 (256) from command:
    /u01/app/product/11.2.0/oracle/bin/crsctl start resource ora.ctssd -init -env
    USR_ORA_ENV=CTSS_REBOOT=TRUE
    Start of resource "ora.ctssd -init -env
    USR_ORA_ENV=CTSS_REBOOT=TRUE" failed
    Failed to start CTSS
    Failed to start Oracle Clusterware stack
    [root@vmorarac2 dev]#
    [root@vmorarac2 dev]#
    try again:
    [root@vmorarac2 bin]# ./crsctl start resource ora.ctssd
    -init -env USR_ORA_ENV=CTSS_REBOOT=TRUE
    CRS-2672: Attempting to start 'ora.cssd' on
    'vmorarac2'
    CRS-2672: Attempting to start 'ora.diskmon' on
    'vmorarac2'
    CRS-2674: Start of 'ora.diskmon' on 'vmorarac2' failed
    CRS-2679: Attempting to clean 'ora.diskmon' on
    'vmorarac2'
    CRS-5016: Process
    "/u01/app/product/11.2.0/oracle/bin/diskmon" spawned by agent
    "/u01/app/product/11.2.0/oracle/bin/orarootagent.bin" for action "clean" failed:
    details at "(:CLSN00010:)" in
    "/u01/app/product/11.2.0/oracle/log/vmorarac2/agent/ohasd/orarootagent_root/orarootagent_root.log"
    CRS-2681: Clean of 'ora.diskmon' on 'vmorarac2'
    succeeded
    CRS-2674: Start of 'ora.cssd' on 'vmorarac2' failed
    CRS-2679: Attempting to clean 'ora.cssd' on
    'vmorarac2'
    CRS-2681: Clean of 'ora.cssd' on 'vmorarac2' succeeded
    CRS-4000: Command Start failed, or completed with
    errors.
    [root@vmorarac2 bin]# ps -ef | grep u01
    root      8913     1  0 08:37 ?        00:00:06
    /u01/app/product/11.2.0/oracle/bin/ohasd.bin reboot
    oracle   10507     1  0 08:39 ?        00:00:02
    /u01/app/product/11.2.0/oracle/bin/oraagent.bin
    oracle   10522     1  0 08:39 ?        00:00:00
    /u01/app/product/11.2.0/oracle/bin/mdnsd.bin
    oracle   10534     1  0 08:39 ?        00:00:00
    /u01/app/product/11.2.0/oracle/bin/gipcd.bin
    oracle   10548     1  0 08:39 ?        00:00:39
    /u01/app/product/11.2.0/oracle/bin/gpnpd.bin
    root     11723     1  0 11:00 ?        00:00:03
    /u01/app/product/11.2.0/oracle/bin/cssdmonitor
    [oracle@vmorarac2 bin]$ ./crsctl check crs
    CRS-4638: Oracle High Availability Services is online
    CRS-4535: Cannot communicate with Cluster Ready
    Services
    CRS-4530: Communications failure contacting Cluster
    Synchronization Services daemon
    CRS-4534: Cannot communicate with Event Manager

    log:
    Oracle Database 11g Clusterware Release 11.2.0.1.0 - Production Copyright 1996, 2009 Oracle. All rights reserved.
    2013-07-17 08:38:04.485: [    AGFW][3481860944] Starting the agent: /u01/app/product/11.2.0/oracle/log/vmorarac2/agent/ohasd/orarootagent_root/
    2013-07-17 08:38:04.485: [   AGENT][3481860944] Agent framework initialized, Process Id = 10319
    2013-07-17 08:38:04.487: [ USRTHRD][3481860944] Utils::getCrsHome crsHome /u01/app/product/11.2.0/oracle
    2013-07-17 08:38:04.487: [ USRTHRD][3481860944] Process::convertPidToString pid = 10319
    2013-07-17 08:38:04.488: [    AGFW][3481860944] SERVER IPC CONNECT STR: (ADDRESS=(PROTOCOL=IPC)(KEY=OHASD_IPC_SOCKET_11))
    2013-07-17 08:38:04.488: [CLSFRAME][3481860944] Inited lsf context 0x317e9e0
    2013-07-17 08:38:04.488: [CLSFRAME][3481860944] Initing CLS Framework messaging
    2013-07-17 08:38:04.488: [CLSFRAME][3481860944] New Framework state: 2
    2013-07-17 08:38:04.488: [CLSFRAME][3481860944] M2M is starting...
    2013-07-17 08:38:04.490: [ CRSCOMM][3481860944] m_pClscCtx=0x31d1bd0m_pUgblm=0x31d5720
    2013-07-17 08:38:04.490: [ CRSCOMM][3481860944] Starting send thread
    2013-07-17 08:38:04.490: [ CRSCOMM][1119435072] clsIpc: sendWork thread started.
    2013-07-17 08:38:04.491: [ CRSCOMM][1129924928] IPC Client thread started listening
    2013-07-17 08:38:04.491: [ CRSCOMM][1129924928] init data sent from server
    2013-07-17 08:38:04.491: [CLSFRAME][3481860944] New IPC Member:{Relative|Node:0|Process:0|Type:2}:OHASD:vmorarac2
    2013-07-17 08:38:04.491: [CLSFRAME][3481860944] New process connected to us ID:{Relative|Node:0|Process:0|Type:2} Info:OHASD:vmorarac2
    2013-07-17 08:38:04.492: [CLSFRAME][3481860944] Starting thread model named: MultiThread
    2013-07-17 08:38:04.492: [CLSFRAME][3481860944] Starting thread model named: SingleThread
    2013-07-17 08:38:04.492: [CLSFRAME][3481860944] Starting thread model named: SingleThreadT
    2013-07-17 08:38:04.492: [CLSFRAME][3481860944] New Framework state: 3
    2013-07-17 08:38:04.493: [    AGFW][3481860944] Agent Framework started successfully
    2013-07-17 08:38:04.493: [    AGFW][1182374208] Agfw engine module has enabled...
    2013-07-17 08:38:04.493: [CLSFRAME][1182374208] Module Enabling is complete
    2013-07-17 08:38:04.493: [CLSFRAME][1182374208] New Framework state: 6
    2013-07-17 08:38:04.493: [    AGFW][1182374208] Agent is started with userid: root , expected user: root
    2013-07-17 08:38:04.493: [    AGFW][1182374208] Agent sending message to PE: AGENT_HANDSHAKE[Proxy] ID 20484:14
    2013-07-17 08:38:04.505: [    AGFW][1182374208] Agent received the message: RESTYPE_ADD[ora.crs.type] ID 8196:358
    2013-07-17 08:38:04.506: [    AGFW][1182374208] Added new restype: ora.crs.type
    2013-07-17 08:38:04.506: [    AGFW][1182374208] Agent sending last reply for: RESTYPE_ADD[ora.crs.type] ID 8196:358
    2013-07-17 08:38:04.506: [    AGFW][1182374208] Agent received the message: RESTYPE_ADD[ora.ctss.type] ID 8196:360
    2013-07-17 08:38:04.506: [    AGFW][1182374208] Added new restype: ora.ctss.type
    2013-07-17 08:38:04.507: [    AGFW][1182374208] Agent sending last reply for: RESTYPE_ADD[ora.ctss.type] ID 8196:360
    2013-07-17 08:38:04.516: [    AGFW][1182374208] Agent received the message: RESTYPE_ADD[ora.diskmon.type] ID 8196:362
    2013-07-17 08:38:04.516: [    AGFW][1182374208] Added new restype: ora.diskmon.type
    2013-07-17 08:38:04.517: [    AGFW][1182374208] Agent sending last reply for: RESTYPE_ADD[ora.diskmon.type] ID 8196:362
    2013-07-17 08:38:04.519: [    AGFW][1182374208] Agent received the message: RESTYPE_ADD[ora.drivers.acfs.type] ID 8196:364
    2013-07-17 08:38:04.520: [    AGFW][1182374208] Added new restype: ora.drivers.acfs.type
    2013-07-17 08:38:04.520: [    AGFW][1182374208] Agent sending last reply for: RESTYPE_ADD[ora.drivers.acfs.type] ID 8196:364
    2013-07-17 08:38:04.521: [    AGFW][1182374208] Agent received the message: RESOURCE_ADD[ora.diskmon 1 1] ID 4356:366
    2013-07-17 08:38:04.521: [    AGFW][1182374208] Added new resource: ora.diskmon 1 1 to the agfw
    2013-07-17 08:38:04.522: [    AGFW][1182374208] Agent sending last reply for: RESOURCE_ADD[ora.diskmon 1 1] ID 4356:366
    2013-07-17 08:38:04.522: [    AGFW][1182374208] Agent received the message: RESOURCE_START[ora.diskmon 1 1] ID 4098:367
    2013-07-17 08:38:04.522: [    AGFW][1182374208] Preparing START command for: ora.diskmon 1 1
    2013-07-17 08:38:04.522: [    AGFW][1182374208] ora.diskmon 1 1 state changed from: UNKNOWN to: STARTING
    2013-07-17 08:38:04.526: [    AGFW][1161394496] Executing command: start for resource: ora.diskmon 1 1
    2013-07-17 08:38:04.527: [ora.diskmon][1161394496] [start] clsn_agent::start {
    2013-07-17 08:38:04.527: [ora.diskmon][1161394496] [start] DaemonAgent{
    2013-07-17 08:38:04.527: [ora.diskmon][1161394496] [start] }DaemonAgent
    2013-07-17 08:38:04.527: [ora.diskmon][1161394496] [start] DiskmonAgent::DiskmonAgent {
    2013-07-17 08:38:04.527: [ora.diskmon][1161394496] [start] InitAttrs {
    2013-07-17 08:38:04.527: [ora.diskmon][1161394496] [start] __IS_HASD_AGENT=TRUE
    2013-07-17 08:38:04.527: [ora.diskmon][1161394496] [start] }InitAttrs
    2013-07-17 08:38:04.527: [ora.diskmon][1161394496] [start] DiskmonAgent::DiskmonAgent }
    2013-07-17 08:38:04.527: [ora.diskmon][1161394496] [start] DiskmonAgent::start {
    2013-07-17 08:38:04.527: [ora.diskmon][1161394496] [start] Arg Value = -d
    2013-07-17 08:38:04.527: [ora.diskmon][1161394496] [start] Arg Value = -f
    2013-07-17 08:38:04.527: [ora.diskmon][1161394496] [start] Total Count of Environment Variables = 3
    2013-07-17 08:38:04.527: [ora.diskmon][1161394496] [start] Adding Environment Variables _ORA_AGENT_ACTION=TRUE
    2013-07-17 08:38:04.527: [ora.diskmon][1161394496] [start] Adding Environment Variables __IS_HASD_AGENT=
    2013-07-17 08:38:04.527: [ora.diskmon][1161394496] [start] Adding Environment variable from USR_ORA_ENV ORACLE_USER=oracle
    2013-07-17 08:38:04.527: [ora.diskmon][1161394496] [start] Utils:execCmd action = 1 flags = 5 ohome = (null) cmdname = diskmon.
    2013-07-17 08:38:04.528: [ora.diskmon][1161394496] [start] getOracleHomeAttrib: oracle_home = /u01/app/product/11.2.0/oracle
    2013-07-17 08:38:04.528: [ora.diskmon][1161394496] [start] Utils:execCmd Running the binary from /u01/app/product/11.2.0/oracle/bin/diskmon
    2013-07-17 08:38:04.531: [CRSTIMER][1091324224] Timer Thread Starting.
    2013-07-17 08:38:04.533: [ora.diskmon][1161394496] [start] execCmd ret = 0
    2013-07-17 08:38:04.533: [ora.diskmon][1161394496] [start] }DaemonAgent::start
    2013-07-17 08:38:10.534: [ora.diskmon][1161394496] [start] DiskmonAgent::connect {
    2013-07-17 08:38:10.534: [ora.diskmon][1161394496] [start] Process::convertPidToString pid = 10319
    2013-07-17 08:38:10.535: [ora.diskmon][1161394496] [start] DiskmonAgent::connect }
    2013-07-17 08:38:10.535: [ora.diskmon][1161394496] [start] DiskmonAgent::start }
    2013-07-17 08:38:10.535: [ora.diskmon][1161394496] [start] clsn_agent::start }
    2013-07-17 08:38:10.535: [    AGFW][1161394496] Command: start for resource: ora.diskmon 1 1 completed with status: SUCCESS
    2013-07-17 08:38:10.535: [    AGFW][1182374208] Agent sending reply for: RESOURCE_START[ora.diskmon 1 1] ID 4098:367
    2013-07-17 08:38:10.537: [    AGFW][1161394496] Executing command: check for resource: ora.diskmon 1 1
    2013-07-17 08:38:10.538: [ora.diskmon][1161394496] [check] DiskmonAgent::check {
    2013-07-17 08:38:10.538: [ora.diskmon][1161394496] [check] DiskmonAgent::check } 0
    2013-07-17 08:38:10.538: [    AGFW][1161394496] check for resource: ora.diskmon 1 1 completed with status: ONLINE
    2013-07-17 08:38:10.538: [    AGFW][1182374208] ora.diskmon 1 1 state changed from: STARTING to: ONLINE
    2013-07-17 08:38:10.538: [    AGFW][1182374208] Started implicit monitor for:ora.diskmon 1 1
    2013-07-17 08:38:10.538: [    AGFW][1182374208] Agent sending last reply for: RESOURCE_START[ora.diskmon 1 1] ID 4098:367
    2013-07-17 08:38:30.543: [    AGFW][1182374208] CHECK initiated by timer for: ora.diskmon 1 1
    2013-07-17 08:38:30.544: [    AGFW][1161394496] Executing command: check for resource: ora.diskmon 1 1
    2013-07-17 08:38:30.544: [ora.diskmon][1161394496] [check] DiskmonAgent::check {
    2013-07-17 08:38:30.544: [ora.diskmon][1161394496] [check] DiskmonAgent::check } 0
    2013-07-17 08:38:30.545: [    AGFW][1161394496] check for resource: ora.diskmon 1 1 completed with status: ONLINE
    2013-07-17 08:38:50.550: [    AGFW][1182374208] CHECK initiated by timer for: ora.diskmon 1 1
    2013-07-17 08:38:50.551: [    AGFW][1161394496] Executing command: check for resource: ora.diskmon 1 1
    2013-07-17 08:38:50.551: [ora.diskmon][1161394496] [check] DiskmonAgent::check {
    2013-07-17 08:38:50.551: [ora.diskmon][1161394496] [check] DiskmonAgent::check } 0
    2013-07-17 08:38:50.551: [    AGFW][1161394496] check for resource: ora.diskmon 1 1 completed with status: ONLINE
    2013-07-17 08:39:02.782: [    AGFW][1182374208] Agent received the message: RESOURCE_STOP[ora.diskmon 1 1] ID 4099:547
    2013-07-17 08:39:02.782: [    AGFW][1182374208] Preparing STOP command for: ora.diskmon 1 1
    2013-07-17 08:39:02.782: [    AGFW][1182374208] ora.diskmon 1 1 state changed from: ONLINE to: STOPPING
    2013-07-17 08:39:02.783: [    AGFW][1161394496] Executing command: stop for resource: ora.diskmon 1 1
    2013-07-17 08:39:02.783: [ora.diskmon][1161394496] [stop] clsn_agent::stop {
    2013-07-17 08:39:02.783: [ora.diskmon][1161394496] [stop] DiskmonAgent::stop {
    2013-07-17 08:39:02.783: [ora.diskmon][1161394496] [stop] DiskmonAgent::stop }
    2013-07-17 08:39:02.783: [ora.diskmon][1161394496] [stop] clsn_agent::stop }
    2013-07-17 08:39:02.783: [    AGFW][1161394496] Command: stop for resource: ora.diskmon 1 1 completed with status: SUCCESS
    2013-07-17 08:39:02.784: [    AGFW][1161394496] Executing command: check for resource: ora.diskmon 1 1
    2013-07-17 08:39:02.784: [ora.diskmon][1161394496] [check] DiskmonAgent::check {
    2013-07-17 08:39:02.784: [ora.diskmon][1161394496] [check] DiskmonAgent::check } 2
    2013-07-17 08:39:02.784: [    AGFW][1161394496] check for resource: ora.diskmon 1 1 completed with status: PLANNED_OFFLINE
    2013-07-17 08:39:02.784: [    AGFW][1182374208] Agent sending reply for: RESOURCE_STOP[ora.diskmon 1 1] ID 4099:547
    2013-07-17 08:39:02.785: [    AGFW][1182374208] ora.diskmon 1 1 state changed from: STOPPING to: PLANNED_OFFLINE
    2013-07-17 08:39:02.785: [    AGFW][1182374208] Agent sending last reply for: RESOURCE_STOP[ora.diskmon 1 1] ID 4099:547
    2013-07-17 08:39:02.785: [    AGFW][1182374208] Agent has no resources to be monitored.Sending suicide request.
    2013-07-17 08:39:02.786: [    AGFW][1182374208] Agent sending message to PE: AGENT_SUICIDE[Proxy] ID 20486:72
    2013-07-17 08:39:02.789: [    AGFW][1182374208] Agent is commiting suicide.
    2013-07-17 08:39:02.790: [    AGFW][1182374208] Agent is exiting with exit code: 1
    2013-07-17 08:39:12.176: [    AGFW][1664629584] Starting the agent: /u01/app/product/11.2.0/oracle/log/vmorarac2/agent/ohasd/orarootagent_root/
    2013-07-17 08:39:12.176: [   AGENT][1664629584] Agent framework initialized, Process Id = 10581
    2013-07-17 08:39:12.178: [ USRTHRD][1664629584] Utils::getCrsHome crsHome /u01/app/product/11.2.0/oracle
    2013-07-17 08:39:12.178: [ USRTHRD][1664629584] Process::convertPidToString pid = 10581
    2013-07-17 08:39:12.178: [    AGFW][1664629584] SERVER IPC CONNECT STR: (ADDRESS=(PROTOCOL=IPC)(KEY=OHASD_IPC_SOCKET_11))
    2013-07-17 08:39:12.178: [CLSFRAME][1664629584] Inited lsf context 0xd9309e0
    2013-07-17 08:39:12.179: [CLSFRAME][1664629584] Initing CLS Framework messaging
    2013-07-17 08:39:12.179: [CLSFRAME][1664629584] New Framework state: 2
    2013-07-17 08:39:12.179: [CLSFRAME][1664629584] M2M is starting...
    2013-07-17 08:39:12.180: [ CRSCOMM][1664629584] m_pClscCtx=0xd983bd0m_pUgblm=0xd987720
    2013-07-17 08:39:12.180: [ CRSCOMM][1664629584] Starting send thread
    2013-07-17 08:39:12.181: [ CRSCOMM][1115052352] clsIpc: sendWork thread started.
    2013-07-17 08:39:12.181: [ CRSCOMM][1125542208] IPC Client thread started listening
    2013-07-17 08:39:12.181: [ CRSCOMM][1125542208] init data sent from server
    2013-07-17 08:39:12.181: [CLSFRAME][1664629584] New IPC Member:{Relative|Node:0|Process:0|Type:2}:OHASD:vmorarac2
    2013-07-17 08:39:12.181: [CLSFRAME][1664629584] New process connected to us ID:{Relative|Node:0|Process:0|Type:2} Info:OHASD:vmorarac2
    2013-07-17 08:39:12.182: [CLSFRAME][1664629584] Starting thread model named: MultiThread
    2013-07-17 08:39:12.182: [CLSFRAME][1664629584] Starting thread model named: SingleThread
    2013-07-17 08:39:12.182: [CLSFRAME][1664629584] Starting thread model named: SingleThreadT
    2013-07-17 08:39:12.182: [CLSFRAME][1664629584] New Framework state: 3
    2013-07-17 08:39:12.182: [    AGFW][1664629584] Agent Framework started successfully
    2013-07-17 08:39:12.182: [    AGFW][1177991488] Agfw engine module has enabled...
    2013-07-17 08:39:12.183: [CLSFRAME][1177991488] Module Enabling is complete
    2013-07-17 08:39:12.183: [CLSFRAME][1177991488] New Framework state: 6
    2013-07-17 08:39:12.183: [    AGFW][1177991488] Agent is started with userid: root , expected user: root
    2013-07-17 08:39:12.183: [    AGFW][1177991488] Agent sending message to PE: AGENT_HANDSHAKE[Proxy] ID 20484:14
    2013-07-17 08:39:12.192: [    AGFW][1177991488] Agent received the message: RESTYPE_ADD[ora.crs.type] ID 8196:886
    2013-07-17 08:39:12.192: [    AGFW][1177991488] Added new restype: ora.crs.type
    2013-07-17 08:39:12.192: [    AGFW][1177991488] Agent sending last reply for: RESTYPE_ADD[ora.crs.type] ID 8196:886
    2013-07-17 08:39:12.198: [    AGFW][1177991488] Agent received the message: RESTYPE_ADD[ora.ctss.type] ID 8196:888
    2013-07-17 08:39:12.198: [    AGFW][1177991488] Added new restype: ora.ctss.type
    2013-07-17 08:39:12.199: [    AGFW][1177991488] Agent sending last reply for: RESTYPE_ADD[ora.ctss.type] ID 8196:888
    2013-07-17 08:39:12.204: [    AGFW][1177991488] Agent received the message: RESTYPE_ADD[ora.diskmon.type] ID 8196:890
    2013-07-17 08:39:12.204: [    AGFW][1177991488] Added new restype: ora.diskmon.type
    2013-07-17 08:39:12.204: [    AGFW][1177991488] Agent sending last reply for: RESTYPE_ADD[ora.diskmon.type] ID 8196:890
    2013-07-17 08:39:12.209: [    AGFW][1177991488] Agent received the message: RESTYPE_ADD[ora.drivers.acfs.type] ID 8196:892
    2013-07-17 08:39:12.209: [    AGFW][1177991488] Added new restype: ora.drivers.acfs.type
    2013-07-17 08:39:12.210: [    AGFW][1177991488] Agent sending last reply for: RESTYPE_ADD[ora.drivers.acfs.type] ID 8196:892
    2013-07-17 08:39:12.210: [    AGFW][1177991488] Agent received the message: RESOURCE_ADD[ora.diskmon 1 1] ID 4356:894
    2013-07-17 08:39:12.210: [    AGFW][1177991488] Added new resource: ora.diskmon 1 1 to the agfw
    2013-07-17 08:39:12.210: [    AGFW][1177991488] Agent sending last reply for: RESOURCE_ADD[ora.diskmon 1 1] ID 4356:894
    2013-07-17 08:39:12.210: [    AGFW][1177991488] Agent received the message: RESOURCE_START[ora.diskmon 1 1] ID 4098:895
    2013-07-17 08:39:12.211: [    AGFW][1177991488] Preparing START command for: ora.diskmon 1 1
    2013-07-17 08:39:12.211: [    AGFW][1177991488] ora.diskmon 1 1 state changed from: UNKNOWN to: STARTING
    2013-07-17 08:39:12.216: [    AGFW][1167501632] Executing command: start for resource: ora.diskmon 1 1
    2013-07-17 08:39:12.216: [ora.diskmon][1167501632] [start] clsn_agent::start {
    2013-07-17 08:39:12.216: [ora.diskmon][1167501632] [start] DaemonAgent{
    2013-07-17 08:39:12.216: [ora.diskmon][1167501632] [start] }DaemonAgent
    2013-07-17 08:39:12.216: [ora.diskmon][1167501632] [start] DiskmonAgent::DiskmonAgent {
    2013-07-17 08:39:12.216: [ora.diskmon][1167501632] [start] InitAttrs {
    2013-07-17 08:39:12.216: [ora.diskmon][1167501632] [start] __IS_HASD_AGENT=TRUE
    2013-07-17 08:39:12.216: [ora.diskmon][1167501632] [start] }InitAttrs
    2013-07-17 08:39:12.217: [ora.diskmon][1167501632] [start] DiskmonAgent::DiskmonAgent }
    2013-07-17 08:39:12.217: [ora.diskmon][1167501632] [start] DiskmonAgent::start {
    2013-07-17 08:39:12.217: [ora.diskmon][1167501632] [start] Arg Value = -d
    2013-07-17 08:39:12.217: [ora.diskmon][1167501632] [start] Arg Value = -f
    2013-07-17 08:39:12.217: [ora.diskmon][1167501632] [start] Total Count of Environment Variables = 3
    2013-07-17 08:39:12.217: [ora.diskmon][1167501632] [start] Adding Environment Variables _ORA_AGENT_ACTION=TRUE
    2013-07-17 08:39:12.217: [ora.diskmon][1167501632] [start] Adding Environment Variables __IS_HASD_AGENT=
    2013-07-17 08:39:12.217: [ora.diskmon][1167501632] [start] Adding Environment variable from USR_ORA_ENV ORACLE_USER=oracle
    2013-07-17 08:39:12.217: [ora.diskmon][1167501632] [start] Utils:execCmd action = 1 flags = 5 ohome = (null) cmdname = diskmon.
    2013-07-17 08:39:12.217: [ora.diskmon][1167501632] [start] getOracleHomeAttrib: oracle_home = /u01/app/product/11.2.0/oracle
    2013-07-17 08:39:12.217: [ora.diskmon][1167501632] [start] Utils:execCmd Running the binary from /u01/app/product/11.2.0/oracle/bin/diskmon
    2013-07-17 08:39:12.220: [CRSTIMER][1198971200] Timer Thread Starting.
    2013-07-17 08:39:12.220: [ora.diskmon][1167501632] [start] execCmd ret = 0
    2013-07-17 08:39:12.220: [ora.diskmon][1167501632] [start] }DaemonAgent::start
    2013-07-17 08:39:18.222: [ora.diskmon][1167501632] [start] DiskmonAgent::connect {
    2013-07-17 08:39:18.222: [ora.diskmon][1167501632] [start] Process::convertPidToString pid = 10581
    2013-07-17 08:39:18.222: [ora.diskmon][1167501632] [start] DiskmonAgent::connect }
    2013-07-17 08:39:18.222: [ora.diskmon][1167501632] [start] DiskmonAgent::start }
    2013-07-17 08:39:18.222: [ora.diskmon][1167501632] [start] clsn_agent::start }
    2013-07-17 08:39:18.222: [    AGFW][1167501632] Command: start for resource: ora.diskmon 1 1 completed with status: SUCCESS
    2013-07-17 08:39:18.223: [    AGFW][1177991488] Agent sending reply for: RESOURCE_START[ora.diskmon 1 1] ID 4098:895
    2013-07-17 08:39:18.223: [    AGFW][1167501632] Executing command: check for resource: ora.diskmon 1 1
    2013-07-17 08:39:18.223: [ora.diskmon][1167501632] [check] DiskmonAgent::check {
    2013-07-17 08:39:18.224: [ora.diskmon][1167501632] [check] DiskmonAgent::check } 0
    2013-07-17 08:39:18.224: [    AGFW][1167501632] check for resource: ora.diskmon 1 1 completed with status: ONLINE
    2013-07-17 08:39:18.225: [    AGFW][1177991488] ora.diskmon 1 1 state changed from: STARTING to: ONLINE
    2013-07-17 08:39:18.225: [    AGFW][1177991488] Started implicit monitor for:ora.diskmon 1 1
    2013-07-17 08:39:18.225: [    AGFW][1177991488] Agent sending last reply for: RESOURCE_START[ora.diskmon 1 1] ID 4098:895
    2013-07-17 08:39:38.231: [    AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
    2013-07-17 08:39:38.232: [    AGFW][1167501632] Executing command: check for resource: ora.diskmon 1 1
    2013-07-17 08:39:38.232: [ora.diskmon][1167501632] [check] DiskmonAgent::check {
    2013-07-17 08:39:38.232: [ora.diskmon][1167501632] [check] DiskmonAgent::check } 0
    2013-07-17 08:39:38.232: [    AGFW][1167501632] check for resource: ora.diskmon 1 1 completed with status: ONLINE
    2013-07-17 08:39:58.237: [    AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
    2013-07-17 08:39:58.238: [    AGFW][1167501632] Executing command: check for resource: ora.diskmon 1 1
    2013-07-17 08:39:58.238: [ora.diskmon][1167501632] [check] DiskmonAgent::check {
    2013-07-17 08:39:58.238: [ora.diskmon][1167501632] [check] DiskmonAgent::check } 0
    2013-07-17 08:39:58.238: [    AGFW][1167501632] check for resource: ora.diskmon 1 1 completed with status: ONLINE
    2013-07-17 08:40:12.107: [    AGFW][1177991488] Agent received the message: AGENT_HB[Engine] ID 12293:966
    2013-07-17 08:40:18.243: [    AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
    2013-07-17 08:40:18.244: [    AGFW][1167501632] Executing command: check for resource: ora.diskmon 1 1
    2013-07-17 08:40:18.244: [ora.diskmon][1167501632] [check] DiskmonAgent::check {
    2013-07-17 08:40:18.244: [ora.diskmon][1167501632] [check] DiskmonAgent::check } 0
    2013-07-17 08:40:18.244: [    AGFW][1167501632] check for resource: ora.diskmon 1 1 completed with status: ONLINE
    2013-07-17 08:40:38.250: [    AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
    2013-07-17 08:40:38.251: [    AGFW][1167501632] Executing command: check for resource: ora.diskmon 1 1
    2013-07-17 08:40:38.251: [ora.diskmon][1167501632] [check] DiskmonAgent::check {
    2013-07-17 08:40:38.251: [ora.diskmon][1167501632] [check] DiskmonAgent::check } 0
    2013-07-17 08:40:38.251: [    AGFW][1167501632] check for resource: ora.diskmon 1 1 completed with status: ONLINE
    2013-07-17 08:40:42.116: [    AGFW][1177991488] Agent received the message: AGENT_HB[Engine] ID 12293:982
    2013-07-17 08:40:58.245: [    AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
    2013-07-17 08:40:58.246: [    AGFW][1167501632] Executing command: check for resource: ora.diskmon 1 1
    2013-07-17 08:40:58.247: [ora.diskmon][1167501632] [check] DiskmonAgent::check {
    2013-07-17 08:40:58.247: [ora.diskmon][1167501632] [check] DiskmonAgent::check } 0
    2013-07-17 08:40:58.247: [    AGFW][1167501632] check for resource: ora.diskmon 1 1 completed with status: ONLINE
    2013-07-17 08:41:12.125: [    AGFW][1177991488] Agent received the message: AGENT_HB[Engine] ID 12293:998
    2013-07-17 08:41:18.252: [    AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
    2013-07-17 08:41:18.252: [    AGFW][1167501632] Executing command: check for resource: ora.diskmon 1 1
    2013-07-17 08:41:18.253: [ora.diskmon][1167501632] [check] DiskmonAgent::check {
    2013-07-17 08:41:18.253: [ora.diskmon][1167501632] [check] DiskmonAgent::check } 0
    2013-07-17 08:41:18.253: [    AGFW][1167501632] check for resource: ora.diskmon 1 1 completed with status: ONLINE
    2013-07-17 08:41:38.259: [    AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
    2013-07-17 08:41:38.260: [    AGFW][1167501632] Executing command: check for resource: ora.diskmon 1 1
    2013-07-17 08:41:38.260: [ora.diskmon][1167501632] [check] DiskmonAgent::check {
    2013-07-17 08:41:38.260: [ora.diskmon][1167501632] [check] DiskmonAgent::check } 0
    2013-07-17 08:41:38.260: [    AGFW][1167501632] check for resource: ora.diskmon 1 1 completed with status: ONLINE
    2013-07-17 08:41:58.255: [    AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
    2013-07-17 08:41:58.256: [    AGFW][1167501632] Executing command: check for resource: ora.diskmon 1 1
    2013-07-17 08:41:58.256: [ora.diskmon][1167501632] [check] DiskmonAgent::check {
    2013-07-17 08:41:58.256: [ora.diskmon][1167501632] [check] DiskmonAgent::check } 0
    2013-07-17 08:41:58.256: [    AGFW][1167501632] check for resource: ora.diskmon 1 1 completed with status: ONLINE
    2013-07-17 08:42:12.134: [    AGFW][1177991488] Agent received the message: AGENT_HB[Engine] ID 12293:1026
    2013-07-17 08:42:18.261: [    AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
    2013-07-17 08:42:18.262: [    AGFW][1167501632] Executing command: check for resource: ora.diskmon 1 1
    2013-07-17 08:42:18.262: [ora.diskmon][1167501632] [check] DiskmonAgent::check {
    2013-07-17 08:42:18.262: [ora.diskmon][1167501632] [check] DiskmonAgent::check } 0
    2013-07-17 08:42:18.262: [    AGFW][1167501632] check for resource: ora.diskmon 1 1 completed with status: ONLINE
    2013-07-17 08:42:38.268: [    AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
    2013-07-17 08:42:38.269: [    AGFW][1167501632] Executing command: check for resource: ora.diskmon 1 1
    2013-07-17 08:42:38.269: [ora.diskmon][1167501632] [check] DiskmonAgent::check {
    2013-07-17 08:42:38.269: [ora.diskmon][1167501632] [check] DiskmonAgent::check } 0
    2013-07-17 08:42:38.269: [    AGFW][1167501632] check for resource: ora.diskmon 1 1 completed with status: ONLINE
    2013-07-17 08:42:58.265: [    AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
    2013-07-17 08:42:58.266: [    AGFW][1167501632] Executing command: check for resource: ora.diskmon 1 1
    2013-07-17 08:42:58.266: [ora.diskmon][1167501632] [check] DiskmonAgent::check {
    2013-07-17 08:42:58.267: [ora.diskmon][1167501632] [check] DiskmonAgent::check } 0
    2013-07-17 08:42:58.267: [    AGFW][1167501632] check for resource: ora.diskmon 1 1 completed with status: ONLINE
    2013-07-17 08:43:12.144: [    AGFW][1177991488] Agent received the message: AGENT_HB[Engine] ID 12293:1054
    2013-07-17 08:43:18.272: [    AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
    2013-07-17 08:43:18.272: [    AGFW][1167501632] Executing command: check for resource: ora.diskmon 1 1
    2013-07-17 08:43:18.273: [ora.diskmon][1167501632] [check] DiskmonAgent::check {
    2013-07-17 08:43:18.273: [ora.diskmon][1167501632] [check] DiskmonAgent::check } 0
    2013-07-17 08:43:18.273: [    AGFW][1167501632] check for resource: ora.diskmon 1 1 completed with status: ONLINE
    2013-07-17 08:43:38.278: [    AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
    2013-07-17 08:43:38.278: [    AGFW][1167501632] Executing command: check for resource: ora.diskmon 1 1
    2013-07-17 08:43:38.279: [ora.diskmon][1167501632] [check] DiskmonAgent::check {
    2013-07-17 08:43:38.279: [ora.diskmon][1167501632] [check] DiskmonAgent::check } 0
    2013-07-17 08:43:38.279: [    AGFW][1167501632] check for resource: ora.diskmon 1 1 completed with status: ONLINE
    2013-07-17 08:43:42.154: [    AGFW][1177991488] Agent received the message: AGENT_HB[Engine] ID 12293:1070
    2013-07-17 08:43:58.284: [    AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
    2013-07-17 08:43:58.285: [    AGFW][1167501632] Executing command: check for resource: ora.diskmon 1 1
    2013-07-17 08:43:58.285: [ora.diskmon][1167501632] [check] DiskmonAgent::check {
    2013-07-17 08:43:58.285: [ora.diskmon][1167501632] [check] DiskmonAgent::check } 0
    2013-07-17 08:43:58.285: [    AGFW][1167501632] check for resource: ora.diskmon 1 1 completed with status: ONLINE
    2013-07-17 08:44:12.154: [    AGFW][1177991488] Agent received the message: AGENT_HB[Engine] ID 12293:1086
    2013-07-17 08:44:18.291: [    AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
    2013-07-17 08:44:18.292: [    AGFW][1167501632] Executing command: check for resource: ora.diskmon 1 1
    2013-07-17 08:44:18.292: [ora.diskmon][1167501632] [check] DiskmonAgent::check {
    2013-07-17 08:44:18.292: [ora.diskmon][1167501632] [check] DiskmonAgent::check } 0
    2013-07-17 08:44:18.292: [    AGFW][1167501632] check for resource: ora.diskmon 1 1 completed with status: ONLINE
    2013-07-17 08:44:38.296: [    AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
    2013-07-17 08:44:38.297: [    AGFW][1157011776] Executing command: check for resource: ora.diskmon 1 1
    2013-07-17 08:44:38.297: [ora.diskmon][1157011776] [check] DiskmonAgent::check {
    2013-07-17 08:44:38.297: [ora.diskmon][1157011776] [check] DiskmonAgent::check } 0
    2013-07-17 08:44:38.297: [    AGFW][1157011776] check for resource: ora.diskmon 1 1 completed with status: ONLINE
    2013-07-17 08:44:42.163: [    AGFW][1177991488] Agent received the message: AGENT_HB[Engine] ID 12293:1102
    2013-07-17 08:44:58.302: [    AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
    2013-07-17 08:44:58.303: [    AGFW][1157011776] Executing command: check for resource: ora.diskmon 1 1
    2013-07-17 08:44:58.303: [ora.diskmon][1157011776] [check] DiskmonAgent::check {
    2013-07-17 08:44:58.304: [ora.diskmon][1157011776] [check] DiskmonAgent::check } 0
    2013-07-17 08:44:58.304: [    AGFW][1157011776] check for resource: ora.diskmon 1 1 completed with status: ONLINE
    2013-07-17 08:45:12.174: [    AGFW][1177991488] Agent received the message: AGENT_HB[Engine] ID 12293:1118
    2013-07-17 08:45:18.309: [    AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
    2013-07-17 08:45:18.309: [    AGFW][1157011776] Executing command: check for resource: ora.diskmon 1 1
    2013-07-17 08:45:18.310: [ora.diskmon][1157011776] [check] DiskmonAgent::check {
    2013-07-17 08:45:18.310: [ora.diskmon][1157011776] [check] DiskmonAgent::check } 0
    2013-07-17 08:45:18.310: [    AGFW][1157011776] check for resource: ora.diskmon 1 1 completed with status: ONLINE
    2013-07-17 08:45:38.315: [    AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
    2013-07-17 08:45:38.316: [    AGFW][1157011776] Executing command: check for resource: ora.diskmon 1 1
    2013-07-17 08:45:38.316: [ora.diskmon][1157011776] [check] DiskmonAgent::check {
    2013-07-17 08:45:38.316: [ora.diskmon][1157011776] [check] DiskmonAgent::check } 0
    2013-07-17 08:45:38.316: [    AGFW][1157011776] check for resource: ora.diskmon 1 1 completed with status: ONLINE
    2013-07-17 08:45:42.183: [    AGFW][1177991488] Agent received the message: AGENT_HB[Engine] ID 12293:1134
    2013-07-17 08:45:58.312: [    AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
    2013-07-17 08:45:58.313: [    AGFW][1157011776] Executing command: check for resource: ora.diskmon 1 1
    2013-07-17 08:45:58.313: [ora.diskmon][1157011776] [check] DiskmonAgent::check {
    2013-07-17 08:45:58.313: [ora.diskmon][1157011776] [check] DiskmonAgent::check } 0
    2013-07-17 08:45:58.313: [    AGFW][1157011776] check for resource: ora.diskmon 1 1 completed with status: ONLINE
    2013-07-17 08:46:12.192: [    AGFW][1177991488] Agent received the message: AGENT_HB[Engine] ID 12293:1150
    2013-07-17 08:46:18.318: [    AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
    2013-07-17 08:46:18.319: [    AGFW][1157011776] Executing command: check for resource: ora.diskmon 1 1
    2013-07-17 08:46:18.319: [ora.diskmon][1157011776] [check] DiskmonAgent::check {
    2013-07-17 08:46:18.319: [ora.diskmon][1157011776] [check] DiskmonAgent::check } 0
    2013-07-17 08:46:18.319: [    AGFW][1157011776] check for resource: ora.diskmon 1 1 completed with status: ONLINE
    2013-07-17 08:46:38.325: [    AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
    2013-07-17 08:46:38.326: [    AGFW][1157011776] Executing command: check for resource: ora.diskmon 1 1
    2013-07-17 08:46:38.326: [ora.diskmon][1157011776] [check] DiskmonAgent::check {
    2013-07-17 08:46:38.326: [ora.diskmon][1157011776] [check] DiskmonAgent::check } 0
    2013-07-17 08:46:38.326: [    AGFW][1157011776] check for resource: ora.diskmon 1 1 completed with status: ONLINE
    2013-07-17 08:46:42.203: [    AGFW][1177991488] Agent received the message: AGENT_HB[Engine] ID 12293:1166
    2013-07-17 08:46:58.332: [    AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
    2013-07-17 08:46:58.333: [    AGFW][1157011776] Executing command: check for resource: ora.diskmon 1 1
    2013-07-17 08:46:58.333: [ora.diskmon][1157011776] [check] DiskmonAgent::check {
    2013-07-17 08:46:58.333: [ora.diskmon][1157011776] [check] DiskmonAgent::check } 0
    2013-07-17 08:46:58.333: [    AGFW][1157011776] check for resource: ora.diskmon 1 1 completed with status: ONLINE
    2013-07-17 08:47:12.203: [    AGFW][1177991488] Agent received the message: AGENT_HB[Engine] ID 12293:1182
    2013-07-17 08:47:18.338: [    AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
    2013-07-17 08:47:18.339: [    AGFW][1157011776] Executing command: check for resource: ora.diskmon 1 1
    2013-07-17 08:47:18.339: [ora.diskmon][1157011776] [check] DiskmonAgent::check {
    2013-07-17 08:47:18.339: [ora.diskmon][1157011776] [check] DiskmonAgent::check } 0
    2013-07-17 08:47:18.340: [    AGFW][1157011776] check for resource: ora.diskmon 1 1 completed with status: ONLINE
    2013-07-17 08:47:38.345: [    AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
    2013-07-17 08:47:38.345: [    AGFW][1157011776] Executing command: check for resource: ora.diskmon 1 1
    2013-07-17 08:47:38.346: [ora.diskmon][1157011776] [check] DiskmonAgent::check {
    2013-07-17 08:47:38.346: [ora.diskmon][1157011776] [check] DiskmonAgent::check } 0
    2013-07-17 08:47:38.346: [    AGFW][1157011776] check for resource: ora.diskmon 1 1 completed with status: ONLINE
    2013-07-17 08:47:42.211: [    AGFW][1177991488] Agent received the message: AGENT_HB[Engine] ID 12293:1198
    2013-07-17 08:47:58.351: [    AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
    2013-07-17 08:47:58.352: [    AGFW][1157011776] Executing command: check for resource: ora.diskmon 1 1
    2013-07-17 08:47:58.352: [ora.diskmon][1157011776] [check] DiskmonAgent::check {
    2013-07-17 08:47:58.352: [ora.diskmon][1157011776] [check] DiskmonAgent::check } 0
    2013-07-17 08:47:58.352: [    AGFW][1157011776] check for resource: ora.diskmon 1 1 completed with status: ONLINE
    2013-07-17 08:48:12.220: [    AGFW][1177991488] Agent received the message: AGENT_HB[Engine] ID 12293:1214
    2013-07-17 08:48:18.358: [    AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
    2013-07-17 08:48:18.359: [    AGFW][1157011776] Executing command: check for resource: ora.diskmon 1 1
    2013-07-17 08:48:18.359: [ora.diskmon][1157011776] [check] DiskmonAgent::check {
    2013-07-17 08:48:18.359: [ora.diskmon][1157011776] [check] DiskmonAgent::check } 0
    2013-07-17 08:48:18.359: [    AGFW][1157011776] check for resource: ora.diskmon 1 1 completed with status: ONLINE
    2013-07-17 08:48:38.365: [    AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
    2013-07-17 08:48:38.366: [    AGFW][1157011776] Executing command: check for resource: ora.diskmon 1 1
    2013-07-17 08:48:38.366: [ora.diskmon][1157011776] [check] DiskmonAgent::check {
    2013-07-17 08:48:38.366: [ora.diskmon][1157011776] [check] DiskmonAgent::check } 0
    2013-07-17 08:48:38.366: [    AGFW][1157011776] check for resource: ora.diskmon 1 1 completed with status: ONLINE
    2013-07-17 08:48:42.230: [    AGFW][1177991488] Agent received the message: AGENT_HB[Engine] ID 12293:1230
    2013-07-17 08:48:58.370: [    AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
    2013-07-17 08:48:58.371: [    AGFW][1157011776] Executing command: check for resource: ora.diskmon 1 1
    2013-07-17 08:48:58.371: [ora.diskmon][1157011776] [check] DiskmonAgent::check {
    2013-07-17 08:48:58.371: [ora.diskmon][1157011776] [check] DiskmonAgent::check } 0
    2013-07-17 08:48:58.371: [    AGFW][1157011776] check for resource: ora.diskmon 1 1 completed with status: ONLINE
    2013-07-17 08:49:12.238: [    AGFW][1177991488] Agent received the message: AGENT_HB[Engine] ID 12293:1246
    2013-07-17 08:49:18.377: [    AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
    2013-07-17 08:49:18.378: [    AGFW][1157011776] Executing command: check for resource: ora.diskmon 1 1
    2013-07-17 08:49:18.378: [ora.diskmon][1157011776] [check] DiskmonAgent::check {
    2013-07-17 08:49:18.379: [ora.diskmon][1157011776] [check] DiskmonAgent::check } 0
    2013-07-17 08:49:18.379: [    AGFW][1157011776] check for resource: ora.diskmon 1 1 completed with status: ONLINE
    2013-07-17 08:49:23.262: [    AGFW][1177991488] Agent received the message: RESOURCE_STOP[ora.diskmon 1 1] ID 4099:1310
    2013-07-17 0

  • Ora.asm -init failed on second node root.sh

    Hi All,
    Installing Grid Infrastructure for a 11gr2 Cluster on two nodes Oracle Linux 5 + Vsware vSphere v4, shared disk on same host machine. When run root.sh, first node was success but the second node got following error message (actually the first node was cloned from the seoncd):
    CRS-2672: Attempting to start 'ora.ctssd' on 'wandrac2'
    Start action for octssd aborted
    CRS-2676: Start of 'ora.ctssd' on 'wandrac2' succeeded
    CRS-2672: Attempting to start 'ora.drivers.acfs' on 'wandrac2'
    CRS-2672: Attempting to start 'ora.asm' on 'wandrac2'
    CRS-2676: Start of 'ora.drivers.acfs' on 'wandrac2' succeeded
    CRS-2676: Start of 'ora.asm' on 'wandrac2' succeeded
    CRS-2664: Resource 'ora.ctssd' is already running on 'wandrac2'
    CRS-4000: Command Start failed, or completed with errors.
    Command return code of 1 (256) from command: /orapp/racsl/11.2.0/bin/crsctl start resource ora.asm -init
    Start of resource "ora.asm -init" failed
    Failed to start ASM
    Failed to start Oracle Clusterware stack
    Thanks in advance for any information and helps,

    Hi,
    I came across this error and I am about to start a fresh installation of the grid. (ealier one failed because it was unable to read the memory in rac2 )
    Is there anything specific I can change before I start my installation.
    PS - I didnt get what exactly is going on with the hosts file.
    My files are as follows :
    RAC1 - etc/hosts
    [oracle@falcen6a ~]$ cat /etc/hosts
    # Do not remove the following line, or various programs
    # that require network functionality will fail.
    127.0.0.1 localhost.localdomain localhost
    ::1 localhost6.localdomain6 localhost6
    # Public
    192.168.100.218 falcen6a.a.pri falcen6a
    192.168.100.219 falcen6b.a.pri falcen6b
    # Private
    192.168.210.101 falcen6a-priv.a.pri falcen6a-priv
    192.168.210.102 falcen6b-priv.a.pri falcen6b-priv
    # Virtual
    192.168.100.212 falcen6a-vip.a.pri falcen6a-vip
    192.168.100.213 falcen6b-vip.a.pri falcen6b-vip
    # SCAN
    #192.168.100.208 falcen6-scan.a.pri falcen6-scan
    #192.168.100.209 falcen6-scan.a.pri falcen6-scan
    #192.168.100.210 falcen6-scan.a.pri falcen6-scan
    on RAC2
    [oracle@falcen6b ~]$ cat /etc/hosts
    # Do not remove the following line, or various programs
    # that require network functionality will fail.
    127.0.0.1 localhost.localdomain localhost
    ::1 localhost6.localdomain6 localhost6
    #Public
    192.168.100.218 falcen6a.a.pri falcen6a
    192.168.100.219 falcen6b.a.pri falcen6b
    # Private
    192.168.210.101 falcen6a-priv.a.pri falcen6a-priv
    192.168.210.102 falcen6b-priv.a.pri falcen6b-priv
    # Virtual
    192.168.100.212 falcen6a-vip.a.pri falcen6a-vip
    192.168.100.213 falcen6b-vip.a.pri falcen6b-vip
    # SCAN
    #192.168.100.208 falcen6-scan.a.pri falcen6-scan
    #192.168.100.209 falcen6-scan.a.pri falcen6-scan
    #192.168.100.210 falcen6-scan.a.pri falcen6-scan
    Can someone please confirm this??

  • Inst_loc inventory pointer missing on second node in RAC

    Hi,
    I have installed Clusterware 11.1.0.6 on Windows 2003 on two nodes and I'd like to patch it to 11.1.0.7. However, when calling "opatch lsinventory -all" I noticed that it works on the first node (the node where the installation was performed), but doesn't work on the second node:
    C:\product\11.1.0\crs\OPatch>opatch lsinventory -all
    Invoking OPatch 11.1.0.6.0
    Oracle Interim Patch Installer version 11.1.0.6.0
    Copyright (c) 2007, Oracle Corporation.  All rights reserved.
    Oracle Home       : C:\product\11.1.0\crs
    Central Inventory : n/a
       from           : n/a
    OPatch version    : 11.1.0.6.0
    OUI version       : 11.1.0.6.0
    OUI location      : C:\product\11.1.0\crs\oui
    Log file location : C:\product\11.1.0\crs\cfgtoollogs\opatch\opatch2010-11-28_10-22-19AM.log
    OPatch cannot find a valid oraInst.loc file to locate Central Inventory.
    OPatch failed with error code = 104
    C:\product\11.1.0\crs\OPatch>I checked in the registry key "HKEY_LOCAL_MACHINE\SOFTWARE\ORACLE" on both nodes and found out that the first node has an inventory pointer "inst_loc" wich points to "C:\Program Files\Oracle\Inventory", but on the second node this value is missing. If I manually set it on the second node, then "opatch lsinventory" works but I'm not sure if it's ok to set it manually?
    I also checked the documentation and here http://download.oracle.com/docs/cd/B28359_01/em.111/b31207/oui5_cluster_environment.htm#OUICG267 it says: "After you click Next, the Oracle Universal Installer checks whether the remote inventories are set. If they are not set, the Oracle Universal Installer sets up the remote inventories by setting registry keys." So I suppose the registry key should exist also on the second node?
    Thanks in advance for any answers.
    Regards,
    Jure

    Hi, mmm ... looks like remote operations fail during the installation process. Possible reasons for this error could be either a missing oraInst.loc file or permission issues with oraInst.loc file. Ensure user have read/write priviligies on orainst.loc file as well as on the actual path of orainventory location. If you could not fix the issue and if you know the inventory location you may want try following solution: "opatch apply -invPtrLoc C:\mypath\mypath\oraInst.loc" where "mypath" should be replaced by your windows locations. Also review you have all the oracle binaries in the 2nd. node.
    Edited by: Jose Valerio on Nov 28, 2010 2:21 PM
    Edited by: Jose Valerio on Nov 28, 2010 2:21 PM

Maybe you are looking for